Add secure opt-in fleet metrics and Git synchronization reporting

[cooktheryan]

FetchIt previously exposed liveness and invocation attempts, leaving fleet operators unable to tell whether configured Git methods had applied the latest observed revision. Add an opt-in Prometheus /metrics listener reporting configured method counts, synchronized/out-of-sync/unknown counts, observation freshness, and host identity.

The dedicated FETCHIT_METRICS_ADDR listener exposes only metrics, avoiding publication of the existing status endpoint's repository URLs. FETCHIT_HOSTNAME identifies the underlying host in metrics, response headers, and status JSON. Scrapes read memory and do not invoke Git or Podman; labels exclude repository URLs, method names, paths, credentials, error text, and revision SHAs. The guide documents explicit port publication, trusted-network access restrictions, the absence of native TLS/authentication, an existing remote Prometheus scrape configuration, and alert examples.

Git reconciliation explicitly records observations and successful application/baseline completion. Failed application, rollback, and skipped bad commits remain out of sync; clone/replay/source failures remain unknown. Failed source checks retain their previous observation timestamp. Generation and run tokens reject retired or superseded results. Disconnected/non-Git methods are counted as configured declarations but excluded from connected Git synchronization counts.

Validation:

• Full Go suite, targeted race checks, and go vet passed.
• Strict Sphinx documentation build and promtool check rules passed.
• Built the production Dockerfile on disposable Fedora 44 EC2 (Podman 5.8.7).
• Real Podman integration exercised scheduled Git update, malformed workload application, source loss, unchanged observation freshness, and recovery without engine restart.
• Prometheus 3.5.0 on a separate EC2 instance scraped the private endpoint and recorded state transitions with hostname and target identity; promtool check metrics passed.
• Verified private collector access, denial of public TCP 8080, slow-header timeout, rejected status/debug/write routes, metadata exclusion, and repeated read-only scrapes.
• Added the reconciliation integration test and metrics race coverage to CI.

Fixes #418.

Summary by Sourcery

Add secure opt-in fleet metrics and Git synchronization reporting for operators.

New Features:

• Add an opt-in, metrics-only Prometheus listener reporting configured methods, Git synchronization state, observation freshness, and host identity.
• Expose the configured host identity through metrics, status JSON, and response headers.

Bug Fixes:

• Ensure Git reconciliation reports unknown, out-of-sync, and synchronized states accurately across failed applications, rollbacks, skipped commits, source outages, cancellation, and recovery.
• Prevent stale or superseded runs from publishing observations after configuration replacement or newer runs.

Enhancements:

• Keep metrics read-only and bounded, excluding sensitive repository, method, path, credential, error, and revision metadata while distinguishing connected Git methods from non-applicable declarations.

CI:

• Run the reconciliation metrics integration test in Docker CI and include metrics race coverage in engine tests.

Documentation:

• Document metrics configuration, network exposure restrictions, authentication and TLS considerations, Prometheus scraping, and alerting examples.

Tests:

• Add unit, concurrency, route-security, metadata, freshness, rollback, and Podman-based integration coverage for metrics and Git state transitions.

[sourcery-ai]

Sorry @cooktheryan, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 17 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

This PR adds an opt-in, dedicated Prometheus endpoint that reports secure fleet-level Git synchronization state and freshness without exposing repository metadata, integrates generation-safe reconciliation observations across reloads and failures, propagates validated host identity through metrics and status APIs, and validates the behavior through unit/race tests, CI integration, documentation, and real Podman scenarios.

Sequence diagram for Git reconciliation metrics reporting

sequenceDiagram
    participant Scheduler
    participant Method
    participant Metrics
    participant Git
    participant Target

    Scheduler->>Metrics: begin(ctx, m)
    Scheduler->>Method: Process(ctx, conn, skew)
    Method->>Git: fetch latest revision
    Git-->>Method: latest revision
    Method->>Metrics: observeGit(ctx, unknown, true)
    Method->>Target: getCurrent(...)
    Target-->>Method: current revision
    Method->>Metrics: observeGit(ctx, out_of_sync, false)
    alt application succeeds or baseline is complete
        Method->>Target: applyWithRecovery(...)
        Method->>Metrics: observeGit(ctx, synced, false)
    else source, clone, replay, or application failure
        Method-->>Scheduler: reconciliation error
    end
    Note over Metrics: Scrapes read registry memory only

Loading

Flow diagram for secure fleet metric collection

flowchart TD
    Start[FetchIt starts] --> Host[configuredHostname]
    Host --> Registry["metrics.replace(methods, hostname)"]
    Registry --> OptIn{FETCHIT_METRICS_ADDR configured?}
    OptIn -->|no| Run[Run configured methods]
    OptIn -->|yes| Listener[startMetricsServer]
    Listener --> Endpoint[GET /metrics]
    Endpoint --> Snapshot[Read registry memory under lock]
    Snapshot --> Labels[Emit bounded kind and state labels]
    Labels --> Output[Emit configured counts, Git sync state, freshness, hostname]
    Run --> Observe[Reconciliation observations update registry]
    Observe --> Snapshot

Loading

File-Level Changes

Change	Details	Files
Added an opt-in, metadata-safe Prometheus metrics listener with host identity reporting.	Starts a dedicated listener only when FETCHIT_METRICS_ADDR is configured and restricts it to GET/HEAD /metrics. Aggregates configured method counts, connected Git synchronization states, and oldest successful observation timestamps by bounded method kind. Validates FETCHIT_HOSTNAME and exposes it in metrics, status JSON, and response headers without repository or secret-bearing labels. Uses bounded HTTP timeouts and synchronized in-memory snapshots so scrapes are read-only and safe during reloads and concurrent runs.	pkg/engine/metrics.go pkg/engine/metrics_test.go pkg/engine/fetchit.go pkg/engine/status.go
Integrated Git reconciliation outcomes and lifecycle tokens into synchronization metrics.	Records unknown after Git observation, out-of-sync before application, and synced only after successful application or baseline completion. Preserves observation freshness across source failures and leaves failed application, rollback, and skipped bad commits out of sync. Excludes disconnected and non-Git methods from connected Git synchronization totals while retaining them as configured declarations. Rejects updates from cancelled, retired, superseded, or replaced runs using generation-specific context tokens.	pkg/engine/common.go pkg/engine/quadlet.go pkg/engine/fetchit.go
Added automated unit, race, CI, and production-style integration coverage for metrics behavior and security boundaries.	Adds tests for reconciliation transitions, freshness, rollback and replay behavior, metadata exclusion, route restrictions, hostname validation, retirement, cancellation, reloads, and concurrent scrapes. Runs the metrics reconciliation integration test in Docker CI and includes metrics tests in the engine race-check selection. Exercises scheduled Git updates, malformed workloads, source loss and recovery with real Podman and validates read-only scrapes and private listener behavior.	pkg/engine/metrics_test.go tests/integration/metrics/test-metrics.sh tests/integration/metrics/README.md tests/integration/live-update/git-server.py .github/workflows/docker-image.yml .github/workflows/unit-tests.yml
Documented deployment, security, and Prometheus integration requirements for fleet metrics.	Documents explicit port publication and trusted-network restrictions because the listener has no native TLS or authentication. Explains metric semantics, excluded labels, hostname configuration, remote scraping, and alert examples.	docs/metrics.rst docs/index.rst docs/status.rst

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#418	Provide a metrics endpoint for fleet operators to inspect FetchIt configuration and synchronization status.	✅
#418	Expose useful metrics, including configured method counts, synchronized/out-of-sync/unknown Git states, and observation freshness.	✅
#418	Include sufficient host identity in metrics and document secure, opt-in usage of the endpoint.	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.