Skip to content

Add opt-in full-capture cadence and publication backlog metrics - #63

Merged
FujitsuPolycom merged 2 commits into
mainfrom
perf/cache-capture-telemetry
Sep 6, 2026
Merged

Add opt-in full-capture cadence and publication backlog metrics#63
FujitsuPolycom merged 2 commits into
mainfrom
perf/cache-capture-telemetry

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Long delta histories increase publication and restore work, while capture-ownership metrics do not show how long the background saver remains pending. This change adds an optional token-based full-capture cadence and additive saver/maintenance telemetry on top of the bounded restore and publication-lifetime implementation in the base branch.

Status: implemented; performance observations are research-only. This PR is stacked on #62 (perf/verified-cache-io); review that dependency before merging these controls.

spark_cache_page_snapshot_interval_tokens defaults to 0, disabled. Its environment fallback is SPARK_CONTEXT_CACHE_PAGE_SNAPSHOT_INTERVAL_TOKENS. Before asynchronous sparse capture, the worker selects complete state when the result span crosses an interval bucket relative to the selected base. With interval 16,384, a 14,336→16,384 transition selects full capture; 16,384→18,432 remains sparse.

The decision performs no filesystem or history reads. Complete captures use the existing format and ring admission limits; busy or oversized captures skip safely. This policy is a token cadence, not a universal history-depth bound. Cache identity and default publication behavior are unchanged.

The worker-statistics and Prometheus path adds:

  • vllm:sparkcache_publication_pending_rank_slots: saver admissions summed across physical ranks, not unique user requests.
  • vllm:sparkcache_publication_oldest_pending_seconds: maximum reported admission age, including capture, queueing, commit, and reconciliation.
  • vllm:sparkcache_maintenance_active_ranks: ranks reporting an active scan or survivor reconciliation.

Terminal success, failure, and abort clear admission age. A shutdown timeout with a live saver remains pending. Metrics do not wait for the capacity lock. Values reflect the last worker reports and may stop refreshing while the engine is idle; another Prometheus scrape does not make the stored age a live clock. Existing report fields are preserved.

Validation:

  • python -m pytest sparkcache deploy -q: 1,231 passed, 8 skipped; Ruff clean.
  • Tests cover settings and precedence, unchanged identity, bucket transitions, complete multi-group capture, ring rejection, independent snapshot publication, lifetime cleanup, blocked maintenance, aggregation, report freshness, and millisecond-to-second conversion.
  • A CPU-only fixture with 25 extensions, eight appended attention layers, four overwritten recurrent layers, and payload/object widths scaled down by eight measured median commit 325→137 ms and restore 317→125 ms when publishing a full snapshot every eight extensions. Staged bytes increased 48.6%, with zero deduplication.
  • In a combined four-GB10 MTP3 run with two concurrent conversations and 2 GiB cache capacity per rank, the 49,152-token full snapshot measured 400 ms median commit versus 2,541 ms for the 47,104-token delta-backed entry. Later restores measured 197 ms and 876 ms, respectively. These observations include other runtime changes and are not an isolated policy comparison.

The README explains the write/capture tradeoff and sizing watermarks from the reusable working set, largest admitted publication, and measured publication/reclamation rates. The original 40 GiB issue remains outside the small-capacity comparison's qualification scope.

Related: #60 and #61. Neither issue is automatically closed by this draft.

Companion runtime integration and the repository-contained sustained-run evidence: SparkRing #227.

Select complete page captures at configured token-interval boundaries to limit repeated history processing. Expose pending publication count, reported age, and maintenance activity without waiting for capacity scans.

The interval defaults to disabled; cache identity, persisted formats, and native ABI are unchanged, with no namespace migration. Metrics retain worker-report freshness semantics. Validation: 1,231 package/deployment tests passed, eight skipped; Ruff clean. Full captures trade additional writes for shorter history processing.
Retain the full-capture and telemetry behavior while binding deployment receipts to the combined source tree. No cache wire identity or placement ABI changes. Validation: 1231 tests pass with 9 platform/optional skips on Windows; the C++ traversal test separately passes under WSL. GPU benchmark additions are not hardware-qualified.
@FujitsuPolycom
FujitsuPolycom changed the base branch from perf/verified-cache-io to main September 6, 2026 19:05
@FujitsuPolycom
FujitsuPolycom marked this pull request as ready for review September 6, 2026 19:05
@FujitsuPolycom
FujitsuPolycom merged commit faed627 into main Sep 6, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant