Add opt-in full-capture cadence and publication backlog metrics - #63
Merged
Conversation
Select complete page captures at configured token-interval boundaries to limit repeated history processing. Expose pending publication count, reported age, and maintenance activity without waiting for capacity scans. The interval defaults to disabled; cache identity, persisted formats, and native ABI are unchanged, with no namespace migration. Metrics retain worker-report freshness semantics. Validation: 1,231 package/deployment tests passed, eight skipped; Ruff clean. Full captures trade additional writes for shorter history processing.
This was referenced Sep 6, 2026
Retain the full-capture and telemetry behavior while binding deployment receipts to the combined source tree. No cache wire identity or placement ABI changes. Validation: 1231 tests pass with 9 platform/optional skips on Windows; the C++ traversal test separately passes under WSL. GPU benchmark additions are not hardware-qualified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Long delta histories increase publication and restore work, while capture-ownership metrics do not show how long the background saver remains pending. This change adds an optional token-based full-capture cadence and additive saver/maintenance telemetry on top of the bounded restore and publication-lifetime implementation in the base branch.
Status: implemented; performance observations are research-only. This PR is stacked on #62 (
perf/verified-cache-io); review that dependency before merging these controls.spark_cache_page_snapshot_interval_tokensdefaults to 0, disabled. Its environment fallback isSPARK_CONTEXT_CACHE_PAGE_SNAPSHOT_INTERVAL_TOKENS. Before asynchronous sparse capture, the worker selects complete state when the result span crosses an interval bucket relative to the selected base. With interval 16,384, a 14,336→16,384 transition selects full capture; 16,384→18,432 remains sparse.The decision performs no filesystem or history reads. Complete captures use the existing format and ring admission limits; busy or oversized captures skip safely. This policy is a token cadence, not a universal history-depth bound. Cache identity and default publication behavior are unchanged.
The worker-statistics and Prometheus path adds:
vllm:sparkcache_publication_pending_rank_slots: saver admissions summed across physical ranks, not unique user requests.vllm:sparkcache_publication_oldest_pending_seconds: maximum reported admission age, including capture, queueing, commit, and reconciliation.vllm:sparkcache_maintenance_active_ranks: ranks reporting an active scan or survivor reconciliation.Terminal success, failure, and abort clear admission age. A shutdown timeout with a live saver remains pending. Metrics do not wait for the capacity lock. Values reflect the last worker reports and may stop refreshing while the engine is idle; another Prometheus scrape does not make the stored age a live clock. Existing report fields are preserved.
Validation:
python -m pytest sparkcache deploy -q: 1,231 passed, 8 skipped; Ruff clean.The README explains the write/capture tradeoff and sizing watermarks from the reusable working set, largest admitted publication, and measured publication/reclamation rates. The original 40 GiB issue remains outside the small-capacity comparison's qualification scope.
Related: #60 and #61. Neither issue is automatically closed by this draft.
Companion runtime integration and the repository-contained sustained-run evidence: SparkRing #227.