Reduce maintenance rescans and attribute accepted prompt reuse - #65
Merged
Conversation
Count consumed local prefixes, verified external reuse, and accepted target prompt computation across preemption attempts. Optional scheduler callbacks prevent offers, cancelled restores, and late worker drains from becoming reuse credit. Emit bounded per-request diagnostics only when reuse tracing is enabled. Cache identity and namespace are unchanged. Validation: 1,136 CPU tests passed with eight documented skips, 22 focused attribution tests passed, and Ruff passed.
Add opt-in unlink-attempt limits and cooldown between maintenance passes. Preserve protected reference graphs and durable root barriers; reclaim orphan debt before selecting further live roots. Expose deferred-work and cooldown telemetry. Full inventory scans remain unbounded; defaults, cache identities, and persisted formats are unchanged. Validation: 1136 SparkCache tests passed with 8 skips; 111 deployment tests passed with 1 skip; Ruff clean. GPU and serving performance qualification remains pending.
Keep a process-local pressure target until bounded cleanup reaches the low watermark. Continue background retries and non-forced maintenance below the high watermark while work remains, including after streaming commit finalization. Disabled deletion budgets preserve existing behavior and persisted cache identities are unchanged. Validation: 1251 SparkCache and deployment tests passed with 9 skips. Budget sizes 1, 2, and 4 converge to the same reclaimed bytes as unrestricted cleanup at 99/80 percent watermarks.
Compute inventory ordering keys once instead of repeatedly normalizing Path components during comparisons. Canonical namespace and digest ordering remains deterministic; stored identities and namespaces do not change. Validation: 1,156 CPU tests passed with eight skips, Ruff passed, and Linux profiling removed Path comparison calls without establishing a single-pass latency improvement.
Qualify surviving roots from validated descriptors and unique-file sizes during the exclusive pass, avoiding repeated root reads and shared-chunk probes. Preserve exact-root precedence and defer stale-inventory withdrawals when offers change, including same-digest republication. Restore integrity checks, cache namespaces, and persisted formats are unchanged. CPU validation: 1279 tests passed with 9 skips; Ruff clean. A Windows NTFS fixture with 257 flat-page roots reduced complete maintenance/reconciliation median latency from 175.23 ms to 72.22 ms and chunk metadata probes from 4995 to 514. GPU, serving, and NVMe qualification remains pending.
Separate source-build capabilities from the immutable published image, describe cumulative accepted-prompt accounting, and state the remaining full-scan and workload-validation limits. No runtime behavior or cache identity changes.
Resolve the measured baseline from its recorded source hashes and retain the combined-source repeat with its own timings and hashes. This evidence records CPU maintenance behavior only; it does not claim serving qualification.
This was referenced Sep 6, 2026
FujitsuPolycom
marked this pull request as ready for review
September 9, 2026 00:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capacity maintenance validated the full root inventory, then offer reconciliation reread surviving manifests and probed shared payload metadata once per reference. Reuse the metadata-qualified survivor set from the same exclusive inventory pass. Preserve exact-root precedence, declared payload sizes, shared references, and restore-time checksum verification. Concurrent inventory changes defer withdrawal instead of removing newly published offers, including republication of the same digest.
Status: implemented and CPU-tested. Serving-performance qualification remains research-only.
The connector also accepts optional authoritative scheduler events for per-request local reuse, verified external reuse, and accepted target-prompt computation. Offers and cancelled restores receive no consumed-reuse credit; preemption attempts remain explicit. This requires the matching SparkRing scheduler instrumentation. Tracing is opt-in and adds no request-ID Prometheus labels.
Optional deletion-attempt budgets and cooldowns pace maintenance without claiming a hard bound on inventory scans or individual filesystem operations. Both default to zero. The low-watermark target persists across paced passes, existing orphan debt is reclaimed before further live-root eviction, and failed unlink attempts consume the budget. Smaller budgets can increase total scan work and are not recommended as an unqualified performance default. Canonical string ordering removes repeated Python Path comparisons.
Compatibility: cache identity, namespace, and persisted object formats are unchanged. Deployment source receipts identify the combined package. Published images are unchanged by merging this code.
Validation: 1,279 SparkCache/deployment tests passed with nine documented skips; Ruff and whitespace checks passed. Tests cover cancelled and failed restores, shared GPU leases, preemption, stale completions, protected roots, corrupt/truncated objects, alias precedence, concurrent republication, cooldown, and multi-pass convergence to the low watermark. Runtime tests execute the transformed scheduler against the real attribution ledger.
On Windows/NTFS, a CPU fixture with eight branches of 32 flat page extensions (257 roots and 4,481 references, eviction disabled) reduced a complete forced maintenance/reconciliation cycle from a 175.23 ms baseline median to 78.41 ms with the combined source. Seven observations and exact source hashes are recorded in
docs/maintenance-inventory-validation.json. Root reads fell from 514 to 257 and chunk metadata probes from 4,995 to 514. These results do not establish DGX4 throughput or resolution of the original near-full workload.Related: #60 and #61 remain open for instrumented workload validation. The issue reporter's offered rerun or private probe artifacts are still needed for a matched reproduction.
Companion runtime: SparkRing #239 supplies the required scheduler events and combined source build.