Model heat-aware admission and SSD write pressure offline - #36
Merged
Conversation
Provide bounded hit counters, verified-span savings, shared-object cost accounting, TinyLFU shadow replay, staged-write budget reports, write-amplification estimates, and NVMe SMART counter parsing outside the shipped SparkCache package. Document the serving integration prerequisites and unsupported enforcement boundaries. Cache identity and on-disk namespaces are unchanged. Validate the prototype independently and preserve the pinned deployable-source hash.
This was referenced Aug 31, 2026
Identify committed unique bytes and staged write bytes already exposed by serving reports. Keep research-ledger ingestion and budget enforcement explicitly unsupported, and replace the obsolete fixed performance threshold with matched workload evidence requirements. No cache identity or runtime changes. Validation: 1123 tests passed, 8 skipped across sparkcache, deploy, and research.
FujitsuPolycom
marked this pull request as ready for review
September 6, 2026 19:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resulting behavior
Adds an offline, GPU-free research model for bounded hit heat, verified-restore recomputation savings, shared-trunk reference value, marginal retained-byte cost, simplified TinyLFU trace replay, hourly and daily staged-write budget reports, logical-to-host write-amplification estimates, and NVMe SMART Data Units Written deltas.
The research package is excluded from SparkCache wheels and runtime source identities. CI runs its tests separately and a static regression rejects imports from serving modules.
Technical reason
SparkCache retention uses TTL and manifest-recency LRU. Operators can observe device endurance, but the serving connector exposes publication-byte telemetry but does not implement frequency-aware admission, write-budget enforcement, or cache-attributed physical writes. The offline model makes those policy inputs testable without allowing advisory metadata to affect restore eligibility or inference.
Compatibility
Cache identity, digest salts, chunk geometry, manifest schemas, and on-disk namespaces are unchanged. No serving configuration, connector behavior, package artifact, image, or deployment profile changes. Heat metadata cannot make cached state eligible for restoration.
Status and limitations
Research-only. Budget reports do not enforce limits. TinyLFU evaluation is a simplified shadow model. Publication byte values are caller-supplied. Serving reports expose committed unique and staged bytes; automated ingestion into this research ledger is not implemented. SMART parsing consumes an operator-supplied 512-byte log page; it does not acquire device telemetry. Data Units Written measures quantized host writes for the whole device, not NAND-media amplification or directory-specific writes.
Validation
python -m pytest research -q: 22 passedpython -m pytest sparkcache deploy research -q: 1,123 passed, 8 skipped after integration with mainpython -m ruff check .: passed