[danielmawhirter]

Superseded by #169, which hosts the development branch directly in NVIDIA/kumo-engine. Production-readiness work and CI validation continue there.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[danielmawhirter]

Change map

Requested evaluation: port source PR #424 onto current Engine and reproduce production gaps. Existing default mmap behavior and seeded draws are retained while adding the complete proposed backend and benchmark surface. Focused verification covers affected Rust crates, native I/O/error/fork diagnostics, parent/train-table Python parity, and dependency notices. This draft stops at a reviewable candidate and documented failing release gates; production enablement waits for the focused follow-ups.

File	What changed	Why
3rdparty/notices/inventory.json	Regenerate resolved artifact dependencies and license notices.	Account for the new shipped I/O and profiling dependencies under the existing policy.
THIRD-PARTY.txt	Regenerate resolved artifact dependencies and license notices.	Account for the new shipped I/O and profiling dependencies under the existing policy.
packages/diskgraph/Cargo.lock	Add backend and benchmark dependencies while retaining current versions and optional TUI features.	Build the proposal without reverting current dependency and packaging controls.
packages/diskgraph/README.md	Describe the proposed backend, generator or benchmark entry points.	Connect the imported tools with their usage and evidence.
packages/diskgraph/diskgraph-datagenerator/README.md	Describe the proposed backend, generator or benchmark entry points.	Connect the imported tools with their usage and evidence.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_1pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_25pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_50pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_5pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_full_scale.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_100pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_10pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_1pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_25pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_50pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_5pct.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/pinterest_omnisage_pruned_organic.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/pinterest_omnisage_pruned_organic_graph.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/pinterest_omnisage_pruned_organic_size_matched.json	Carry compressed identity, ecommerce and organic Pinterest workload configuration.	Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/src/config.rs	Add source generator distributions, feature data and compression support.	Generate the complete proposed benchmark workloads.
packages/diskgraph/diskgraph-datagenerator/src/generator.rs	Add source generator distributions, feature data and compression support.	Generate the complete proposed benchmark workloads.
packages/diskgraph/diskgraph-engine/Cargo.toml	Add backend and benchmark dependencies while retaining current versions and optional TUI features.	Build the proposal without reverting current dependency and packaging controls.
packages/diskgraph/diskgraph-engine/src/bin/dg-engine-bench.rs	Add strict end-to-end sampler or page-I/O benchmark commands and telemetry.	Measure equivalent workloads with output and I/O verification gates.
packages/diskgraph/diskgraph-engine/src/engine.rs	Integrate explicit page planning and reusable sampling state; preserve current seeded selection and engine features.	Expose the new backend through the existing relation, projection and per-seed commit paths.
packages/diskgraph/diskgraph-engine/src/output.rs	Integrate explicit page planning and reusable sampling state; preserve current seeded selection and engine features.	Expose the new backend through the existing relation, projection and per-seed commit paths.
packages/diskgraph/diskgraph-engine/src/state.rs	Integrate explicit page planning and reusable sampling state; preserve current seeded selection and engine features.	Expose the new backend through the existing relation, projection and per-seed commit paths.
packages/diskgraph/diskgraph-engine/src/tests_unit.rs	Integrate explicit page planning and reusable sampling state; preserve current seeded selection and engine features.	Expose the new backend through the existing relation, projection and per-seed commit paths.
packages/diskgraph/diskgraph-index/Cargo.toml	Add backend and benchmark dependencies while retaining current versions and optional TUI features.	Build the proposal without reverting current dependency and packaging controls.
packages/diskgraph/diskgraph-index/src/bin/dg-page-io-bench.rs	Add strict end-to-end sampler or page-I/O benchmark commands and telemetry.	Measure equivalent workloads with output and I/O verification gates.
packages/diskgraph/diskgraph-index/src/lib.rs	Add cached explicit I/O, bounded record planning and optional mmap prefetch.	Implement the proposed sampler storage path while retaining computed offset positions.
packages/diskgraph/diskgraph-index/src/page_io.rs	Add cached explicit I/O, bounded record planning and optional mmap prefetch.	Implement the proposed sampler storage path while retaining computed offset positions.
packages/diskgraph/diskgraph-index/src/page_io_readiness_tests.rs	Add opt-in native roundtrip, cancellation and bounded fork/error diagnostics.	Reproduce readiness blockers with real I/O and deterministic ownership interleavings.
packages/diskgraph/diskgraph-index/src/prefetch.rs	Add cached explicit I/O, bounded record planning and optional mmap prefetch.	Implement the proposed sampler storage path while retaining computed offset positions.
packages/diskgraph/diskgraph-index/src/reader.rs	Add cached explicit I/O, bounded record planning and optional mmap prefetch.	Implement the proposed sampler storage path while retaining computed offset positions.
packages/diskgraph/diskgraph-index/tests/page_io_readiness.rs	Add opt-in native roundtrip, cancellation and bounded fork/error diagnostics.	Reproduce readiness blockers with real I/O and deterministic ownership interleavings.
packages/diskgraph/diskgraph-ingest/src/lib.rs	Carry the encode-budget override and buffered LZ4 output while retaining current ingestion behavior.	Preserve proposal tuning and performance changes without reverting newer budget and recovery handling.
packages/diskgraph/diskgraph-pybind/python/tests/test_native_io_readiness.py	Exercise parent/train-table topology, feature projection and automatic backend dispatch.	Verify the actual Python runtime boundary and expose the unsupported fused combination.
packages/diskgraph/docs/cached-direct-io-sampler.html	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/proposals/buffered-io-uring-sampler.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/proposals/direct-io-cache-sampler.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/README.md	Index the imported benchmark reports and current readiness review.	Keep historical performance evidence distinct from current release findings.
packages/diskgraph/docs/reports/diskgraph-all-edge-3hop-100pct-mmap-vs-direct-2026-08-07.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-all-edge-3hop-scale-matrix-2026-08-08.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-direct-cache-optimized-2026-08-07.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-ecommerce-feature-io-benchmark-2026-08-08.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-i8g-ecommerce-feature-io-benchmark-2026-08-10.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-i8g-identity-scale-matrix-2026-08-10.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-localssd-ecommerce-feature-mmap-vs-direct-2026-08-19.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-localssd-mmap-vs-direct-2026-08-18.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-pinterest-hotspot-real-pkey-2026-08-21.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-pinterest-i8g-io-benchmark-2026-08-09.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-pinterest-organic-capped-bfs-2026-08-21.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-real-pkey-e2e-mmap-vs-direct-2026-08-20.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-sampler-io-benchmark-2026-08-06.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/pr-424-production-readiness-2026-10-30.md	Record provenance, measured failures, validation and release gates.	Make the production decision and focused follow-up work reviewable.
packages/diskgraph/docs/sampling-mmap-baseline-2026-08-06.md	Retain the proposal design, methodology and historical benchmark report.	Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/scripts/bench_fio_iops.sh	Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction.	Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_fio_raw_matrix.sh	Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction.	Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_pinterest_organic_pkey.sh	Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction.	Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_pinterest_pruned_pkey.sh	Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction.	Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_sampling_iops.sh	Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction.	Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/evict_file_cache.py	Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction.	Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/third_party/artifact-manifest.json	Regenerate resolved artifact dependencies and license notices.	Account for the new shipped I/O and profiling dependencies under the existing policy.
tools/migrations/diskgraph/check_native_runtime.py	Add a bounded Kumo iterable, DataLoader and runtime-adapter diagnostic for each backend.	Verify the actual training integration and reproduce worker failure after parent I/O.