Skip to content

Install Rust with rustup and a rust-toolchain.toml - #174

Merged
aaron-kumo merged 1 commit into
mainfrom
rust-toolchain-toml
Nov 5, 2026
Merged

aaron-kumo merged 1 commit into
mainfrom
rust-toolchain-toml

Conversation

@aaron-kumo

@aaron-kumo aaron-kumo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What

Pin the rust version (1.99.1) and replace dtolnay/rust-toolchain with a local action that installs the pinned toolchain with the runner's own rustup and makes it the default. scripts/diskgraph reads the same file for the wheel settings and cache key.

Why

NVIDIA's enterprise Actions allowlist permits dtolnay/rust-toolchain only at its current SHA. Dependabot's bump (#18) makes every workflow that uses it fail at startup, including testing.yml, kumo-engine-pr.yml, playwright_ui.yml and the required kumo-core-gpu.yml, and every future bump would do the same. It's better to rely on native rustup than some external action.

Behavior changes

  • Tests and clippy move from 1.98.1 to 1.99.1, the stable release that PR and release wheels already build with. Wheels now stay on that pinned version instead of following the runner image.
  • Clippy 1.99 flags the #[must_use] that async-trait generates on DatasetRegistry and GraphStorage, so double_must_use is allowed there until the 1.100 bump.
  • Backend Docker images build DiskGraph with 1.99.1, because the toml travels with packages/diskgraph.
  • Developers building under packages/diskgraph get 1.99.1, which rustup 1.28.1 or later installs automatically.
  • Every Rust cache key changes once.

Change map and validation

@aaron-kumo
aaron-kumo had a problem deploying to gha-kumo-workflow-tests November 3, 2026 21:32 — with GitHub Actions Error
@aaron-kumo

aaron-kumo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Change map

File Function / scope What changed Why
packages/diskgraph/rust-toolchain.toml New file Pin Rust 1.99.1. One source for the version, shared by CI, wheels, Docker and developers.
.github/actions/rust-toolchain/action.yml New composite action Install the pinned toolchain with the runner's rustup, make it the default, add components, and export CARGO_HOME, CARGO_INCREMENTAL=0 and CARGO_TERM_COLOR=always. Replaces dtolnay/rust-toolchain; the exports keep build settings and rust-cache keys as before.
.github/workflows/_diskgraph-wheel.yml, .github/workflows/kumo-engine-pr.yml, .github/workflows/kumo-engine-wheel.yml, .github/workflows/testing.yml Rust setup steps Use the local action. The allowlist rejects the action's new SHA (#18).
scripts/diskgraph wheel_rust_toolchain, cmd_wheel_env, cmd_wheel_key Read the toolchain from the toml. Keep the wheel settings and cache key on the pin.
packages/diskgraph/diskgraph-core/src/dataset.rs, packages/diskgraph/diskgraph-core/src/storage.rs DatasetRegistry, GraphStorage #[allow(clippy::double_must_use)]. Clippy 1.99 flags async-trait's generated #[must_use]; clippy 1.100 fixes this.

Requested change: stop using dtolnay/rust-toolchain, which fails to start on #18, and pin the latest stable instead of 1.98.1.

Verification: actionlint, shellcheck, the workflow security policy, and the change-map checker pass. Rust CI and the GPU check pass, and the manual kumo-engine-wheel.yml run built all four platforms.

Out of scope: the 1.100 bump, which removes the double_must_use allows.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The change pins the diskgraph Rust toolchain to 1.99.1 and adds a local composite action to install it. Wheel settings read the configured channel and include it in the cache key. Wheel and testing workflows use the local action. The diskgraph core traits allow the double_must_use lint.

Merge Risk: Minimal · up to 56f48

No actionable merge-blocking risk is established from the available evidence.

Pre-merge checks: Passed 2 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive The required PR conversation comment beginning <!-- kumo-pr-change-map --> is not available. The checkout contains only the repository guidance that requires this comment (`docs/development/review.m… Provide the full conversation comment beginning <!-- kumo-pr-change-map --> or enable PR conversation retrieval, then reassess each proposed change against the repository diff and consumers.
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed The pull request changes 9 files. This does not exceed the failure threshold of 40 files.
Public Resource Access Passed The pull request adds no inaccessible external URL or external repository reference. It replaces the external dtolnay/rust-toolchain references with the local ./.github/actions/rust-toolchain acti…
Full details: Public Resource Access

Explanation

The pull request adds no inaccessible external URL or external repository reference. It replaces the external dtolnay/rust-toolchain references with the local ./.github/actions/rust-toolchain action. The new Rust toolchain source is the public Rust 1.99.1 channel, and its manifest and checksum were accessible anonymously from static.rust-lang.org with HTTP 200.

Full details: Change Map Design Review

Explanation

The required PR conversation comment beginning &lt;!-- kumo-pr-change-map --&gt; is not available. The checkout contains only the repository guidance that requires this comment (docs/development/review.md:7-13), while the supplied PR description provides only a link and summary. The code diff is inspectable, but it cannot establish whether the proposed change map documented the observed failure, necessity, focused verification, and stopping condition in that same comment.

Resolution

Provide the full conversation comment beginning &lt;!-- kumo-pr-change-map --&gt; or enable PR conversation retrieval, then reassess each proposed change against the repository diff and consumers.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 5d17c1d and 56f4874.

Files selected for processing: 4

  • .github/workflows/_diskgraph-wheel.yml
  • .github/workflows/kumo-engine-pr.yml
  • .github/workflows/kumo-engine-wheel.yml
  • .github/workflows/testing.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up Comments

Minor follow-up Run the shared backend wheel build from the staged source directory.

5-7: Minor · Functional Correctness · Quick win

Run the shared backend wheel build from the staged source directory.

The CPU and GPU builders invoke docker/build_diskgraph_wheel from /home/${USERNAME}. Its cargo update --manifest-path call does not use the staged rust-toolchain.toml, so both backend paths perform a Cargo operation with the Docker-installed stable toolchain instead of 1.98.1.

Change the shared script so its Cargo and wheel-build operations run from source_dir.

Suggested fix
     if [ ! -f "${source_dir}/pyproject.toml" ]; then
         echo "No staged DiskGraph pyproject.toml found at ${source_dir}" >&2
         exit 1
     fi
+    cd "$source_dir"
     ensure_cargo

Open the source location

Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 70637d3 and e8c09ca.

Files selected for processing: 7
  • .github/actions/rust-toolchain/action.yml
  • .github/workflows/_diskgraph-wheel.yml
  • .github/workflows/kumo-engine-pr.yml
  • .github/workflows/kumo-engine-wheel.yml
  • .github/workflows/testing.yml
  • packages/diskgraph/rust-toolchain.toml
  • scripts/diskgraph

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@aaron-kumo
aaron-kumo temporarily deployed to gha-kumo-workflow-tests November 3, 2026 22:20 — with GitHub Actions Inactive
@aaron-kumo
aaron-kumo added this pull request to stack #194 November 4, 2026 16:27
aaron-kumo added a commit that referenced this pull request Nov 4, 2026
pyo3 0.28's #[pyclass] expansion trips clippy 1.99's
clone_on_copy, which #174 allows in diskgraph-pybind. pyo3
0.29.3 fixes it, so drop the allow. arrow-pyarrow needs pyo3
0.29 from 59.1 and pyo3-log 0.13.5 accepts it, so arrow,
parquet and pyo3-log move together.

Follow their API changes: FixedSizeBinaryArray's Vec
conversions are now TryFrom (tests only),
MutableArrayData::extend is deprecated for try_extend, and
pyo3-build-config fields are read through getters. Regenerate
Cargo.lock, the artifact manifest, the notice inventory and
THIRD-PARTY.txt. Drop build_diskgraph_wheel's cargo update of
time: the staged source's Cargo.lock already pins 0.3.47.

Parquet files DiskGraph writes now mark float columns with
IEEE 754 total order, so pyarrow no longer prunes row groups
on their min/max. The upgrade also clears seven RUSTSEC
advisories in pyo3, zstd and thrift.
@aaron-kumo
aaron-kumo temporarily deployed to gha-kumo-workflow-tests November 4, 2026 17:51 — with GitHub Actions Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick comments: 1
.github/workflows/testing.yml: 1

370-370: Trivial · Security & Privacy · Quick win

Use the $/ self-repository reference for the local Rust toolchain action.

./.github/actions/rust-toolchain resolves from the job workspace. $/.github/actions/rust-toolchain resolves from this repository at the running commit and does not need a checkout. The zizmor self-repository audit flags ./ references to in-repo actions and recommends $/.

The current checkout order means both forms load the same action file, so this change does not alter the toolchain that gets installed. Apply the same edit to the other new rust-toolchain references in _diskgraph-wheel.yml, kumo-engine-pr.yml, and kumo-engine-wheel.yml.

Proposed change
-      - uses: ./.github/actions/rust-toolchain
+      - uses: $/.github/actions/rust-toolchain

Apply the same change to the diskgraph-clippy job at line 405.

Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 41a9c5d and 9fb7ae5.

Files selected for processing: 1
  • .github/workflows/testing.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@aaron-kumo
aaron-kumo removed this pull request from stack #194 November 4, 2026 18:39
@aaron-kumo
aaron-kumo temporarily deployed to gha-kumo-workflow-tests November 4, 2026 18:54 — with GitHub Actions Inactive
haoweiyu-kumo
haoweiyu-kumo previously approved these changes Nov 4, 2026
The enterprise allowlist permits dtolnay/rust-toolchain only at
its current SHA, so Dependabot's bump (#18) stops every workflow
that uses it from starting. Pin Rust in
packages/diskgraph/rust-toolchain.toml and install it with the
runner's rustup through a local action, which keeps exporting
CARGO_HOME, CARGO_INCREMENTAL=0 and CARGO_TERM_COLOR=always.

Pin 1.99.1, the stable release wheel builds already use, instead
of the 1.98.1 that tests and clippy used. Its clippy flags the
GraphStorage; allow it there. Backend images built from this
source follow the same pin.
@aaron-kumo
aaron-kumo merged commit 8494ede into main Nov 5, 2026
122 of 124 checks passed
@aaron-kumo
aaron-kumo deleted the rust-toolchain-toml branch November 5, 2026 19:39
danielmawhirter added a commit that referenced this pull request Nov 6, 2026
## Summary

Enable NVIDIA's federated GCP authentication in the existing
GCP-containing general shards and GCS integration workflow. Remove
Google login from Snowflake, whose selected tests do not need GCP and
whose workflow is excluded by the provider policy.

GCS credential setup runs after pytest skip selection and before
resource fixtures, including session workers. Explicit ADC takes
precedence; BigQuery workload mode requires federated ADC. Existing test
shards, coverage commands, and check names are preserved. GCS runs for
every PR and exposes failures; general tests skip only known
documentation changes. Credential lifetime/type cleanup follows in #225.

## Change map

[Change
map](#224 (comment))

## Validation

Signed head `386c680b` includes main's merged #174 Rust action fix.

- 58 focused local tests pass; changed-file flake8/isort and diff checks
pass. General/GCS actionlint passes; Snowflake retains ten pre-existing
missing-step references.
- [GPU CI
passes](https://github.com/NVIDIA/kumo-engine/actions/runs/42147408907/job/127662279622).
- [Testing and Coverage
passes](https://github.com/NVIDIA/kumo-engine/actions/runs/42147407662),
including every pytest shard, package suites, native checks, and
aggregate coverage.
- [All three GCP integration jobs
pass](https://github.com/NVIDIA/kumo-engine/actions/runs/42147407538):
10 tests across GCS, BigQuery + S3, and BigQuery ingestion.
-
[Connector/config](https://github.com/NVIDIA/kumo-engine/actions/runs/42147407662/job/127662203622):
1915 passed, 41 skipped, 1 xfailed. Both restricted BigQuery permission
tests executed and passed.
- Real-source Snowflake collection with invalid encoded GCP credentials
succeeds: 104 filesystem tests skip without credential resolution;
connector/REST collection includes 1513 nodes.

The [Snowflake connector
job](https://github.com/NVIDIA/kumo-engine/actions/runs/42147407528/job/127662134899)
still fails one assertion because metadata includes static shared
`HM_SAMPLED.TEST_TABLE`. This identical failure predates these PRs on
[main](https://github.com/NVIDIA/kumo-engine/actions/runs/41911685209/job/126860619478).
Its creation/cleanup ownership has not been established; leftover state
and permission drift remain hypotheses. The assertion is retained
pending a supported fixture/contract fix. Existing mixed-shard
continue-on-error behavior and broader provider isolation remain outside
scope.
danielmawhirter added a commit that referenced this pull request Nov 6, 2026
## Summary

Google integration tests require explicit local ADC files instead of
retrieving keys from AWS, decoding environment keys, or discovering
home-directory credentials. Typed BigQuery identities select restricted
key files or impersonation targets; workload mode requires federated
ADC.

Credential setup follows test skip selection and precedes resource
fixtures. The base identity survives worker teardown, each test restores
credential environment and GCS client cache, and input files remain
unchanged. Mixed local invocations that already started DiskGraph
workers fail with instructions to use a separate invocation. Depends on
#224; production connector authentication semantics and existing check
names are preserved.

## Change map

[Change
map](#225 (comment))

## Validation

Signed head `9920279c` includes main's merged #174 Rust action fix
through #224.

- 84 combined local credential/workflow tests pass; changed-file
flake8/isort and diff checks pass. Documentation checker: 257 files
pass.
- [GPU CI
passes](https://github.com/NVIDIA/kumo-engine/actions/runs/42147417204/job/127662289614).
- [Testing and Coverage
passes](https://github.com/NVIDIA/kumo-engine/actions/runs/42147415744),
including every pytest shard, package suites, native checks, and
aggregate coverage.
- [All three GCP integration jobs
pass](https://github.com/NVIDIA/kumo-engine/actions/runs/42147415537):
10 tests across GCS, BigQuery + S3, and BigQuery ingestion.
-
[Connector/config](https://github.com/NVIDIA/kumo-engine/actions/runs/42147415744/job/127662257242):
1915 passed, 41 skipped, 1 xfailed. Both restricted BigQuery permission
tests executed and passed.
- Real Snowflake filesystem invocation with invalid encoded GCP
credentials: 104 tests skip without credential resolution.

The [Snowflake connector
job](https://github.com/NVIDIA/kumo-engine/actions/runs/42147415442/job/127662161433)
still fails one assertion because metadata includes static shared
`HM_SAMPLED.TEST_TABLE`. This identical failure predates these PRs on
[main](https://github.com/NVIDIA/kumo-engine/actions/runs/41911685209/job/126860619478).
Its creation/cleanup ownership has not been established; leftover state
and permission drift remain hypotheses. The assertion is retained
pending a supported fixture/contract fix.

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — 56f4874e Deployed Nov 5, 2026 by aaron-kumo via import #1569
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants