Skip to content

Support AutoEnzyme Jacobians inside Reactant kernels - #1067

Closed
ChrisRackauckas-Claude wants to merge 1 commit into
JuliaDiff:mainfrom
ChrisRackauckas-Claude:fix/reactant-jacobian
Closed

Support AutoEnzyme Jacobians inside Reactant kernels#1067
ChrisRackauckas-Claude wants to merge 1 commit into
JuliaDiff:mainfrom
ChrisRackauckas-Claude:fix/reactant-jacobian

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown

Ignore this PR until it has been reviewed by @ChrisRackauckas.

What changed and why

AutoEnzyme() Jacobian preparation scalar-indexes Reactant traced arrays, so ordinary DifferentiationInterface Jacobian calls fail inside Reactant.@jit. This adds a Reactant weak extension that detects compilation and routes forward/default AutoEnzyme Jacobians through Enzyme's traced-compatible batched forward autodiff path (onehot directions plus DI pushforwards). Both out-of-place and mutating functions return Reactant arrays and support allocating, mutating, and value-plus-Jacobian APIs.

The higher-level Enzyme.jacobian convenience function is not used because its forward implementation currently fails in tupstack for traced arrays; the underlying public Enzyme primitives work. The mutating-function route wraps the residual out of place because differentiating a mutated traced output directly produced zero tangents.

Closes #1066

This is the DifferentiationInterface prerequisite for removing the custom Jacobian lowering in SciML/NonlinearSolve.jl#1197.

Verification

Failing before the fix, using the exact new test file against unmodified main at f0fb136d:

$ TMPDIR="$PWD/../di_base_repro/.tmp" JULIA_DEPOT_PATH="$PWD/../di_repro/.julia_depot" ~/.juliaup/bin/julia +1.12.6 --project=../di_base_repro -e 'include("DifferentiationInterface/test/Back/EnzymeReactant/test.jl")'
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 2 errored, 0 broken.
Scalar indexing is disallowed.
Test Summary:                       | Error  Total   Time
AutoEnzyme Jacobian inside Reactant |     2      2  28.1s

Passing with the fix:

$ TMPDIR="$PWD/../di_repro/.tmp" JULIA_DEPOT_PATH="$PWD/../di_repro/.julia_depot" JULIA_DI_TEST_GROUP=EnzymeReactant ~/.juliaup/bin/julia +1.12.6 --project=DifferentiationInterface/test/Back/EnzymeReactant -e 'using Pkg; Pkg.develop([PackageSpec(path="DifferentiationInterface"), PackageSpec(path="DifferentiationInterfaceTest")]); Pkg.instantiate(); include("DifferentiationInterface/test/Back/run_backend.jl")'
Test Summary:  | Pass  Total     Time
EnzymeReactant |    8      8  1m10.9s

Existing Enzyme backend group:

$ TMPDIR="$PWD/../di_repro/.tmp" JULIA_DEPOT_PATH="$PWD/../di_repro/.julia_depot" JULIA_DI_TEST_GROUP=Enzyme ~/.juliaup/bin/julia +1.12.6 --project=DifferentiationInterface/test/Back/Enzyme -e 'using Pkg; Pkg.develop(path="DifferentiationInterface"); Pkg.develop(path="DifferentiationInterfaceTest"); Pkg.instantiate(); include("DifferentiationInterface/test/Back/run_backend.jl")'
Test Summary: |  Pass  Broken  Total      Time
Enzyme        | 93886       2  93888  80m26.4s

Core formalities, including Aqua, JET, documentation, and ExplicitImports:

$ TMPDIR="$PWD/../di_repro/.tmp" JULIA_DEPOT_PATH="$PWD/../di_repro/.julia_depot" JULIA_DI_TEST_GROUP=Internals ~/.juliaup/bin/julia +1.12.6 -e 'using Pkg; Pkg.activate("./DifferentiationInterface/test"); Pkg.develop(path="./DifferentiationInterfaceTest"); Pkg.instantiate(); Pkg.activate("./DifferentiationInterface"); Pkg.test("DifferentiationInterface"; allow_reresolve=false)'
Test Summary:                      | Pass  Broken  Total     Time
DifferentiationInterface.jl (Core) |  162       2    164  1m48.0s
Testing DifferentiationInterface tests passed

Repository pre-commit hooks passed for every changed file: typos, YAML/TOML validation, large-file and conflict checks, whitespace/EOF checks, and Runic formatting.

Not verified

  • Reactant compilation with explicit reverse-mode AutoEnzyme is not supported by this change; default and forward mode are the intended Jacobian route.
  • Julia 1.10 and 1.11 Reactant runs were not executed locally. Draft CI exercises Julia 1.12; the non-draft matrix will exercise all three supported Julia versions.
  • The docs build was not run because no docstring, rendered documentation, or public API changed.
  • GPU-specific Reactant execution was not run; the test used the PJRT CPU runtime.

Reviewer notes

  • Reactant is a weak dependency with minimum compat 0.2.283, the version used for the reproducer and verification.
  • Reactant and DifferentiationInterface are both MIT-licensed.

🤖 Generated with Claude Code
https://chatgpt.com/codex/tasks/01a03a07-4f58-7d73-90d2-5e7aa3ba9fd7

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>

Co-Authored-By: Claude <noreply@anthropic.com>

Claude-Session: https://chatgpt.com/codex/tasks/01a03a07-4f58-7d73-90d2-5e7aa3ba9fd7
@codecov

codecov Bot commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.70130% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.24%. Comparing base (f0fb136) to head (571fc17).

Files with missing lines Patch % Lines
...fferentiationInterface/src/first_order/jacobian.jl 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1067      +/-   ##
==========================================
- Coverage   97.40%   93.24%   -4.17%     
==========================================
  Files         143      142       -1     
  Lines        8299     8334      +35     
==========================================
- Hits         8084     7771     -313     
- Misses        215      563     +348     
Flag Coverage Δ
DI 94.53% <98.70%> (-3.39%) ⬇️
DIT 89.71% <ø> (-6.33%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gdalle

gdalle commented Aug 26, 2026

Copy link
Copy Markdown
Member

Mister bot (and his owner), opening a PR when I reacted to the issue saying it should be fixed upstream is not cool. If you want a tracing-friendly Jacobian function, I think DI.AutoForwardFromPrimitive(DI.AutoEnzyme()) would do the job nicely (haven't tested locally though).

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Author

CI audit (2026-08-26): every executable check is green, including the new EnzymeReactant backend, the full Enzyme backend suite, docs, pre-commit, and Buildkite. codecov/patch is also green at 98.70% (one changed line is uncovered).

codecov/project remains red at 93.24%. This is the aggregate report for the reduced draft-PR upload set compared with the fuller base-branch upload set, rather than a changed-line coverage failure. The same green-patch/red-project pattern is visible on another current draft PR (#975). The PR remains a draft as required for review by @ChrisRackauckas.

@ChrisRackauckas

Copy link
Copy Markdown
Member

Mister bot (and his owner), opening a PR when I reacted to the issue saying it should be fixed upstream is not cool.

Sorry this and the issue were "done at the same time", so your comment came "after" the PR. In timing-wise, yes you did slip the comment in during a 1 hour period while it was doing a local validation of the code, and it should've validated the code before opening the issue, but the issue+PR were done at exactly the same time for reasons I hope are pretty clear.

If you want a tracing-friendly Jacobian function, I think DI.AutoForwardFromPrimitive(DI.AutoEnzyme()) would do the job nicely (haven't tested locally though).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AutoEnzyme Jacobian preparation scalar-indexes Reactant traced arrays

3 participants