Enable graph optimization for the tract backend - #1368
Draft
sebitokazu wants to merge 1 commit into
Draft
Conversation
`ort-tract` runs tract's `into_optimized()` only when the session requests a graph optimization level, and `Session::builder()` sets none by default, so every target except aarch64 Windows ran the unoptimized graph. Requesting `Level3` — ONNX Runtime's own default, leaving the native backend unaffected — makes `predict()` over a 2 s window 7.4x faster on an M-series release build (534.5 ms to 72.5 ms median, n=25). Fixture scores are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before you submit your PR
Make sure the following is true before submitting your PR:
PR description
livekit-wakewordbuilds every session with a bareSession::builder()and never sets a graph optimization level:On every target except aarch64 Windows the crate uses the
ort-tractbackend, andort-tractdecides whether to optimize the graph at all from that level:ort_tract::SessionOptionsderivesDefault, so with no level setperform_optimizationsisfalseand tract'sinto_optimized()is never called. Wake word inference runs the unoptimized graph.This adds one call,
with_optimization_level(GraphOptimizationLevel::Level3), shared by both session constructors.Level3is ONNX Runtime's own default graph optimization level, so the native backend on aarch64 Windows is unaffected; only the tract path changes.To reproduce, time
WakeWordModel::predictover a 2 s window on any non-Windows target before and after — details below.Breaking changes
None. No public API change; sessions are constructed the same way, with an optimization level now requested.
MSRV
Unchanged.
Testing
The existing
livekit-wakewordintegration tests pass unmodified (cargo test -p livekit-wakeword --release, 3 passed).There is no new unit test: the change alters no output, only how long producing it takes, so there is nothing to assert that would not be a timing assertion. Instead I measured both halves of that claim with a throwaway example against
tests/fixtures/, on this branch and onmain, back to back on the same machine (Apple M-series, release build):mainpredict,mainpredict, this branchpositive.wav, n=15negative.wav, n=15Scores are identical to every digit printed, which is the property that matters: graph optimization is supposed to be output-preserving here, and on these models it is. The timings are from a laptop and the spread between the two fixture rows on
mainis machine noise rather than signal — treat the magnitude (roughly 6-7x) as the claim, not the individual figures.I also saw this on aarch64 Linux (Rockchip, 4x Cortex-A55), against the published 0.1.3 rather than
main: 4511 ms to 1572 ms median, 2.87x. Host speedups do not transfer one-for-one, so that is corroboration of direction, not a second measurement of the same quantity.Async
No change. The production diff introduces no
.awaitand no runtime dependency.