Skip to content

Enable graph optimization for the tract backend - #1368

Draft
sebitokazu wants to merge 1 commit into
livekit:mainfrom
eighteen-labs:wakeword/enable-graph-optimization
Draft

Enable graph optimization for the tract backend#1368
sebitokazu wants to merge 1 commit into
livekit:mainfrom
eighteen-labs:wakeword/enable-graph-optimization

Conversation

@sebitokazu

Copy link
Copy Markdown
Contributor

Before you submit your PR

Make sure the following is true before submitting your PR:

  • I have read the contributing guidelines and validated that this PR will be accepted.
  • I have read and followed the principles regarding breaking changes, testing, and code quality.

PR description

livekit-wakeword builds every session with a bare Session::builder() and never sets a graph optimization level:

Ok(Session::builder()?.commit_from_memory(bytes)?)

On every target except aarch64 Windows the crate uses the ort-tract backend, and ort-tract decides whether to optimize the graph at all from that level:

// ort-tract 0.2.0, api.rs:168
options.perform_optimizations = graph_optimization_level != GraphOptimizationLevel::ORT_DISABLE_ALL;

// ort-tract 0.2.0, session.rs:63
let graph = Arc::new(if options.perform_optimizations {
    model.into_optimized()?
} else {
    model.into_typed()?
});

ort_tract::SessionOptions derives Default, so with no level set perform_optimizations is false and tract's into_optimized() is never called. Wake word inference runs the unoptimized graph.

This adds one call, with_optimization_level(GraphOptimizationLevel::Level3), shared by both session constructors. Level3 is ONNX Runtime's own default graph optimization level, so the native backend on aarch64 Windows is unaffected; only the tract path changes.

To reproduce, time WakeWordModel::predict over a 2 s window on any non-Windows target before and after — details below.

Breaking changes

None. No public API change; sessions are constructed the same way, with an optimization level now requested.

MSRV

Unchanged.

Testing

The existing livekit-wakeword integration tests pass unmodified (cargo test -p livekit-wakeword --release, 3 passed).

There is no new unit test: the change alters no output, only how long producing it takes, so there is nothing to assert that would not be a timing assertion. Instead I measured both halves of that claim with a throwaway example against tests/fixtures/, on this branch and on main, back to back on the same machine (Apple M-series, release build):

fixture score, main score, this branch median predict, main median predict, this branch
2 s 440 Hz sine, n=25 534.5 ms 72.5 ms
positive.wav, n=15 0.999688 0.999688 467.4 ms 82.5 ms
negative.wav, n=15 0.000943 0.000943 857.1 ms 71.8 ms

Scores are identical to every digit printed, which is the property that matters: graph optimization is supposed to be output-preserving here, and on these models it is. The timings are from a laptop and the spread between the two fixture rows on main is machine noise rather than signal — treat the magnitude (roughly 6-7x) as the claim, not the individual figures.

I also saw this on aarch64 Linux (Rockchip, 4x Cortex-A55), against the published 0.1.3 rather than main: 4511 ms to 1572 ms median, 2.87x. Host speedups do not transfer one-for-one, so that is corroboration of direction, not a second measurement of the same quantity.

Async

No change. The production diff introduces no .await and no runtime dependency.

`ort-tract` runs tract's `into_optimized()` only when the session requests a
graph optimization level, and `Session::builder()` sets none by default, so
every target except aarch64 Windows ran the unoptimized graph.

Requesting `Level3` — ONNX Runtime's own default, leaving the native backend
unaffected — makes `predict()` over a 2 s window 7.4x faster on an M-series
release build (534.5 ms to 72.5 ms median, n=25). Fixture scores are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant