feat(ingest): AI span classifier - #451
Open
JeremyFunk wants to merge 4 commits into
Open
Conversation
The `AiVendor` column speaks a closed slug vocabulary. Vendor knowledge itself stays in Rust on the write path; this is only the slug set and its display labels, so the read path can enumerate vendors without re-implementing a rule. Exposed as the `@maple/domain/ai` subpath rather than from the root barrel: it is a leaf concern, and the root barrel is imported by web and cli. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The classifier decides, per span, which AI framework emitted it and — when the framework carries one — what its conversation/session identity is. Rules are Rust: static per-vendor tables of predicates over the resource, the instrumentation scope, the span name and the attribute list, resolved first-match against a fixed vendor order, with the resource and scope work hoisted so the per-span cost is the attribute pass alone. Nothing here writes a row or reads a config. `ResourceContext::new` + `scope()` + `classify_span()` is the whole surface; the write path lands next. Also in this commit, because they are what makes the rules reviewable: - A vendored classification fixture replayed by default `cargo test`, with a manifest pinning each file's sha256 (hence the oxfmt ignore — a reformat would fail the gate). Fixture values sourced from non-seed captures are sanitized at generation: identifiers are pseudonymized with distinctness and shape preserved, free-text content masked. The raw captures stay in the private trace-capture repo. - A corpus replay that asserts at most one vendor claims any span, so a new rule cannot quietly steal another vendor's traffic. - A `cityHash64` port, because the session-key hash has to match ClickHouse's. - An `ai_classifier` bench group: the classifier's own cost, isolated from the pipeline. `any_value_string` becomes `pub(crate)` so the classifier canonicalizes attribute values through the same function the row writer uses — one canonicalization, so protobuf and JSON deliveries of the same span classify identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 13, 2026
… comments Move the reserved unknown:* VendorId variants to UNKNOWN_BASE (1000+) so new vendors extend the dense 0..N range without renumbering the buckets; VendorId::index() folds them back onto the tail for the bitmask/table paths. Compress the vendor rule comments to maintainer-grade docs — design-doc section references and review-process narratives removed, decoy-key and trap documentation kept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same facts — integrity model, re-vendor steps, format limits, golden provenance — in plain language instead of review-dossier prose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack position: 1 of 3. Base:
main. Followed byai2/02-ingest-write-path→ai2/03-vendors-rollup.Adds the AI span classifier to
apps/ingestand the vendor slug vocabulary theread path will speak. Nothing calls the classifier yet — this PR contains no
write-path wiring, no schema change and no new column. It is the pure function
plus its test corpus.
This series replaces the earlier registry-artifact + Rust/SQL-equivalence design
(the 6-PR stack, superseded). The rationale for the change lives in the write-side
plan's "What changed from v1" section and its migration appendix; this PR
description does not restate it.
What lands
ai_vendors.rsai_registry.rsai_classifier.rsResourceContext/ scope hoistingcityhash102.rsfixtures/classification/packages/domain/src/ai/@maple/domain/aisubpath: the closedAiVendorslug set and labels24 slugs: 21 vendors — 20 frameworks plus one synthesized for the OpenInference
OpenAI scope that several of them share — and the three
unknown:*tiers.RULES_VERSIONstarts at 1, established here as the initial production ruleset.Design properties worth reviewing against:
Purity is what keeps determinism testable and a future retro-fit expressible.
arrival order, or cross-span state.
second compiler. Vendor knowledge is code; the corpus is the contract.
AiSessionKeyHashisunsalted by design — the identifier it digests stays in the clear in
SpanAttributeson the same row, so a salt would add no confidentiality whilebreaking read-path computability.
Verification
cargo testinapps/ingest: 143 lib tests + 70 binary tests, 0 failuresacross the whole stack; on this PR alone, 133 lib + 69 binary.
against per-capture goldens (span totals, the entry vendor's span count and its
session-key-state histogram) plus a whole-fixture weighted vendor histogram
asserted by exact equality, in default
cargo test. The fixture's 2,117 linesand
expectations.jsonare checked againstmanifest.json's sha256(
4d907896…), so a reformat fails rather than silently changing the corpus, andmanifest.sourcenames the trace-capture commit, seed-set hash and generatorhash the numbers came from.
.oxfmtrc.jsoncignores the fixture directory forthe same reason. The anchor is re-derivable: trace-capture's corpus and
generator are committed,
manifest.source.dirtyisfalse, andgit checkout <commit> && bun run fixtureover there reproduces these bytes.The test asserts
dirty == falserather than type-checking the field, so are-vendor from an uncommitted tree — where the recorded commit names a revision
that cannot produce the bytes — fails instead of shipping decorative provenance.
described, by exact equality rather than
<= N: a rules change that lowers anumber is a ratchet to be taken in the same PR, one that raises it is a
regression.
openrouter(3,968 spans) may claim no specific vendor;eve_slack(2,311) andeve_slack_no_messages(1,927) are held at 92 and 60vercel_ai_sdkspans, which are genuine — eve is built on the AI SDK. Theai.eve.turnorchestration spans that the bareai.prefix used to claim areat 0 false positives, and the 9 per capture are separately asserted to land in
unknown:other, so "no longer a false positive" cannot silently mean "droppedout of the unknown tier". The test pins the set of captures it visits so a
regeneration cannot drop one.
with naive evaluate-every-vendor over every span in the fixture, so an
optimization cannot silently change semantics.
vs absent, garbage attributes never panic, protobuf and OTLP/JSON transports
agree.
Bench envelope (
cargo bench -- ai_classifier, local run, Apple silicon):The budget the plan sets is ~500 ns per span for everything — decode, classify,
row build, serialize — and cost scales with attribute count, not span count. The
typical case is the first two rows; the 60-attribute row is a deliberate worst
case built from the corpus' fattest record. The criterion leg in
ingest-rust-tests.ymlis opt-in behind thebench-ingestlabel, so these are ameasurement taken on demand, not an unconditional CI gate.
Notes for review
telemetry::any_value_stringbecomespub(crate). That is the only change tothe write path in this PR: the classifier canonicalizes attribute values through
the same function the row writer uses, so one logical span classifies identically
over both transports.
serde_yamlis a dev-dependency only — the corpus replay parses thetrace-capture seed goldens. It is never linked into the binary.
packages/domain/src/aiis deliberately not re-exported from the root@maple/domainbarrel, which web and cli import.packages/domain/src/ai/vendors.tsitself (include_str!undercfg(test),so release builds never open the path) and names the missing slug. There is no
codegen, no build-step coupling and — deliberately — no hand-copied second list:
that was the original mechanism and it made the test a tautology. Because the
Rust test reads the TypeScript file,
ingest-rust-tests.ymlnow also triggers onit, so a TS-only edit still runs
cargo test.captures are masked, and identifier-bearing keys (
session.id,user.id,gen_ai.conversation.id, the vertex invocation id, mastra's run id, and theai.settings.context.prefix) are pseudonymized by a keyed digest that preservesdistinctness, emptiness and UUIDv7 shape — which is what the classifier's rules
actually read, so the sanitization cannot change a verdict. 139 distinct
identifiers survive as distinct.
manifest.redactionrecords the scheme and thecounts. Raw captures live only in the private trace-capture repo; nothing in this
repo reaches them, and CI stays hermetic.
openrouterandeve_slackwere live captures duringrule development; both have been rolled back to their committed state and the
ingest URL moved at the origin, so the exact-equality histogram is now a stable
contract rather than a dated snapshot. The histogram's failure message still
distinguishes a rules regression from a re-vendor, because a re-vendor remains
the one legitimate way for those rows to move. Both files are byte-identical to
the revision
manifest.source.commitnames.expectations.jsonis transcribed from the seeds' own goldens, produced byapplying each seed's rules to its captures and then human-reviewed. A green
replay proves maple's Rust rules compute what the reviewed seed rules compute —
transcription fidelity, not ground truth. Six golden fields across four captures
are outside the seed algebra and were predicted by hand; that set is asserted so
it cannot grow silently, and
fixtures/classification/README.mdnames the twowith weaker gates.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.