Skip to content

feat(ingest): AI span classifier - #451

Open
JeremyFunk wants to merge 4 commits into
mainfrom
ai2/01-ai-classifier
Open

feat(ingest): AI span classifier#451
JeremyFunk wants to merge 4 commits into
mainfrom
ai2/01-ai-classifier

Conversation

@JeremyFunk

@JeremyFunk JeremyFunk commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Stack position: 1 of 3. Base: main. Followed by
ai2/02-ingest-write-pathai2/03-vendors-rollup.

Adds the AI span classifier to apps/ingest and the vendor slug vocabulary the
read path will speak. Nothing calls the classifier yet — this PR contains no
write-path wiring, no schema change and no new column. It is the pure function
plus its test corpus.

This series replaces the earlier registry-artifact + Rust/SQL-equivalence design
(the 6-PR stack, superseded). The rationale for the change lives in the write-side
plan's "What changed from v1" section and its migration appendix; this PR
description does not restate it.

What lands

File Role
ai_vendors.rs Vendor knowledge as plain Rust — one block per vendor, wire evidence in doc comments
ai_registry.rs Prefilter, exact-key map and span-name dispatch built from those declarations
ai_classifier.rs Evaluation: first-match resolution, session-key state ladder, ResourceContext / scope hoisting
cityhash102.rs Frozen in-crate port of the CityHash 1.0.2 variant ClickHouse vendors
fixtures/classification/ Vendored capture fixture + expectations + sha256 manifest
packages/domain/src/ai/ @maple/domain/ai subpath: the closed AiVendor slug set and labels

24 slugs: 21 vendors — 20 frameworks plus one synthesized for the OpenInference
OpenAI scope that several of them share — and the three unknown:* tiers.
RULES_VERSION starts at 1, established here as the initial production ruleset.

Design properties worth reviewing against:

  • Pure and I/O-free. No config fetch, no clock, no I/O inside classification.
    Purity is what keeps determinism testable and a future retro-fit expressible.
  • Per-span and order-independent. No decision depends on the trace tree, span
    arrival order, or cross-span state.
  • One evaluation target. There is no registry artifact, no rule DSL and no
    second compiler. Vendor knowledge is code; the corpus is the contract.
  • Hash is a storage format, not a secrecy boundary. AiSessionKeyHash is
    unsalted by design — the identifier it digests stays in the clear in
    SpanAttributes on the same row, so a salt would add no confidentiality while
    breaking read-path computability.

Verification

cargo test in apps/ingest: 143 lib tests + 70 binary tests, 0 failures
across the whole stack; on this PR alone, 133 lib + 69 binary.

  • Corpus replay — 41 seed-named captures (57 total, 9,945 spans) replay
    against per-capture goldens (span totals, the entry vendor's span count and its
    session-key-state histogram) plus a whole-fixture weighted vendor histogram
    asserted by exact equality, in default cargo test. The fixture's 2,117 lines
    and expectations.json are checked against manifest.json's sha256
    (4d907896…), so a reformat fails rather than silently changing the corpus, and
    manifest.source names the trace-capture commit, seed-set hash and generator
    hash the numbers came from. .oxfmtrc.jsonc ignores the fixture directory for
    the same reason. The anchor is re-derivable: trace-capture's corpus and
    generator are committed, manifest.source.dirty is false, and
    git checkout <commit> && bun run fixture over there reproduces these bytes.
    The test asserts dirty == false rather than type-checking the field, so a
    re-vendor from an uncommitted tree — where the recorded commit names a revision
    that cannot produce the bytes — fails instead of shipping decorative provenance.
  • False-positive ceiling — asserted on the negative sets rather than
    described, by exact equality rather than <= N: a rules change that lowers a
    number is a ratchet to be taken in the same PR, one that raises it is a
    regression. openrouter (3,968 spans) may claim no specific vendor;
    eve_slack (2,311) and eve_slack_no_messages (1,927) are held at 92 and 60
    vercel_ai_sdk spans, which are genuine — eve is built on the AI SDK. The
    ai.eve.turn orchestration spans that the bare ai. prefix used to claim are
    at 0 false positives, and the 9 per capture are separately asserted to land in
    unknown:other, so "no longer a false positive" cannot silently mean "dropped
    out of the unknown tier". The test pins the set of captures it visits so a
    regeneration cannot drop one.
  • Exclusivity — at most one vendor may claim any fixture or corpus span.
  • Indexed vs direct differential — the prefilter/candidate fast path agrees
    with naive evaluate-every-vendor over every span in the fixture, so an
    optimization cannot silently change semantics.
  • Robustness — order-independence, duplicate-key determinism, present-but-empty
    vs absent, garbage attributes never panic, protobuf and OTLP/JSON transports
    agree.

Bench envelope (cargo bench -- ai_classifier, local run, Apple silicon):

Case Time
non-AI span, 5 attributes 35.5 ns
non-AI span, 15 attributes 55.8 ns
AI span, 60 attributes 610 ns
AI span, 60 attributes, screened out 241 ns
hoist resource + scope (once per scope) 119 ns
20 AI spans behind one hoist 12.2 µs (610 ns/span)

The budget the plan sets is ~500 ns per span for everything — decode, classify,
row build, serialize — and cost scales with attribute count, not span count. The
typical case is the first two rows; the 60-attribute row is a deliberate worst
case built from the corpus' fattest record. The criterion leg in
ingest-rust-tests.yml is opt-in behind the bench-ingest label, so these are a
measurement taken on demand, not an unconditional CI gate.

Notes for review

  • telemetry::any_value_string becomes pub(crate). That is the only change to
    the write path in this PR: the classifier canonicalizes attribute values through
    the same function the row writer uses, so one logical span classifies identically
    over both transports.
  • serde_yaml is a dev-dependency only — the corpus replay parses the
    trace-capture seed goldens. It is never linked into the binary.
  • packages/domain/src/ai is deliberately not re-exported from the root
    @maple/domain barrel, which web and cli import.
  • The TS and Rust vendor lists are kept in step by a Rust test that parses
    packages/domain/src/ai/vendors.ts itself
    (include_str! under cfg(test),
    so release builds never open the path) and names the missing slug. There is no
    codegen, no build-step coupling and — deliberately — no hand-copied second list:
    that was the original mechanism and it made the test a tautology. Because the
    Rust test reads the TypeScript file, ingest-rust-tests.yml now also triggers on
    it, so a TS-only edit still runs cargo test.
  • Fixture values are sanitized at generation, not here. Values from non-seed
    captures are masked, and identifier-bearing keys (session.id, user.id,
    gen_ai.conversation.id, the vertex invocation id, mastra's run id, and the
    ai.settings.context. prefix) are pseudonymized by a keyed digest that preserves
    distinctness, emptiness and UUIDv7 shape — which is what the classifier's rules
    actually read, so the sanitization cannot change a verdict. 139 distinct
    identifiers survive as distinct. manifest.redaction records the scheme and the
    counts. Raw captures live only in the private trace-capture repo; nothing in this
    repo reaches them, and CI stays hermetic.
  • The corpus is frozen. openrouter and eve_slack were live captures during
    rule development; both have been rolled back to their committed state and the
    ingest URL moved at the origin, so the exact-equality histogram is now a stable
    contract rather than a dated snapshot. The histogram's failure message still
    distinguishes a rules regression from a re-vendor, because a re-vendor remains
    the one legitimate way for those rows to move. Both files are byte-identical to
    the revision manifest.source.commit names.
  • expectations.json is transcribed from the seeds' own goldens, produced by
    applying each seed's rules to its captures and then human-reviewed. A green
    replay proves maple's Rust rules compute what the reviewed seed rules compute —
    transcription fidelity, not ground truth. Six golden fields across four captures
    are outside the seed algebra and were predicted by hand; that set is asserted so
    it cannot grow silently, and fixtures/classification/README.md names the two
    with weaker gates.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

JeremyFunk and others added 2 commits August 13, 2026 13:21
The `AiVendor` column speaks a closed slug vocabulary. Vendor knowledge itself
stays in Rust on the write path; this is only the slug set and its display
labels, so the read path can enumerate vendors without re-implementing a rule.

Exposed as the `@maple/domain/ai` subpath rather than from the root barrel: it
is a leaf concern, and the root barrel is imported by web and cli.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The classifier decides, per span, which AI framework emitted it and — when the
framework carries one — what its conversation/session identity is. Rules are
Rust: static per-vendor tables of predicates over the resource, the
instrumentation scope, the span name and the attribute list, resolved
first-match against a fixed vendor order, with the resource and scope work
hoisted so the per-span cost is the attribute pass alone.

Nothing here writes a row or reads a config. `ResourceContext::new` +
`scope()` + `classify_span()` is the whole surface; the write path lands next.

Also in this commit, because they are what makes the rules reviewable:

- A vendored classification fixture replayed by default `cargo test`, with a
  manifest pinning each file's sha256 (hence the oxfmt ignore — a reformat
  would fail the gate). Fixture values sourced from non-seed captures are
  sanitized at generation: identifiers are pseudonymized with distinctness and
  shape preserved, free-text content masked. The raw captures stay in the
  private trace-capture repo.
- A corpus replay that asserts at most one vendor claims any span, so a new
  rule cannot quietly steal another vendor's traffic.
- A `cityHash64` port, because the session-key hash has to match ClickHouse's.
- An `ai_classifier` bench group: the classifier's own cost, isolated from the
  pipeline.

`any_value_string` becomes `pub(crate)` so the classifier canonicalizes
attribute values through the same function the row writer uses — one
canonicalization, so protobuf and JSON deliveries of the same span classify
identically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JeremyFunk and others added 2 commits August 13, 2026 15:00
… comments

Move the reserved unknown:* VendorId variants to UNKNOWN_BASE (1000+) so
new vendors extend the dense 0..N range without renumbering the buckets;
VendorId::index() folds them back onto the tail for the bitmask/table
paths. Compress the vendor rule comments to maintainer-grade docs —
design-doc section references and review-process narratives removed,
decoy-key and trap documentation kept.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same facts — integrity model, re-vendor steps, format limits, golden
provenance — in plain language instead of review-dossier prose.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant