feat(ingest): write AI classification columns on the ingest path

[JeremyFunk]

Stack position: 2 of 3. Base: ai2/01-ai-classifier. Followed by
ai2/03-vendors-rollup. Review 01 first — it adds the classifier this PR calls.

Wires the classifier into the row writer and gives its verdict somewhere to live:
five columns on traces, in ClickHouse, in Tinybird, and in the local store.

Rationale for the overall design lives in the write-side plan (§2–§4 and the
migration appendix); this description covers what the diff does and how it is
verified.

What lands

Storage — ClickHouse migration 0016, mirrored on the Tinybird datasource and
in local schema v6:

Column	Type	Default
AiVendor	LowCardinality(String)	'' — not classified as AI
AiSessionKeyState	UInt8	0 — never examined
AiSessionKeyHash	UInt64	0
AiRulesVersion	UInt32	0 — row predates classification
AiRollupHour	DateTime('UTC')	toDateTime(0)

Plus idx_ai_vendor (set(0), unbounded because the vendor domain is a closed
allowlist and a capped set() degrades to always-match) and idx_scope_name
(tokenbf_v1, token-level lookups for scope prefixes).

Every column carries a constant DEFAULT, which is what makes the ALTER
metadata-only: no part is rewritten, and pre-classifier rows read the defaults.
There is deliberately no MATERIALIZE INDEX — a whole-table mutation on traces
is the expensive mistake, and the 30-day TTL retires unindexed parts on its own.

0016 is requiredForIngest. The gateway's INSERT names all five columns, so a
BYO cluster that has not applied it resolves clickhouse_ready = false and routes
to the managed pipeline until its schema syncs — clickHouseSchemaVersion becomes
"16". That fallback is the designed behavior, not an incident.

(Numbering note: main shipped its own migration 0015 — the minutely service
overview — and its own local schema v5 while this stack was open, so the AI
columns are 0016 / local v6. The two numbering schemes are unrelated: the
ClickHouse migration number and the local-store schema version happen to move
together here only by coincidence.)

Write path — classification runs in encode_traces, with the classifier
context hoisted once per ResourceSpans and once per ScopeSpans on every path,
including the attribute-remapping one: the contexts borrow the resource and the
scope, which mapping rules never touch, and only the rewritten attribute list is
per span. It runs after attribute remapping, so an org that remaps a custom key
onto a rule key classifies by the shape the row actually stores.

Flag — INGEST_AI_CLASSIFICATION_ENABLED, default off, read once per batch and
never per span. It exists to ramp the migration window and is deleted once
classification is unconditional in production. There is no full-clock-hour
condition and no ordering against MV creation.

AiRollupHour is written on every span, flag on or off. It is
toStartOfHour(span start) when the start time is within [receive − 7d, receive + 1d], else toStartOfHour(receive time). Client timestamps are attacker-
and replay-controlled and this column becomes a partition key downstream, so an
unclamped value means unbounded partition creation and rows whose TTL never fires.
Clamping at write time is the only deterministic option: a view-side clamp needs
now(), which a later partition rebuild re-evaluates.

Observability is batch-level, not per span — per-span classification spans on
this hot path are what the self-observability rule forbids. The accept span records
maple.ingest.ai.enabled and maple.ingest.ai.spans_examined;
ingest_ai_spans_examined_total is labeled by signal exactly like native_rows,
so any divergence between the two series is a bug (a code path building rows
without classifying, or a partially-flagged fleet).

Verification

cargo test in apps/ingest: 143 lib + 70 binary, 0 failures, plus one
#[ignore]d fixture-regeneration helper.
bun typecheck green; @maple/domain 487 passed, @maple/cli 435 passed;
bun run clickhouse:schema:check (generator, insert-mapping, lint and the
append-only local-schema gate) green against origin/main.

• Adversarial fixture (ai_adversarial_fixtures.rs +
  fixtures/adversarial/adversarial-spans.jsonl) — hand-built hostile spans driven
  through the real row writer: typed and valueless AnyValues, present-but-empty
  values, duplicate keys, near-miss key spellings, astral-plane and NUL-bearing
  UTF-8, 64 KiB values that spill out of the inline attribute path, one span
  carrying six vendors' evidence at once, and the full session-state ladder per
  vendor. Per span it asserts the four columns the row writer emitted equal a
  direct classifier call, and that the hash equals city_hash64 over the winning
  key — so writer and classifier cannot drift apart silently. Reproducibility and
  branch coverage are asserted, which is what makes it a golden. It lives here
  rather than in 01 because it drives encode_traces, which does not exist until
  this PR.
• End-to-end through HTTP — ai_classification_flag_reaches_the_clickhouse_row
  drives a request through key resolution and AppConfig to the NDJSON body
  ClickHouse actually receives.
• Flag-off path — asserted to write zeros and a real rollup hour.
• Clamp — stale and future timestamps clamp to receive time; the rendered
  format matches ClickHouse's DateTime('UTC') wire shape.
• Hash contract e2e — cityHash64 on a real ClickHouse equals city_hash64 in
  Rust, over the adversarial fixture's raw winning session-key bytes, exercising
  the ≤32-byte and >64-byte length bands where CityHash 1.1 diverges from the 1.0.2
  variant ClickHouse vendors. Passed against a local server (4 tests). It lives in
  packages/domain, so every other ClickHouse CI step skipped it — this PR adds a
  step that runs it with --filter=@maple/domain, and widens the job's path filter
  to the CityHash port, the one input it otherwise could not see change.
• Physical schema probe — migration 0016 is storage-only, so the only thing
  that can prove it applied is the schema: column types, index types and
  granularities asserted against a real server, plus a row naming none of the new
  columns to prove the defaults read back. Runs inside the existing warehouse E2E
  suite (4 tests, passed).

Notes for review

• attr_map keeps last-wins for duplicate keys — the historical JSON-object
  behavior, for every key. The classifier's contract is first-occurrence-wins, and
  it holds on every path: the remapped path builds a parallel first-wins view
  (attr_map_first_wins, gated on an exact length comparison — attr_map only ever
  collapses duplicates, so equal lengths mean the two rules agree and the common
  path allocates nothing), and a two-way regression test pins that an unrelated
  mapping rule cannot change a span's verdict. That is a determinism rule for
  matching, not a storage rule. Residual caveat, documented at the call site: a
  future retro-fit re-reading written rows would see the last duplicate where the
  live classifier used the first.
• The five columns declare snake_case JSONPaths, not identity paths. The
  insert-mapping generator drops a column that has both a DEFAULT and an identity
  path, assuming the gateway never emits it — these are emitted on every span, so
  they must not match that shape. datasources.contract.test.ts now applies the
  generator's actual rule rather than its previous DEFAULT-only approximation.
• The local store runs no classifier, so four columns stay at their defaults;
  AiRollupHour is written for real through a port of the Rust clamp, with one
  receive time per batch so two spans in the same request cannot land on different
  anchors. The port is a hand-translation of rollup_hour_secs +
  format_datetime_secs, so encode.test.ts now asserts the same boundaries the
  Rust test does, at one-second resolution: exactly −7 d and exactly +1 d stay in
  window, one second past either edge clamps to the receive hour, a zero or
  unparseable timestamp clamps too, and the rendering is DateTime('UTC') rather
  than DateTime64(9). Without those, the existing key-set test proved the field
  was present, not that its value was right — and the value is a partition key.
• The v5 → v6 local migration runs migration 0016's exported ALTER list, not a
  copy of it. The two were previously duplicated under a "change these together"
  comment, which meant retuning idx_scope_name in 0016 alone left every migrated
  local store on the old index with nothing failing: v5 → v6's verify compares
  against the frozen v6 manifest, which records an index by name and not by its
  parameters. One definition, shared the same way SERVICE_AI_VENDORS_HOURLY_SELECT_SQL
  already is. The registry test still asserts v6 as a column/index delta against the
  frozen v5 manifest, so a stray table cannot ride along on the version bump.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}