feat(ingest): classify AI spans on the write path and store the columns

[JeremyFunk]

What this layer contains

The schema change and the wiring that makes classification actually happen — the largest layer in the stack.

• Schema: five defaulted trailing columns on traces (AiVendor, AiSessionKeyState, AiSessionKeyHash, AiRulesVersion, AiRollupHour) plus the idx_ai_vendor and idx_scope_name skip indexes, as ClickHouse migration 0015 and local-store migration v4 → v5 (local schema version 5, frozen local-schema-v5.sql, history entry, manifest check).
• Write path: the classifier from feat(ingest): add the pure AI span classifier and registry loader #432 wired into apps/ingest's row writer (telemetry.rs), metrics and OTLP spans, and into the local-mode encoder in apps/cli.
• Regenerated artifacts: clickhouse-schema.ts, tinybird-project-manifest.ts, local-schema.sql, local-inserts.json and clickhouse_insert_mappings.rs — all produced by bun run clickhouse:schema + bun run tinybird:manifest, never hand-edited.
• Verification: the AI-schema assertions added to WarehouseQueryService.clickhouse.e2e.test.ts, and the datasource contract test.

Why it is shaped this way

Every new column is DEFAULT-valued, which makes the ALTER metadata-only: no MATERIALIZE, no OPTIMIZE, no whole-table mutation on traces, whose 30-day TTL retires unindexed parts on its own. That also keeps rows written before the classifier shipped readable, with AiRulesVersion = 0 meaning "never examined" as distinct from "examined and found not to be AI". AiRollupHour is receive-time-clamped and written unconditionally for every span, classification on or off, so a skewed client cannot open a partition in 2038.

⚠️ Operational flag: BYO-ClickHouse routing

0015 is requiredForIngest: true, so clickHouseSchemaVersion moves from 13 to 15. The gateway's INSERT now names all five AI columns, and a BYO cluster without them would reject every direct insert. Orgs stamped below 15 therefore resolve clickhouse_ready = false and route to the managed pipeline until their schema syncs 0015. That fallback is the designed behaviour rather than a regression, but it is the reason this PR should not merge ahead of the schema rollout.

Gates run at this level

• bun typecheck — 37/37 tasks pass.
• bun run clickhouse:schema:check — all five generators up to date, lint OK (37 tables, 39 MVs), local manifest up to date at schema v5.
• packages/domain vitest — 533 passed. apps/cli vitest — 430 passed.
• cargo test --locked (+ TRACE_CAPTURE_DIR) — 134 lib + 75 bin, 0 failures.
• bun run ch:test against a real ClickHouse — 3 files, 182 passed.

Dependencies

Requires #432 (the classifier) and #433 (the registry TS layer).

Partition note: CURRENT_SCHEMA_PROJECT_REVISION is pinned to the recorded v5 identity (09513d18…) so it matches the append-only history tip, which check-local-schema-manifest.ts enforces. The generated artifacts at this level carry 062342f1…: the recorded v5 revision was cut when the AI columns had no jsonPaths, and jsonPaths move the Tinybird-wide project revision without changing any DDL. Confirmed by reproducing 09513d18… exactly with the jsonPaths removed. The two converge again at the top of the stack.

🤖 Generated with Claude Code

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[JeremyFunk]

Superseded by the v2 stack: #451 → #452 → #453. The registry-artifact + SQL-equivalence design this stack implemented was dropped in the write-side plan's v2 revision; vendor rules are now plain Rust with the vendored classification fixture as the CI gate.

[github-actions]

🍁 Maple PR preview

Note

Preview resources were removed when this pull request closed.

Final commit 19aa5a1 · View workflow run