feat(domain): hourly per-service AI vendor discovery rollup

[JeremyFunk]

Stack position: 3 of 3 (final). Base: ai2/02-ingest-write-path. Review 01 and
02 first — this PR aggregates the columns 02 writes.

Adds service_ai_vendors_hourly: one AggregatingMergeTree row per
(org, service, vendor, hour), populated by a materialized view over the
classified spans landed in 02. It is what makes "which AI frameworks does this
service use, and how well can we resolve its sessions" answerable without scanning
raw spans.

Design rationale is in the write-side plan (§5 and the rollup sections); this
description covers the shape, the reader contract and the verification.

What lands

• ClickHouse migration 0017 + the Tinybird datasource and MV, sharing one
  exported SELECT constant.
• Local schema v7 and its v6 → v7 migration module.
• AI_VENDORS_ROLLUP_ENABLEMENT_HOUR_ENV and AI_VENDORS_ROLLUP_TABLE — the
  reserved enablement-boundary name, the table it governs, and the reader rule.
  Nothing consumes them yet; the read path does not exist yet.
• The rollup e2e suite, wired into the existing ClickHouse CI leg and ch:test.

(Numbering: main took migration 0015 and local schema v5 for the minutely
service overview while this stack was open, so 02's AI columns are 0016 / local
v6 and this PR is 0017 / local v7.)

Columns: span and sample-weighted span counts, the four session-key state
counters (absent / invalid / sub-session / session), uniqCombined(12) states for
traces total, traces with a key and approximate sessions, and rules-version
provenance for both the rows and the aggregate.

The decisions worth reviewing

• WHERE AiVendor != '' is the cost model and the semantics. The platform's
  HTTP and DB traffic never reaches the aggregate expressions or the HLL states, so
  cost scales with AI traffic rather than total traffic. It is also what makes
  "no rows for a service-hour" mean genuinely no AI spans post-enablement.
• Grouping is the stored AiRollupHour, not toStartOfHour(Timestamp). A
  clock-skewed client must not be able to open a partition in 2038, and a stored
  clamped hour survives a partition rebuild where a now()-based one would not.
• No POPULATE. Correctness depends on the source rows having been classified,
  not on the view having existed, so there is nothing safe to backfill. An earlier
  draft cited a runbook gate ("classifier at 100% for a full clock hour before MV
  creation"); that gate is retracted — the MV ships in the same migration chain as
  the columns and is a no-op until the ramp, so it cannot be ordered after it. The
  recorded enablement hour covers the boundary instead (plan §7 step 4).
• Daily partitions, OrgId first in the sorting key. Daily is the granularity a
  registry-fix rebuild works at — a per-closed-day, atomic
  ALTER TABLE … REPLACE PARTITION … FROM <shadow>, which monthly partitions would
  make unaffordable — and org deletion, the one sanctioned mutation on this table,
  prunes on the sort-key prefix.
• 400-day TTL against a 30-day source, and the retention matrix grows its first
  400-day tier to say so. The asymmetry is deliberate and comes with its
  constraint: past the raw horizon these rows cannot be rebuilt.
• The MV runs synchronously inside the INSERT pipeline. This target's part-count
  and merge-lag alerts are trace-ingestion health, not side-table health — a rollup
  target that hits too many parts fails the INSERT into traces.
• 0017 is not requiredForIngest. It adds a read-path rollup and touches
  nothing the gateway inserts, so the ingest gate stays at 16 rather than
  un-readying every BYO-CH org over a table their ingest path never writes.

Reader contract

Plain sum() / min() / max() with GROUP BY for the SimpleAggregateFunction
columns, uniqCombinedMerge(12) for the states. Parts are never guaranteed
merged — no reader may assume one row per key, and no reader may use FINAL.

Headline coverage is trace-level and service-level:
uniqCombinedMerge(12)(TracesWithKey) / uniqCombinedMerge(12)(TracesTotal) grouped
by (OrgId, ServiceName), merging across vendor rows. Per-vendor ratios are
diagnostic only — a trace mixing a passthrough vendor with the framework that
carries the key counts in both vendors' TracesTotal but only one's
TracesWithKey.

WeightedSpanCount is sum(SampleRate), not sum(1/SampleRate): Maple's
SampleRate is the adjusted-count convention. The two inverses are pinned in the
column doc because picking the wrong one still produces a plausible number.

Hours strictly before the enablement boundary are nonexistent, not zero. The
boundary is the hour INGEST_AI_CLASSIFICATION_ENABLED reaches 100% across the
fleet — not MV creation, which truncates an empty hour and costs nothing because
WHERE AiVendor != '' matches no unclassified row. That hour is partly classified
and looks perfectly healthy: every counter is internally consistent and nothing in
the row says it is partial. "No AI spans this hour" and "the rollup was not
recording this hour" are different claims and only the first may be shown to a
customer. Nothing mechanical enforces recording it; it is an operator step, and
every site that mentions the boundary says so rather than implying a mechanism.

Verification

bun typecheck, cargo test (143 lib + 70 binary), @maple/domain (490 passed),
@maple/cli (435 passed): green. bun run clickhouse:schema:check — including the
append-only local-schema history gate against origin/main — green at schema v7.

ClickHouse e2e against a real server — 12 tests passed. SQL-text tests cannot
catch any of what this suite is shaped for, because every failure mode still
renders a plausible percentage:

• the EligibleSpanCount = KeyAbsent + KeyInvalid + KeySubSession + KeySession
  identity;
• each key deliberately written across several INSERTs, so a reader that assumes
  merged parts reads low and FINAL is not an escape hatch;
• the two-vendor trace where merged coverage is 100% while the passthrough vendor's
  own row reads 0% on a fully resolvable trace;
• WeightedSpanCount's floor — a span with SampleRate 0 contributes 1.0, not 0
  and not an infinity;
• the grouping being the stored AiRollupHour — seeded with a clock-skewed span
  whose raw Timestamp is in 2038 and whose stored hour is inside the fixture, so a
  toStartOfHour(Timestamp) regression fails rather than reproducing the fixture.
  The divergence between that span's raw timestamp and its stored hour is asserted
  first, so a future seed edit that removes it fails loudly instead of turning the
  rest of the test back into a tautology;
• non-AI traffic never entering the rollup.

Also passed on this branch: the hash-contract suite (4) and the warehouse suite
carrying the 0016 schema probe (4).

Structural tests: migration and deployed MV bodies asserted to contain the
shared SELECT byte-for-byte (a migrated cluster and a bootstrapped one must compute
the same ratio); the SELECT's column order asserted against the target's as a
readability convention — ClickHouse matches an MV's output to its target by
name, and an alias with no matching column raises THERE_IS_NO_COLUMN, which,
because the view runs inside the INSERT pipeline, hard-fails every INSERT into
traces; no POPULATE; no backfill statements; v7 asserted as a pure name delta
against the frozen v6 manifest.

Note

A local store runs no classifier, so its rollup stays empty rather than reporting a
confident zero — which is the honest answer there.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}