compute: delay retracting a dropped export's lifecycle rows

[antiguru]

Stacked on #38403. Base is claude/hydration-visibility-compute-js1ycm, so the diff here is just the two commits on top. Review #38403 first.

Motivation

The lifecycle log as it stands retracts an export's rows the moment it is dropped, so an object's history vanishes with the object. That loses exactly the episodes worth looking at: a dataflow dropped before it hydrated, or one whose hydration is the reason someone opened the log. It also means a short-lived dataflow can come and go inside one introspection interval and leave no trace at all.

Part of CPU-226.

Description

Delay the retraction, and record the drop as a stage of its own.

Two delays, because the two populations churn at rates too different to share one value.

flag	default	population
compute_lifecycle_retraction_delay	5 min	user objects, dropped by DDL
compute_lifecycle_retraction_delay_transient	5 s	transient exports, one per peek and per subscribe

A transient export's rate is the query rate. At eight workers and two hundred peeks per second, a five minute window holds around 1.4 million rows, hundreds of megabytes, which is not a reasonable thing to hold in a replica's memory for introspection. A few seconds costs single-digit megabytes and is enough for a subscribe to observe them. A user object is dropped by DDL, so a thousand drops inside a five minute window is under ten megabytes, and there the history is the thing worth keeping. So the split is required rather than a nicety.

Both are floored at the logging interval in code, not by convention. The demux rounds update timestamps up to that interval, so a delay shorter than one interval can round to the same timestamp as the insertion, and the rows are then never separately visible, which defeats the point.

The dropped stage is what makes the delay readable. Without it a lingering row says only that an export reached some stage, not whether it still exists, so "hydrated but never written" could not be told apart from "dropped before it wrote". With it the object's last event names its fate, the delay is pure retention, and a row whose export_id no longer appears in mz_objects is explained by its own dropped event rather than reading as a leak. It is a per-worker event, since each worker's demux handles the drop of its own export.

Replica scoping. The delays are read per batch inside the demux rather than at construction, following prometheus.rs. ComputeCommand::UpdateConfiguration applies updates in place to the same ConfigSet the logging dataflow holds, so a change takes effect without recreating the dataflow, and CreateInstance carries an initial_config snapshot so a fresh replica starts with the right values. specialize_command_for_replica merges per-replica ConfigUpdates into both, with no per-flag allowlist, so a replica under investigation can be told to retain longer than the rest without any additional plumbing.

The second commit is unrelated to the delay and corrects a claim I put in the design doc in #38403: that catalog_server_explain.slt needs no change when a builtin log is added. The o.id NOT LIKE 'si%' filter is real, so no new EXPLAIN entry appears, but the existing plans embed the inlined builtin VALUES sets as Constant (N rows) nodes, so every count over a catalog relation that gained a row moves. #38403 has the golden fix; this records the question that catches it.

Verification

cargo check -p mz-compute -p mz-compute-types --all-targets is clean. bin/fmt --check passes for rustfmt, black and ruff (buf is not installed in this environment, and there are no proto changes).

test/testdrive/compute-lifecycle-events.td gains coverage for the new behaviour: it sets both delays to '2s' via ALTER SYSTEM SET rather than disabling them, so the delayed path is the one under test, asserts a dropped event appears, and then asserts the rows are retracted. dropped is added to the closed event vocabulary asserted under set-max-tries max-tries=1.

One thing CI is the first to prove. The testdrive file sets the two delays through ALTER SYSTEM SET, which assumes dyncfgs are settable that way. compute_temporal_bucketing_summary is a Config<Duration> registered in the same UNINTERESTING_SYSTEM_PARAMETERS list, so the shape matches a working precedent, but I have not executed this file. If it turns out not to work, the fix is to pass the values via --system-parameter-default instead.

Both flag names are registered in misc/python/materialize/mzcompose/__init__.py and misc/python/materialize/parallel_workload/action.py, without which check-test-flags fails.

---

Generated by Claude Code

[linear-code]

CPU-226