Characterize scheduled cleanup across shard ownership changes

[taylan-oai]

Summary

Characterize scheduled cleanup that deletes a task written by a successor shard owner, so the cleanup contract can be reviewed with a repeatable test.

Problem

Scheduled cleanup selects a time range without the originating shard's ownership version. An owner can authorize cleanup through a boundary that exists only in memory. After ownership changes, the successor restores its scheduling floor from durable metadata and can assign a timer below that boundary before its first reader poll. A delayed cleanup from the predecessor then removes the new task even though ordinary writes from the predecessor are rejected.

The test acquires two real shard ownership ranges and uses the normal task-key assignment and persistence APIs. A timer written by the successor is visible before the predecessor's cleanup and absent afterward. Two controls retain the task: making the predecessor's reader frontier durable before transfer, or polling the successor's queue boundary before assigning the task.

Approach

Add comments at scheduled range cleanup and batched queue-state persistence, plus a passing SQLite-backed characterization test. It records the unsafe outcome explicitly; a future fix must preserve the successor's task and keep it reachable to queue readers.

Two possible follow-ups, neither implemented nor validated here:

1. Make the authorized cleanup frontier durable before deletion. Require a completed, ownership-fenced shard update before issuing cleanup through that frontier. The successor can then restore a scheduling floor beyond the predecessor's cleanup range. Calling the current SetQueueState earlier is insufficient because it can return after an in-memory update. This option needs checks for write failures, batching cost, reader holes and the successor's restored read positions.
2. Bound scheduled deletion by the originating owner's task-ID allocation range as well as time. Successor task IDs belong to a later range. Carrying the predecessor's upper ID bound through cleanup could preserve those rows. Every store must enforce that predicate correctly, including before page limits and across retries; query cost and non-wrapping ID assumptions need validation.

A separate ownership read immediately before deletion leaves a check/write race. Advancing the successor's local time floor earlier narrows startup exposure but does not establish the predecessor's maximum authorized boundary.

Validation

• The three-case characterization passed with native in-memory SQLite and separately with Cassandra 5.0.9 using a local test overlay.
• The SQLite characterization passed with the race detector in three repeated runs.
• Full shard-package tests passed (go test -p 4 -tags test_dep ./service/history/shard -count=1 -timeout=5m).
• Native changed-package lint passed with the repository configuration and fixes disabled.

The committed test needs no external datastore. It runs real shard acquisition, task-key assignment and persistence operations, with mocked engine callbacks controlling publication and polling. It does not run a complete History engine or reproduce end-to-end workflow loss. The Cassandra overlay is local corroboration and is not included in the diff.

Risks, rollout, and scope

This is a problem-scoping draft containing comments and tests. The passing assertions describe the defect and do not certify either repair option. No production frequency, replica-failure behavior or complete recovery protocol is established by this test.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}