Add LEDGER_START_PAUSED so a migration can be deployed stopped - #8
Merged
Conversation
Migrations get deployed ahead of the customer's go-ahead, but the engine
started as soon as a pod became ready, and every pause check sat after the
mapping pass — so even an immediate pause let a pod build the {cd:1,_id:1}
index on the source and cut the chunk grid first.
With LEDGER_START_PAUSED=true a pod comes up, serves the dashboard and holds
before mapping: nothing is read, mapped or indexed until an operator presses
Start (POST /control/resume). Preflight, index builds and the dry run stay
available from the dashboard while it waits.
The gate lives in the ledger (mig_run_config.start_gate_open) rather than in
one pod's memory, so one Start releases every pod rather than only the one
serving the request, pods that join later start immediately, and a pod that
restarts after Start comes back running instead of held. It is keyed by
effective run id, so opening a dry run's gate does not pre-authorise the
real run.
Also honours a pause at the top of the map loop, so a pause issued during a
long mapping or top-up pass is observed there rather than at the next claim.
The dashboard labels the held state NOT STARTED and offers Start instead of
Resume.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Migrations are deployed before the customer's go-ahead, but the engine begins as soon as a pod is ready. Pausing right after boot does not help: every pause check sits after the mapping pass, so a paused pod still builds the
{cd:1,_id:1}index on the source and cuts the chunk grid.replicas: 0avoids that but leaves no dashboard to run preflight from.What this adds
LEDGER_START_PAUSED=true— a pod comes up, serves the dashboard and holds before mapping. Nothing is read, mapped or indexed until an operator presses Start (POST /control/resume). Preflight, index builds and the dry run remain available from the dashboard while it waits.The gate lives in the ledger (
mig_run_config.start_gate_open), not in one pod's memory:<runId>-dry) does not pre-authorise the real run — which matters because the Helm chart installs in dry-run mode by default.Two smaller changes that come with it:
NOT STARTEDand shows Start instead of Resume, with a hint that nothing has been touched yet.Default is
false, so behaviour is unchanged unless the flag is set.Files
src/config/schema.ts,src/config/loader.tsledger.startPaused/LEDGER_START_PAUSEDsrc/state/ledger-store.tsisStartGateOpen()/openStartGate()onmig_run_configsrc/runtime/chunk-orchestrator.tsnot-startedpause reasonsrc/runtime/ledger-engine.ts/control/resumeopens the gatesrc/http/ledger-viz-route.ts.env.example,k8s/migration.yamlTesting
tsc --noEmitpasses. The integration suite was not run here: it points at an unauthenticatedmongodb://localhost:27017, and the environment I had available was a remote cluster with SCRAM auth.Not yet exercised end to end — the intended check is: deploy with the flag on, confirm the pods report
NOT STARTEDand thatmig_rangesstays empty with no index build on the source, then press Start once and confirm all pods begin. Worth doing before this is relied on for a customer.