Repository navigation
subscriber: do not mask a failed BEGIN with 'cannot rollback' (fleet #46912) - #40
echoomegaprime wants to merge 1 commit into
Conversation
…46912) echo-certforge-dispatcher flapped at start-up on 2026-10-19 with sqlite3.OperationalError: cannot rollback - no transaction is active. When BEGIN IMMEDIATE itself fails in _migrate_legacy_platform_tables (for example 'database is locked' once the busy timeout expires while another process holds the write lock), the except branch ran ROLLBACK with no transaction open. That raised a second error, hid the real cause, and crashed the worker. The handler now rolls back only when connection.in_transaction is set, so the original error propagates. Tests (red first): tests/test_subscriber_migration_rollback.py - a failed BEGIN surfaces 'database is locked', not 'cannot rollback' (reproduces the production error before the fix); - an error inside the transaction still rolls back and leaves the DB unchanged; - a needed legacy rename is still committed. Subscriber suites: 61 passed. The 2 test_deploy_environment_boots... tests fail identically on unpatched main (they need the deployment environment). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec
…ompensation Port PR #40 and PR #41 onto deployed d8ad9fd without unrelated main changes. Retain historical crash credits by tenant/run identity and use a separate run-compensation namespace for new credits. Add atomicity, concurrency, legacy-credit, original-period, collision-order, and migration rollback tests. Verified: 705 passed, 1 Docker-dependent skip; P1/P6/P7 acceptance green; 87 main integration tests passed; 7/7 isolated mutation checks killed. No deployment, production state repair, or live compensation authorized.
|
Integration acceptance update, November 10, 2026: independently applied PR #40 then PR #41 to isolated worktree at main |
|
Integration coordination (Nov 10, 2026): The exact-target combined draft is now PR #49: #49, commit |
Summary
Fixes Echo fleet item #46912: the
echo-certforge-dispatcher.servicerestart storm on 2026-10-19 (incident #49782, auto-resolved).Root cause. In
SubscriberGovernance._migrate_legacy_platform_tables, theexcept BaseExceptionhandler always ranROLLBACK. WhenBEGIN IMMEDIATEitself fails, there is no transaction open, soROLLBACKraisessqlite3.OperationalError: cannot rollback - no transaction is active. A failedBEGIN IMMEDIATEhappens, for example, when another process holds the write lock past the 10 s busy timeout and SQLite reportsdatabase is locked. That second error replaced the real one and crashed the dispatcher at start-up. This is the traceback in the incident.Fix. Roll back only when
connection.in_transactionis set, so the original error propagates and is the one logged.Tests
New
tests/test_subscriber_migration_rollback.py, written to fail first:BEGIN IMMEDIATEsurfacesdatabase is locked, notcannot rollback. Before the fix this test reproduces the production error exactly.With
tests/test_p7_subscriber_governance.pyas well: 61 passed. The 2test_deploy_environment_boots_and_authenticated_smoke_is_green[...]cases fail identically on unpatchedmain(58 passed + the same 2), because they need the deployment environment.Deployment
Production currently runs release
d8ad9fd(2026-09-26), which has the same code, so the defect is latent there. It only triggers when start-up loses the write-lock race. I did not hot-patch the certification authority. This change should ship throughdeploy/deploy_forge.shonce reviewed.🤖 Generated with Claude Code
https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec