Repository navigation
bug(coroot): bound metadata database connections during recovery #4652
Description
🤖 Generated by the Agentic Engineer
Coroot reports PostgreSQL connection exhaustion during database recovery, while a second server does not become ready. The released server and current public upstream constructor leave database/sql with an unlimited open-connection pool. That permits concurrent work to consume the database connection budget instead of applying client-side backpressure.
Investigate and reproduce the pool behavior, then bound it without weakening readiness or increasing database capacity to mask unlimited growth. Confirm recovery and useful application reads after deployment; a healthy database alone is not sufficient evidence.
Upstream necessity: the pool is constructed in Coroot itself, outside platform configuration. Raising PostgreSQL limits merely moves the exhaustion threshold, and inserting another stateful session-pooling service is not a sufficient minimal in-portfolio repair. A small tested Coroot contribution is the appropriate owner-level fix. Public upstream research is scoped to this dependency and must follow its contribution and AI policies. No private operational evidence belongs in that contribution.
Acceptance criteria:
- A regression demonstrates bounded simultaneous database connections.
- Existing connection timeout and SQLite behavior are preserved.
- The normal signed publication and reviewed deployment adopt the fix.
- Post-deployment readback proves recovery; it does not merely hide errors.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
devantler commentedon Oct 8, 2026
The bounded upstream repair is now published as coroot/coroot#1046, following the target contribution conventions. It limits each PostgreSQL pool to 25 open connections, preserves SQLite behavior, and adds constructor regression tests plus HA budget documentation. RED-to-GREEN, the database race suite, the complete upstream test and lint suites, and a read-only-module build passed.
This is not yet a production fix: upstream review, publication, and protected platform adoption remain open. Fresh OIDC reads still show connection-exhaustion errors and one unready Coroot server. The init-log deployment failed on protected drains; its automatic recovery then stopped at the unready Coroot baseline check. No readiness, disruption, or recovery gate has been weakened.
devantler commentedon Oct 8, 2026
A related source-level defect is also confirmed: GetCheckConfigs returns on malformed stored JSON without closing its query rows. That can retain a pooled connection on each failed read. I am preparing a separate minimal regression-tested upstream cleanup fix, so it remains independently reviewable from the pool limit.
This is a proven cleanup defect, not yet proof that malformed configuration caused the current production outage; the available OIDC logs do not establish that link. The fix belongs in Coroot and has no equivalent platform configuration change.
devantler commentedon Oct 8, 2026
The separate query-cleanup repair is published as coroot/coroot#1048. Its real SQLite regression reproduced one retained connection on malformed JSON before the fix and zero afterward. The database race suite, complete tests, complete lint, and read-only-module build all pass. The implementation is one deferred close; there are no schema, timeout, permission, or configuration changes.
Both upstream proposals still need maintainer review/publication and protected platform adoption. Production recovery and clean-state verification remain open.
devantler commentedon Oct 8, 2026
Status 2026-10-08 15:00Z — this now blocks every production deploy.
No gate was changed. The check has a designed exit (a reverted namespace label skips it); I offered that to the maintainer as the slower alternative rather than taking it unattended, because it lowers a security setting between two deploys.
Blocker: restart of the Coroot server holding the metadata database connections | authority | last-verified 2026-10-08: one server unready, deploys refused at the Coroot baseline check | asked slack 2026-10-08
devantler commentedon Oct 8, 2026
Still unrecovered at 14:55 UTC, and it now blocks all delivery. A fresh read-only check shows one Coroot server unready since 00:25 UTC and the other reporting connection-slot errors continuously; the database cluster itself is healthy. Because the pre-publish Coroot check requires all six workloads fully ready, every production deploy since 01:52 UTC has been refused (27 in a row), the recovery deploy included. #4653 was removed from the merge queue by it at 02:53 UTC.
The upstream pool fixes remain the lasting repair, but they cannot unblock production today. Restarting the Coroot servers so they release their connections is a production action outside the deploy pipeline, so it has been asked of the maintainer (Slack, 8 October 14:58 UTC: https://devantlertech.slack.com/archives/D0B1JG2EWLW/p1791471491624039). The blast radius is filed separately as #4658.
devantler commentedon Oct 8, 2026
Why production is still blocked at 15:00Z on 2026-10-08, and the one step that unblocks it
Read from production, read-only, at 14:57–15:00Z.
Every production deploy since 00:20Z fails at the same step, the Coroot baseline check before publication: a Coroot template is not operator-owned or not fully ready. That includes the restore-main deploy and the queued OpenBao repair (#4653, evicted at 02:53Z). Today's queue has cycled dependency updates through that failure about every ten minutes.
Exactly one workload fails that check. Of the six Coroot-owned workloads, five are fully ready. The Coroot server set has one of two replicas ready.
The unready replica is not crash-looping; it is hung. It exited nine times while the database was recovering (the exhaustion this issue describes), started a tenth time at 00:25Z, logged that its database type is postgres, and has logged nothing since. Its web server never started: the readiness probe has been refused about 5,600 times. It has no liveness probe, so nothing will ever restart it. The database has been healthy on three of three instances since about 00:23Z, and the other server replica has stayed ready throughout.
So the pool and cleanup fixes proposed upstream are worth having, but they do not unblock production. A hung process with no liveness probe stays hung whatever the next release does. What unblocks production is restarting that one replica. That is a change to the live cluster, so it is the maintainer's step; it has been put to him in an interactive session with the exact command.
After the restart, when the server set reads two of two ready, the baseline check passes again, the OpenBao repair can go back through the queue, and the secrets store returns to three servers.
Two gaps this exposes, to be filed separately: the Coroot server has no liveness or startup probe, so a start-up hang is permanent; and a failure in the observability stack blocks every deploy including the restore deploy and repairs, with no reviewed way through.