Skip to content

fix(api,index): prevent /health threadpool starvation and coalesce SQLite rebuilds (ticket-285) - #348

Merged
ifuri-validator-agent[bot] merged 1 commit into
mainfrom
ticket/285-prevent-healthcheck-starvation
Nov 16, 2026
Merged

ifuri-validator-agent[bot] merged 1 commit into
mainfrom
ticket/285-prevent-healthcheck-starvation

Conversation

@tom-sapletta-com

Copy link
Copy Markdown
Contributor

Summary (ticket-285)

Fixes live /health liveness timeouts (>5s) and planfile_store_lock_timeout contention under --workers 2 multi-worker deployments:

  1. Event-loop /health handler: Convert GET /health in planfile/api/server.py from synchronous def to async def so FastAPI serves health probes directly on the asyncio event loop even when AnyIO's worker threadpool is saturated by concurrent API reads or SQLite maintenance.
  2. Non-blocking .rebuild.lock coalescing: Probe store.ticket_index_rebuild_lock(timeout_seconds=0.0) in _repair_ticket_index(), returning {"rebuilt": False, "coalesced": True} on contention, and coalesce (StoreLockTimeoutError, TicketIndexContentionError) in _maintain_ticket_index() so secondary workers yield in <1 ms instead of pinning threadpool workers for 10s and emitting spurious warnings.
  3. Bounded SQLite WAL retention: Enforce PRAGMA busy_timeout=0, PRAGMA journal_size_limit=67108864, and PRAGMA wal_checkpoint(TRUNCATE) post-commit in SQLiteTicketIndex.rebuild() without acquiring write locks inside _connect() on reader probes.

Verification

  • ruff check --no-cache passed.
  • pytest -q tests/test_index_rebuild_contention.py tests/test_index_repair.py tests/test_sqlite_ticket_index.py tests/test_health_check.py (119 passed).
  • ./project/governance-check.sh & wellman check passed.

@ifuri-validator-agent ifuri-validator-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validator approval after policy checks for exact head 06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d.

Ticket: ticket-285
Correlation ID: local-semcod-planfile-pr-348-ticket-285
Model: openai/cursor-auto
Reviewed diff chunks: 2
Advisory LLM verdict: APPROVE
Advisory summary: Reviewed all 2 diff chunk(s). The visible changes safely move the health handler off the AnyIO worker pool, coalesce concurrent index rebuild attempts with a non-blocking lock, and perform a best-effort post-commit WAL truncation without disrupting rebuild success. No code or security defects are evident in this chunk. | The visible test adds appropriate coverage confirming that a forced index rebuild completes promptly while an open SQLite read snapshot exists and that WAL retention remains bounded. No correctness or security defects are evident in this chunk.
Advisory findings: none
The LLM output above is advisory and was not used as the approval trust root.
Semantic review prerequisite: not_required; policy 676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7.

Actual PR impact radar

Exact range: 726921be567b99cec1a1587f3c24397d4723f5d0...06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d
Change digest: 0c321415eb72a3065f3a6f493b1212aae17c70e89794117e12dc0ffc685f0716
Score: 64/100 (L), estimated 68 min, split recommended: true
Affected services/components: repository-wide/unclassified

Machine-readable radar JSONL and SVG
{"actual_change":{"additions":220,"base_sha":"726921be567b99cec1a1587f3c24397d4723f5d0","binary_files":0,"categories":{"code":2,"configuration":1,"docs":1,"tests":1},"change_digest":"0c321415eb72a3065f3a6f493b1212aae17c70e89794117e12dc0ffc685f0716","comparison":"726921be567b99cec1a1587f3c24397d4723f5d0...06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d","deletions":5,"file_count":5,"files":["planfile/api/server.py","planfile/core/sqlite_index.py","project/ticket-285/README.md","project/ticket-285/intent.json","tests/test_index_rebuild_contention.py"],"head_sha":"06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d","service_count":0,"services":[]},"assessment_mode":"observed-pr","axes":{"coupling":4,"delivery":3,"scope":4,"uncertainty":3,"validation":2},"complexity":"L","confidence":0.9,"diagnostics":["RADAR-ACCEPTANCE-MISSING","RADAR-BUDGET-EXCEEDED"],"estimate":{"budget_minutes":30,"minutes":68,"within_budget":false},"impact":{"components":["planfile","project","tests"],"files":["planfile/api/server.py","planfile/core/sqlite_index.py","project/governance-check.sh","project/ticket-285/README.md","project/ticket-285/intent.json","tests/test_health_check.py","tests/test_index_rebuild_contention.py","tests/test_index_repair.py","tests/test_sqlite_ticket_index.py"],"public_interfaces":["planfile/api/server.py"],"runtime_dependencies":0},"schema":"subactor.ticket-radar/v1","score":64,"split":{"parts":[{"estimated_minutes":20,"name":"Define contract and acceptance boundary","scope":["planfile/api/server.py"]},{"estimated_minutes":13,"name":"Implement planfile","scope":["planfile"]},{"estimated_minutes":13,"name":"Implement project","scope":["project"]},{"estimated_minutes":13,"name":"Implement tests","scope":["tests"]},{"estimated_minutes":15,"name":"Validate and project to trackers","scope":["tests","planfile","github/gitlab/jira projections"]}],"reason":"estimated_minutes_exceed_budget","recommended":true},"standards":[{"id":"wellmanifest/dsl","revision":"6c60fc4e0dd1f1bb74f46a7745e28019908d1203","version":"0.1.0-dev"},{"id":"wellmanifest/ticket-lifecycle","revision":"5bf581907a87b46a13a73e6c033d3abe4d9a306f","version":"0.1.0-dev"},{"id":"wellmanifest/git-lifecycle","revision":"7d77d4b7af57e69bc75c3a0290b3a4805c5c4438","version":"0.2.0-dev"},{"id":"wellmanifest/logs","revision":"48c284ef7a069055c0bcb6b900147ce5e65f8b43","version":"0.3.0"}],"ticket_ref":"ticket-285"}
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img"><title>ticket-285: fix(api,index): prevent /health threadpool starvation and coalesce SQLite rebuilds (ticket-285)</title><rect width="128" height="128" rx="12" fill="#f8fafc"/><g stroke-width="1"><polygon points="64,55 72,61 69,71 59,71 56,61" fill="none" stroke="#d7dde5"/><polygon points="64,47 80,59 74,78 54,78 48,59" fill="none" stroke="#d7dde5"/><polygon points="64,38 89,56 79,85 49,85 39,56" fill="none" stroke="#d7dde5"/><polygon points="64,30 97,53 84,92 44,92 31,53" fill="none" stroke="#d7dde5"/><polygon points="64,21 105,51 89,99 39,99 23,51" fill="none" stroke="#d7dde5"/><line x1="64" y1="64" x2="64" y2="21" stroke="#aab4c0"/><line x1="64" y1="64" x2="105" y2="51" stroke="#aab4c0"/><line x1="64" y1="64" x2="89" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="39" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="23" y2="51" stroke="#aab4c0"/></g><polygon points="64,30 97,53 79,85 54,78 39,56" fill="#fb923c" fill-opacity="0.45" stroke="#c2410c" stroke-width="2"/><circle cx="64" cy="64" r="3" fill="#c2410c"/><g font-family="sans-serif" font-size="7" fill="#334155"><text x="64" y="11" text-anchor="middle">SCO</text><text x="114" y="48" text-anchor="middle">COU</text><text x="95" y="107" text-anchor="middle">UNC</text><text x="33" y="107" text-anchor="middle">VAL</text><text x="14" y="48" text-anchor="middle">DEL</text></g><text x="64" y="124" text-anchor="middle" font-family="sans-serif" font-size="8" fill="#0f172a">L · 68m</text></svg>
Merge will be attempted after this approval when explicitly authorized. ## Decision record (recomputable)
DECISION D-285-8643
TICKET ticket-285
HEAD_SHA 06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d
CORRELATION_ID local-semcod-planfile-pr-348-ticket-285
ACTOR agent:ifuri-validator-agent[bot]
APPLIED_RULE P-CORE-015
INPUT author_login = "tom-sapletta-com"
INPUT observed_checks = ["notify=PASS","governance / remote lifecycle=PASS","governance / enforce=PASS","ci-loop=PASS","test (3.10)=PASS","freshness=PASS","test (3.13)=PASS","onedev/local-verify=PASS"]
INPUT required_checks = ["ci-loop","notify","freshness","governance / remote lifecycle","governance / enforce","test (3.10)","test (3.13)","onedev/local-verify"]
INPUT required_checks_source = "protected registry + GitHub applied rules (env/request)"
INPUT reviewer_login = "ifuri-validator-agent[bot]"
INPUT semantic_review_assessment = {"schema":"subactor.validator/semantic-review-assessment/v1","subject":{"repository":"semcod/planfile","pull_request":348,"head_sha":"06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d","base_sha":"726921be567b99cec1a1587f3c24397d4723f5d0","diff_sha256":"fa13ce333afb86bf9b560b76a188722980cb7e27a9b9b5b291b83970c32c2a08"},"policy":{"policy_schema":"subactor.validator/semantic-review-policy/v1","policy_version":1,"policy_sha256":"676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7","required":false,"critical_paths":[],"observed_paths":["planfile/api/server.py","planfile/core/sqlite_index.py","project/ticket-285/README.md","project/ticket-285/intent.json","tests/test_index_rebuild_contention.py"]},"grounding":"full-diff-not-per-finding-proof","execution_authority":false,"status":"not_required","reason":null,"review_sha256":null,"unresolved":[]}
INPUT superseded_checks = []
INPUT ticket_radar_receipt = {"schema":"subactor.ticket-radar/v1","base_sha":"726921be567b99cec1a1587f3c24397d4723f5d0","head_sha":"06d5c13e98e22e5854f4cf5dd4b8f7b1a180419d","change_digest":"0c321415eb72a3065f3a6f493b1212aae17c70e89794117e12dc0ffc685f0716","score":64,"complexity":"L","estimated_minutes":68,"split_recommended":true,"services":[],"authority":"ADVISORY","promotion":"FORBIDDEN"}
VERDICT APPROVE AUTHORITY DETERMINISTIC
REJECTED REQUEST_CHANGES BECAUSE NO_UNSAFE_CHANGE_REASON_FOUND
ADVISORY llm_verdict = "APPROVE" MODEL "openai/cursor-auto"
ASSERT VERDICT_AUTHORITY != "ADVISORY"

@ifuri-validator-agent
ifuri-validator-agent Bot merged commit 123d522 into main Nov 16, 2026
8 checks passed
@ifuri-validator-agent
ifuri-validator-agent Bot deleted the ticket/285-prevent-healthcheck-starvation branch November 16, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant