Repository navigation
Fix upload contention, copy filing codes from the EFSP database, and add guided code search - #282
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical and moderate correctness, data consistency, and deployment issues remain.
Review effort: Lite
Findings: 1
Open (2)
What changed in this PR
This pull request adds indexed filing-code discovery, guided path selection, and improved SQLite contention handling.
Changes:
- Adds resumable catalog indexing, grouped search, facets, and path validation.
- Integrates accessible filing-code search into case confirmation.
- Adds migrations, workers, deployment documentation, and expanded tests.
Review findings:
- Critical, 1 vote:
filing_code_search.pycan raiseKeyErrordue to missing jurisdiction fallback. - Moderate: unresolved retry coverage, stale checkpoints, misleading guidance sources, stale classification fields, incomplete search explanations, and Fly worker provisioning.
- Nit: jurisdiction documentation does not match the current fallback UI.
| File | Description |
|---|---|
testing/README.md |
Documents local index-worker usage |
fly.toml |
Adds the production index-worker process |
efile_app/uv.lock |
Locks the Snowball dependency |
efile_app/tests/confirm-case-editing.spec.js |
Adds browser and accessibility coverage |
efile_app/pytest.ini |
Enables migrations during tests |
efile_app/pyproject.toml |
Updates dependencies and test configuration |
efile_app/efile/views/extraction_review.py |
Applies classification resets |
efile_app/efile/tests/test_sqlite_transactions.py |
Tests SQLite contention handling |
efile_app/efile/tests/test_filing_code_search.py |
Tests indexing and search |
efile_app/efile/tests/test_case_type_guidance.py |
Tests guidance and facets |
efile_app/efile/templates/efile/extraction_review.html |
Integrates search assets |
efile_app/efile/templates/efile/components/filing_code_search.html |
Adds search dialog markup |
efile_app/efile/static/js/filing-code-search.js |
Implements guided search interactions |
efile_app/efile/static/js/extraction-review.js |
Applies validated paths |
efile_app/efile/static/js/court-selector.js |
Handles court selection |
efile_app/efile/static/css/components/filing-code-search.css |
Styles the search interface |
efile_app/efile/settings_dev.py |
Configures SQLite transactions |
efile_app/efile/services/filing_path.py |
Resets changed-path dependencies |
efile_app/efile/services/filing_code_search.py |
Builds, indexes, searches, and validates catalogs |
efile_app/efile/services/case_type_guidance.py |
Provides contextual guidance |
efile_app/efile/models.py |
Adds catalog and path models |
efile_app/efile/migrations/0034_sqlite_filing_search_index.py |
Adds SQLite FTS5 support |
efile_app/efile/migrations/0033_filing_code_search.py |
Adds catalog schema and PostgreSQL indexes |
efile_app/efile/management/commands/refresh_filing_code_index.py |
Runs catalog refreshes |
efile_app/efile/management/commands/process_document_extractions.py |
Retries SQLite lock failures |
efile_app/efile/data/filing_concept_definitions.yaml |
Defines searchable explanations |
efile_app/efile/data/filing_code_search.yaml |
Defines search terms and rules |
efile_app/efile/data/filing_code_facets.yaml |
Defines filing facets |
efile_app/efile/data/case_type_guidance.yaml |
Defines case-family guidance |
efile_app/efile/api/urls.py |
Registers the search endpoint |
efile_app/efile/api/filing_code_search.py |
Implements search and validation APIs |
efile_app/efile/api/dropdown_views.py |
Supports new/existing case timing |
docs/docs/partners-courts/jurisdiction-config.md |
Documents search configuration |
docs/docs/admin/deployment.md |
Documents deployment and workers |
compose.yml |
Adds the local index worker |
.github/workflows/accessibility.yml |
Runs expanded accessibility checks |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| config, _ = rules() | ||
| phrases = [] | ||
| for name, concept in config["concepts"].items(): | ||
| if jurisdiction not in concept.get("jurisdictions", config["jurisdictions"]): |
| draft.previous_case_id = "" | ||
| draft.case_title = "" | ||
| draft.optional_services = [] | ||
| invalidate_fee_quote(draft, save=False) |
Stage and validate per-court bulk exports, then replace only changed paths using PostgreSQL COPY or bounded SQLite batches. Recheck the manifest before downloads or removals are published, preserving the previous index on failure. Add dry runs, forced court imports, and an index for court-scoped replacements. Keep Fly on the existing crawler until the proxy rollout, and start the code index worker from the local launcher.
Filing code search becomes three steps: what you're filing, court and case type, then a check of the full path. Step 1 pairs the search with a case ZIP and short chip questions (what you want to do, kind of case, case category, claim amount, case details), shows strong name matches first, collapses jury/government/fee code variants, mutes shared name prefixes, explains court terms from a per-state glossary, and links to matches in the other case stage. Step 2 asks only the jury, lawyer, amount, and government questions a court's case types differ on. ZIP narrowing works in every state: MACourts and VTCourts where configured, otherwise a Census ZIP-to-county table, keeping courts nothing can place. Shortcuts offer the filer's own ZIP and their three most recent case ZIPs. Search rules add sealing synonyms and fix personal-injury, malpractice, and fee waiver classification. Filing codes are now copied from the EFSP proxy's codes database over a read-only role (litefile_codes_reader: SELECT on five code tables, read only at the server), in one READ ONLY, REPEATABLE READ snapshot, into per-court copies the index is built from. The code index worker syncs daily after the proxy's Tyler update (21:30 Eastern on staging), and superusers can queue "Resync codes" and "Rebuild code search indexes" from the staff tools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nonprofittechy
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uploads could fail with a SQLite write-lock error while a background worker was active, and filing-code discovery made filers work through repeated county codes and near-identical case types. This change fixes local upload contention, copies filing codes from the EFSP's own database on a schedule, and replaces the code lookup with a three-step search that ends in a validated court/category/case-type/filing-type path.
Changes
Upload contention
Filing codes from the EFSP database
FILING_CODE_SYNC_MODE=database). The queries match the proxy's own filing-catalog export. The court list and every court's tables are read in oneREAD ONLY, REPEATABLE READsnapshot. Each court is saved as a local copy (FilingCodeCourtCatalog), and the search index is built from those copies.litefile_codes_reader, a role created for LITEFile on the test EFSP database. It hasSELECTon the five code tables only, because the proxy keeps user data in the same database. It is read only at the server through a role setting, which holds even through Supabase's pooler; that pooler ignores connection-level options. LITEFile holds no write credentials.FILING_CODE_SYNC_TIME(FILING_CODE_SYNC_TIMEZONE), after the proxy's own Tyler update: 21:30 Eastern on staging, whose test EFSP updates at 19:35. Failed states retry every 15 minutes, and the worker syncs immediately when any jurisdiction has no current index.FilingCodeJobs that the worker runs, and each request is audited.bulk) and the full crawler (legacy) as fallbacks. Bulk imports still use PostgreSQL COPY and indexed court replacements.Three-step filing code search
Filing-name matches come first, and case-type-only matches sit behind "Show N more". Codes split only by jury, government-filer, or fee qualifiers appear once. Shared name prefixes are muted. A link points to matches in the other case stage ("10 results in 'existing case' filings").
zip_counties.json.gz, rebuilt byscripts/build_zip_counties.py). Courts nothing can place stay visible.legal_glossary.yaml).Deployment and limits
0033–0037.EFSP_CODES_DATABASE_URL(the read-only role) is staged onlitefile-stagingand takes effect with this deploy. Production's EFSP database needs the same role; the SQL is in the deployment guide. Production's sync time should be about 04:30 Eastern.code_index_workermachine.SELECTmust be granted on it again. Postgres lets every role create session-private temporary tables. Those can't touch shared data, and the role's read-only default refuses them.Validation
litefile_codes_readerfor Illinois, Massachusetts, and Vermont. The role reads only the five code tables and is refused forDELETE,UPDATE, andCREATE TABLEeven with read-only switched off. "Seal eviction" (existing case, ZIP 60603) now finds four sealing and redaction filings. The earlier bulk-import checks on PostgreSQL still stand; the database copy path has run on SQLite only.🤖 Generated with Claude Code