diff --git a/.github/workflows/accessibility.yml b/.github/workflows/accessibility.yml index 94c598f0..70e7c336 100644 --- a/.github/workflows/accessibility.yml +++ b/.github/workflows/accessibility.yml @@ -59,17 +59,18 @@ jobs: curl --retry 30 --retry-connrefused --retry-delay 1 --fail http://127.0.0.1:8000/choose-jurisdiction npm run test:a11y - - name: Run Confirm case editing checks + - name: Run confirm case and filing search accessibility checks working-directory: efile_app env: A11Y_STORAGE_STATE: playwright/.auth/a11y.json # Same server and seeded session as the Axe checks; the court lists - # are mocked in the browser. - run: npm run test:confirm-case + # are mocked in the browser. Includes desktop/mobile Axe audits and + # keyboard checks for contextual filters, claim amounts, and path details. + run: npm run test:confirm-case -- --output=test-results/confirm-case - name: Run party role keyboard checks working-directory: efile_app - run: npx playwright test --config=playwright.parties.config.js + run: npx playwright test --config=playwright.parties.config.js --output=test-results/parties - name: Upload accessibility results if: always() diff --git a/README.md b/README.md index 29fb413e..78375350 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,29 @@ A minimal Django app for form submission and review. The Django project lives un - Python 3.10+ - uv - -### Using uv (recommended) +### Single-command local startup (`run_all.sh`) + +To quickly run the entire local development stack without building the full Docker container: + +```bash +./run_all.sh +``` + +This script: + +- Verifies Astral `uv` and synchronizes the local virtual environment (`efile_app/.venv`) +- Automatically initializes `efile_app/.env` from `.env.example` if needed +- Starts LocalStack (S3 mock) in Docker if Docker is running +- Applies Django database migrations +- Runs the Django development server (`http://127.0.0.1:8000`) and background extraction worker together +- Starts the filing code index worker, which copies changed courts' codes from the EFSP codes database once a day and indexes them +- Cleanly stops all child processes upon pressing `Ctrl+C` + +Filing code search becomes available after the first successful sync, which needs `EFSP_CODES_DATABASE_URL` in `efile_app/.env` (the test EFSP's codes database, read only). `--legacy-code-crawl` explicitly enables the older full crawler instead. + +Run `./run_all.sh --help` to see additional options (e.g., custom ports, running without Docker, or `--no-code-index` to skip the court code index worker). + +### Using uv manually - __0) Install uv__ (one-time) - macOS (Homebrew): diff --git a/compose.yml b/compose.yml index 743cbd3f..07d64aa2 100644 --- a/compose.yml +++ b/compose.yml @@ -87,6 +87,17 @@ services: volumes: *litefile-volumes + code_index_worker: + image: litefile:web + command: uv run python manage.py refresh_filing_code_index --daily --retry-interval 900 --verbosity 2 --cache-dir /data/filing-code-cache + depends_on: + web: + condition: service_healthy + environment: + <<: *litefile-environment + volumes: + *litefile-volumes + volumes: localstack: driver: local diff --git a/docs/docs/admin/configuration.md b/docs/docs/admin/configuration.md index 70ecb01b..d1ed9f36 100644 --- a/docs/docs/admin/configuration.md +++ b/docs/docs/admin/configuration.md @@ -20,6 +20,10 @@ LITEFile follows [Twelve-Factor App](https://12factor.net/) principles, configur | `DATABASE_URL` | Yes (Prod) | `sqlite:///db.sqlite3` | Database connection string (e.g., `postgres://user:pass@host:5432/dbname`). | | `DJANGO_ALLOWED_HOSTS` | Yes (Prod) | `localhost,127.0.0.1` | Comma-separated list of allowed hostnames/domains. | | `EFSP_URL` | No | `https://efile-test.suffolklitlab.org` | Base URL for the EFSP REST API endpoint. | +| `FILING_CODE_SYNC_MODE` | No | `database` | Where filing codes come from. `database` copies them from the EFSP codes database (needs `EFSP_CODES_DATABASE_URL`); `bulk` uses the proxy's HTTP catalog export; `legacy` the older full crawler. | +| `EFSP_CODES_DATABASE_URL` | For `database` sync | — | PostgreSQL URL of the EFSP proxy's codes database, for example `postgresql://user:password@host:5432/postgres?sslmode=require`. Read only; prefer a read-only role. Use the test EFSP's database outside production. | +| `FILING_CODE_SYNC_TIME` | No | `04:30` | Daily time (`HH:MM`) the code index worker copies and indexes codes. Set it after the EFSP's own Tyler update (production 02:13, test 19:35 Eastern). | +| `FILING_CODE_SYNC_TIMEZONE` | No | `America/New_York` | Time zone for `FILING_CODE_SYNC_TIME`. | | `SUFFOLK_EFILE_API_KEY` | No | `""` | API authentication key for the EFSP proxy service. | | `OPENAI_API_KEY` | Optional | `None` | API key for OpenAI or compatible LLM provider used for document extraction. | | `OPENAI_BASE_URL` | Optional | `https://api.openai.com/v1/` | Base URL for OpenAI-compatible endpoint (or local LLM gateway). | diff --git a/docs/docs/admin/deployment.md b/docs/docs/admin/deployment.md index 359fb550..6de34993 100644 --- a/docs/docs/admin/deployment.md +++ b/docs/docs/admin/deployment.md @@ -61,14 +61,17 @@ primary_region = 'lax' [env] DJANGO_SETTINGS_MODULE = "efile.settings_staging" + FILING_CODE_SYNC_MODE = "legacy" [deploy] # Run database migrations before each release release_command = "uv run python manage.py migrate --noinput --fake-initial" + release_command_timeout = "30m" [processes] app = "uv run gunicorn efile.asgi:application -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 --workers 2 --timeout 60" extraction_worker = "uv run python manage.py process_document_extractions" + code_index_worker = "uv run python manage.py refresh_filing_code_index --interval 86400" [http_service] internal_port = 8000 @@ -90,6 +93,107 @@ Document analysis runs outside the web request in the `extraction_worker` proces By default, LITEFile sends only the first 20 PDF pages for analysis. Set `DOCUMENT_EXTRACTION_MAX_PAGES` to a positive integer to change that cap. DOCX text is limited to the first 100,000 characters with `DOCUMENT_EXTRACTION_MAX_TEXT_CHARS`; Word files have no reliable page boundaries. Review identifies when either limit omitted part of a document. `DOCUMENT_EXTRACTION_MAX_ATTEMPTS` controls how many times a failed job is tried before the filer is sent to manual review. +### Filing code search index + +Filing codes come straight from the EFSP proxy's codes database. The proxy +updates its codes from Tyler once a day (production at 02:13, test at 19:35 +Eastern). Each day at `FILING_CODE_SYNC_TIME` in `FILING_CODE_SYNC_TIMEZONE`, the +`code_index_worker` copies every court whose codes changed into LITEFile's own +database, then rebuilds the search index for those courts. Set the time well +after the proxy's update: Fly staging, which uses the test EFSP, runs at 21:30 +Eastern. Production should run at about 04:30. Keep one Machine running in the +`code_index_worker` process group. + +LITEFile connects with `EFSP_CODES_DATABASE_URL` and only ever reads. The court +list and every court's tables are read in one `READ ONLY`, `REPEATABLE READ` +transaction, so the copy is a consistent snapshot and the database refuses any +write. Supabase's connection pooler ignores connection-level settings, which is +why read-only is set per transaction. The queries match the proxy's own +filing-catalog export: courts with all three installed code lists, non-criminal +case categories, and filings that aren't court-use only. + +LITEFile connects as its own role, `litefile_codes_reader`, never with the +proxy's credentials. The proxy keeps user data in the same database, so the role +can read only the five code tables. It is also read only at the server, which +holds even through Supabase's pooler. The test EFSP's database already has it. +For another environment's database, run this once as its `postgres` role, +with a new random password: + +```sql +CREATE ROLE litefile_codes_reader LOGIN PASSWORD '' + NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOREPLICATION NOBYPASSRLS + CONNECTION LIMIT 5; +ALTER ROLE litefile_codes_reader SET default_transaction_read_only = on; +ALTER ROLE litefile_codes_reader SET statement_timeout = '10min'; +GRANT CONNECT ON DATABASE postgres TO litefile_codes_reader; +GRANT USAGE ON SCHEMA public TO litefile_codes_reader; +GRANT SELECT ON public.location, public.installedversion, public.casecategory, + public.casetype, public.filing TO litefile_codes_reader; +``` + +Through Supabase's pooler, the user name is `litefile_codes_reader.`: +`postgresql://litefile_codes_reader.:@:5432/postgres?sslmode=require`. +The proxy's nightly update reloads these tables' rows without dropping them, so +the grants persist. If a proxy schema migration ever recreates one of them, +grant `SELECT` on it again. Postgres lets every role create session-private +temporary tables (a `PUBLIC` default that can't be revoked from one role). Those +can't touch shared data, and the role's read-only default refuses them unless a +client turns read-only off, which LITEFile never does. + +A court whose revision is unchanged is not read again. A court that suddenly +has no filing types usually means the proxy's update is half finished, so the +copy stops and keeps the old one. Failed states are retried every 15 minutes +(`--retry-interval 900`), then the schedule returns to daily. On startup, the +worker syncs immediately if any jurisdiction has no current index. That is the +case on a fresh deployment, or after a search-rules change. + +#### Resync and rebuild from the staff tools + +Superusers see a **Filing codes** page in the staff tools (at the private `LITEFILE_STAFF_PATH`). It shows +each jurisdiction's copy and index, recent runs, and two buttons, for one +jurisdiction or all of them: + +- **Resync codes** copies changed courts from the EFSP now, then updates their + part of the index. +- **Rebuild code search indexes** rebuilds the whole index from the copy already + in LITEFile, without contacting the EFSP. Use it after changing search rules + in `filing_code_search.yaml`. + +The buttons queue a job, which the code index worker starts within a minute. +Each request is recorded in the staff audit log. + +#### From the command line + +Run these from `efile_app/` in the target environment: + +```bash +uv run python manage.py migrate --noinput +uv run python manage.py refresh_filing_code_index # resync every state now +uv run python manage.py refresh_filing_code_index --jurisdiction vermont # one state +uv run python manage.py refresh_filing_code_index --rebuild # rebuild from the local copy +uv run python manage.py refresh_filing_code_index --dry-run --jurisdiction vermont # report changes only +uv run python manage.py refresh_filing_code_index --daily # what the worker runs +``` + +On the test EFSP, Vermont (22 courts, about 1.1 million search paths) copies and +indexes in about two minutes. A run with nothing changed takes seconds. + +`FILING_CODE_SYNC_MODE=bulk` (the proxy's HTTP `filing_catalog` export) and +`legacy` (the full list-API crawler, `--legacy-crawl`) remain available as +fallbacks. Neither needs database access. `--cache-dir` applies only to the +legacy crawler. + +Local Docker Compose and `./run_all.sh` start the same daily worker. They read +`EFSP_CODES_DATABASE_URL` and `FILING_CODE_SYNC_TIME` from `efile_app/.env`. + +Before the first successful sync, the modal explains that search is unavailable +and the usual court lists remain available. The dialog does not trigger indexing, +but retries an unavailable search automatically while it stays open. +Results older than two days carry an out-of-date notice. Every selected path is +checked against the live court lists before it updates the form. A changed +`EFSP_URL` or thesaurus revision requires a fresh index, so test-server codes and +incompatible search terms cannot leak into another environment. + ### Setting Fly.io production secrets: ```bash fly secrets set \ @@ -101,6 +205,7 @@ fly secrets set \ AWS_S3_REGION_NAME="us-east-1" \ OPENAI_API_KEY="sk-..." \ GOTENBERG_URL="https://..." \ + EFSP_CODES_DATABASE_URL="postgresql://:...@:5432/postgres?sslmode=require" \ GOTENBERG_USERNAME="..." \ GOTENBERG_PASSWORD="..." ``` diff --git a/docs/docs/partners-courts/jurisdiction-config.md b/docs/docs/partners-courts/jurisdiction-config.md index 4fbda336..e5868339 100644 --- a/docs/docs/partners-courts/jurisdiction-config.md +++ b/docs/docs/partners-courts/jurisdiction-config.md @@ -405,3 +405,276 @@ See [Control filing availability](./filing-availability.md) for the complete field reference, Cook County hearing-scheduling example, county-prefix rules, message precedence, deployment and re-enabling instructions, troubleshooting, and live-check API. + +## Filing code search terms and explanations + +The “Find the right filing code” modal on Confirm case searches complete paths +across court, case category, case type, and filing type. A selected court does not +limit results. New and existing cases have separate filing-type lists. Search uses +English Snowball stemming, phrase synonyms, and conservative spelling suggestions; +it does not call an LLM or decide which court has jurisdiction over a person's case. + +The modal walks through three steps, each answering one question: + +1. **What are you filing?** A plain-language search, the case ZIP, and short + questions shown as chips. An empty search offers common filing-type starting + points. Results group repeated filing labels and recognized synonyms across + courts, so the same filing does not occupy a separate result for every county. + Codes a court splits only by jury, government-filer, or fee qualifiers + (Cook County's “Complaint / Petition - Fraud (Jury - 12) (Govn't) - Fee”) also + appear once; the exact one is chosen in step 2. Filings whose own names match + the search are listed first. When some do, filings that match only through + their case type are held behind a “Show N more” button. +2. **Choose your court and case type.** Courts that accept the chosen filing, + under the current answers and ZIP, appear as cards (or a dropdown when there + are more than six). A court already selected on the form is offered first and + preselected. Choosing a court lists its exact case-type paths; when a court + names the filing differently, the case type says so. Some courts code one + filing as many case types that differ only by claim amount, a lawyer or + self-represented filer, jury size, or a government filer (Cook County has 28 + for one personal-injury complaint). The court step reads those from the case + type and filing names and asks, as chips, only the ones that court's paths + differ on. It assumes a filer isn't filing for a government body, and shows + that answer so it can be changed. A path that doesn't say stays listed. +3. **Check your choices.** The full court, case category, case type, and filing + type, each with a link back to the step that set it. Applying revalidates the + path with the court before it fills the form. + +When a search matches eight or more filing types, step 1 also asks **What do you +want to do?** (start a case, respond, ask the court, settle or end, supporting +papers, proposed order, notice), with a count on each choice. The answer is a +guess from the filing name alone, from the ordered `actions` rules in +`efile_app/efile/data/filing_code_facets.yaml`; names no rule matches are +shown only under “Not sure”. + +When the matching case types name different claim amounts (“Small Claims - +$0 to $10,000”, “Amount Claimed Greater Than $10,000”), step 1 also asks **How +much are you asking for?**, in any case area. A range is labeled “(small +claims)” when its case types say so and the others don't. Case types that name +no amount stay listed. + +Step 1 also offers **Case category** (Cook's “Civil”, “Personal Injury/Wrongful +Death”, “Probate”, …) when results span more than one, so a filer can set aside +a division that doesn't fit. “Kind of case” and “Case category” list their most +used choices first and collapse the rest behind an “N more” chip. A case +question in `case_type_guidance.yaml` can set `other:` (for example +`other: Something else`) to add a choice that keeps only the case types none +of its options name, such as the general personal-injury lawsuit. + +### Court-term glossary + +`efile_app/efile/data/legal_glossary.yaml` gives plain-language meanings for +court terms that appear in filing and case-type names, such as subrogation, +dram shop, and tort. Each term lists the words that name it (`match`, whole +words, case-insensitive), a one- or two-sentence `text`, and an optional +`source` link. Under `jurisdictions:`, a state can reword a general term or add +its own (Massachusetts' “summary process”); its entries replace general ones +key by key. Wherever a result names a term, code search offers a “What does +‘subrogation’ mean?” disclosure: on each filing type, above a court's case +types, and on the final check. Meanings describe a term; they don't advise +whether to file. + +Grouping does not substitute one court's code for another's, and distinct +documents such as complaints and answers remain separate. Contexts reflect the +paths matching the current search, not unrelated uses of the same filing name. + +Case stage (new or existing) appears as one line under the search, with a +**Change** link. Document kind (main document or attachment/exhibit) sits behind +**More filters**. Starting a claim is not a determination of the filer's party +role: for example, an existing case can include a counterclaim. +`efile_app/efile/data/filing_code_facets.yaml` contains ordered filing-name rules; +the first match supplies the classification. Rules may limit `jurisdictions`. +Unmatched or ambiguous names remain unclassified and are shown when no document +filter is chosen. Filters apply before pagination and carry +through court and exact-path selection. These labels do not change which documents +the court permits as attachments. Choosing a different case stage changes the form +only after the filer applies a validated path. The API still accepts a `purpose` +filter, but the modal no longer offers it. + +Shared rules live in `efile_app/efile/data/filing_code_search.yaml`. The +`jurisdictions` list currently includes Illinois, Massachusetts, and Vermont. +Concepts group search terms, such as eviction, unlawful detainer, and summary +process. A concept can include its own `jurisdictions` list to limit its scope. +These are search associations, not statements that remedies or procedures are +legally interchangeable. Multiword phrases match together: “summary judgment” +does not expand to “summary process.” + +When a result was found through a related term rather than its own name (a +search for “unlawful detainer” finding an eviction filing), it says “Related to +eviction”. Concept definitions are expandable under “About this filing type”. +They describe the topic, not the purpose of every document. + +At the bottom of the results, a link such as “10 results in ‘existing case’ +filings” appears when the same search and answers find filings in the other +case stage. This matters for filings like sealing an eviction, which go in the +existing case. Choosing it switches the case stage and searches again. + +Display definitions live separately in +`efile_app/efile/data/filing_concept_definitions.yaml`, keyed by the same normalized +concept IDs as the search rules. The precedence is general default, state, county, +then exact court. Before a court is chosen, results show the general or statewide +meaning. Selected-court paths use the local meaning and identify its scope. + +For example, the following illustrates where **court-reviewed local wording** +belongs; the placeholder text is not a statement of Cook County procedure: + +```yaml +version: 1 +county_courts: + illinois: + Cook: + names: [Cook County] + prefixes: ["Cook County - "] +concepts: + eviction: + label: Eviction + default: + text: A court process about recovering possession of property. + jurisdictions: + illinois: + text: "" + source: "https://example.org/state-court-source" + counties: + Cook: + text: "" + source: "https://example.org/county-court-source" + courts: + Cook County - Municipal Civil - District 1 - Chicago: + text: "" + source: "https://example.org/court-specific-source" +``` + +County membership uses explicitly configured court names or prefixes, matched +without regard to capitalization. Use a separating boundary in prefixes, as above; +do not assume a court's county from a substring. Exact court overrides take priority +over county overrides. A replacement `text` uses its own `source`; it never inherits +a potentially unrelated source link. An empty `text` suppresses the inherited +definition. `label` can also be overridden. Unconfigured localities use the state +definition, then the general fallback. + +```yaml +concepts: + eviction: + terms: [eviction, evict, unlawful detainer, summary process] + +explanations: + - jurisdictions: [massachusetts] + filing_type: [Summary Process Complaint, Complaint for Summary Process] + text: A complaint asking the court to decide an eviction case. + source: https://www.mass.gov/how-to/file-an-eviction-case +``` + +Explanation rules match whole filing-type names after normalizing capitalization, +accents, and punctuation. They can also specify lists of exact `court`, +`case_category`, and `case_type` names, plus an `initial` boolean. All specified +conditions must match; the first matching rule supplies the explanation and its +source link. Put specific rules before general ones. Use court sources and short, +descriptive text rather than eligibility advice. Avoid matching numerical codes, +which can change between courts or environments. + +Unknown or renamed codes remain searchable without an explanation. The modal says +when no explanation is available and shows the official name and full path. +Live selection validation regenerates the explanation using the current names. +Filing availability restrictions remain visible and prevent choosing blocked paths. + +For changes to indexed terms or filing-code explanation rules in +`filing_code_search.yaml`, deploy with the application, restart workers, and refresh the index +using the [deployment instructions](../admin/deployment.md#filing-code-search-index). +Changes to concept definitions or facet classifications need only a web-process +restart; they are applied to responses outside the cached search results and do +not invalidate or rebuild the filing catalog. +Add search and explanation examples to +`efile_app/efile/tests/test_filing_code_search.py` when changing matching rules. + +### ZIP filters in filing code search + +Every jurisdiction offers an optional case ZIP. A state whose `court_selector` +configures a location `matcher` answers from that package: Massachusetts asks +MACourts and Vermont asks VTCourts which courts serve the ZIP. Courts the +package has no record of are placed by county instead. Other states (Illinois +and any new state) go through counties alone: the ZIP maps to counties through +`efile/data/zip_counties.json.gz`, built from Census 2020 ZCTA relationship files +by `efile_app/scripts/build_zip_counties.py`. Illinois's own +`efile/utils/zip_to_county_il.py` is checked first because it also knows +PO-box ZIPs. A court is placed in a county by its code (`cook:cvd1`), its name +(“Orange Unit”, “Middlesex Probate and Family Court”), or a town at the end of +its name (“District Court - Cambridge”). A court nothing can place, such as an +appeals court or a statewide division, stays visible for every ZIP. + +Filtering happens before grouping, so case-type previews, courts, and exact +paths all use the same scope. Clearing the ZIP restores statewide results. +Unknown ZIPs produce a recoverable message rather than silently showing +statewide results. ZIP matching is a location aid, not a determination of which +court has jurisdiction. The field is never prefilled. Instead, one-click +shortcuts offer the filer's own address ZIP and the last three ZIP codes they +used with an applied filing code. These are kept on the account as +`recent_case_zips`, because filers who help others often file far from their +own address. + +### Context-specific case guidance + +`efile/data/case_type_guidance.yaml` defines 20 starter case families with +plain-language descriptions, source links, and contextual facets. This is a +coverage set, not a statistical ranking of filing volume. Definitions are +matched by names, independently of environment-specific numeric IDs. + +| Case family | Contextual choices | +| --- | --- | +| Eviction | Tenant/landlord, home/business, possession/money | +| Foreclosure | Owner/lender, property, mortgage/other lien | +| Debt collection | Collecting/responding, debt or collection type | +| Small claims | Claimant/responding, dispute type, claim amount | +| Contracts | Claimant/responding, services/goods/insurance | +| Personal injury | Claimant/responding, vehicle/medical/property/death | +| Property damage | Claimant/responding, kind of property | +| Replevin and detinue | Requesting/responding, recovery procedure | +| Divorce and separation | Requesting/responding, children, kind of request | +| Parentage | Requesting/responding, establishing/challenging, order action | +| Parenting and custody | Requesting/responding, decisions/time/moving, order action | +| Child support | Requesting/responding, order action, interstate order | +| Protection orders | Requesting/responding, kind of protection, duration | +| Adult guardianship | Requesting/objecting, personal/financial authority, duration | +| Child guardianship | Requesting/objecting, personal/financial authority, duration | +| Estates and probate | Requesting/objecting, administration/will dispute/claim | +| Name changes | Requesting/objecting, adult/child | +| Adoption | Requesting/objecting, adult/child, relationship | +| Agency review | Challenging/responding, kind of agency decision | +| Employment | Worker/responding, wages/discrimination/dismissal | + +The configuration supports Illinois, Massachusetts, and Vermont. General starter +meanings do not establish eligibility or replace local court requirements. State +entries under a topic's `jurisdictions` can replace descriptions and facet rules. +County entries can provide `court_names` or `court_prefixes` and override `text` +and `source`; exact court entries under `courts` take precedence. Local wording +can differ without changing the state-level discovery classification. Detailed +rules should use explicit terms; absent information stays `unknown`. + +Each facet defines a key, label, and options. An option's regular expression +matches the case label, filing name, or both, according to its `field`. +Dimensions with no recognized values are hidden. Unknown classifications remain +included when filtering, with an explanation in the interface. Role choices +narrow explicit document names and keep shared/unclassified documents; they do +not decide a filer's legal party status. Changing the query or case area resets +its contextual choices. Filtering applies before pagination and remains in +effect when listing courts and exact paths. + +Small claims takes precedence over substantive labels within that track, such +as a contract claim in a small-claims category. The amount dropdown appears when +remaining labels contain different recognizable claim brackets. It parses +amounts in integer cents, including dollar signs, commas, decimal amounts, and +`K` suffixes. It never reads internal code IDs or filing fees. Labels without +recognizable brackets stay visible. A selected amount remains visible and can +be cleared even if another filter removes its matching labeled paths. + +Amount brackets describe catalog choices, not legal eligibility. State-specific +help links explain the usual limits and exceptions: Illinois generally uses +$10,000 excluding interest and costs; Massachusetts generally uses $7,000 with +an automobile property-damage exception; Vermont generally uses $10,000 but +limits consumer-credit and medical-debt claims to $5,000. Brackets crossing the +usual limit are marked for checking. Sources are stored alongside the wording +in the YAML, including the [Illinois court rules](https://www.illinoiscourts.gov/rules/supreme-court-rules/), +[Massachusetts small claims guidance](https://www.mass.gov/info-details/small-claims-court), +and [Vermont small claims guidance](https://www.vtcourts.gov/civil/suing-and-being-sued-small-claims). + +The guidance configuration is cached per web process. Restart the web service +after editing it; no filing catalog rebuild or database migration is required. diff --git a/efile_app/.env.example b/efile_app/.env.example index 36c5e4cb..b707624b 100644 --- a/efile_app/.env.example +++ b/efile_app/.env.example @@ -61,3 +61,9 @@ GOTENBERG_USERNAME = "your-gotenberg-username" GOTENBERG_PASSWORD = "your-gotenberg-password" DOCUMENT_PREPARATION_TIMEOUT_SECONDS = 45 + +# Filing codes are copied, read only, from the EFSP proxy's codes database. +# Use the TEST EFSP's database for development; ask the EFSP maintainers for it. +# EFSP_CODES_DATABASE_URL=postgresql://user:password@host:5432/postgres?sslmode=require +# The test EFSP updates codes from Tyler at 19:35 Eastern; copy them after. +# FILING_CODE_SYNC_TIME=21:30 diff --git a/efile_app/efile/api/dropdown_views.py b/efile_app/efile/api/dropdown_views.py index b1e13509..d2c25d6b 100644 --- a/efile_app/efile/api/dropdown_views.py +++ b/efile_app/efile/api/dropdown_views.py @@ -119,7 +119,8 @@ def get_case_categories(request): # Make API call to external categories endpoint api_url = f"{settings.EFSP_URL}/jurisdictions/{jurisdiction}/codes/courts/{court_code}/categories" - params = {"fileable_only": True, "timing": "Initial"} + timing = "Subsequent" if request.GET.get("existing_case") == "yes" else "Initial" + params = {"fileable_only": True, "timing": timing} # Make the API request with auth tokens if available headers = {} @@ -169,7 +170,8 @@ def get_case_types(request): # Make API call to external case types endpoint path = f"/jurisdictions/{jurisdiction}/codes/courts/{court_code}/case_types/" api_url = f"{settings.EFSP_URL}{path}" - params = {"category_id": category_id, "timing": "Initial"} + timing = "Subsequent" if request.GET.get("existing_case") == "yes" else "Initial" + params = {"category_id": category_id, "timing": timing} # Make the API request with auth tokens if available headers = {} diff --git a/efile_app/efile/api/filing_code_search.py b/efile_app/efile/api/filing_code_search.py new file mode 100644 index 00000000..919daa94 --- /dev/null +++ b/efile_app/efile/api/filing_code_search.py @@ -0,0 +1,146 @@ +"""Read-only search and live validation; applying a result edits the form only.""" + +import json +import logging +import re +from datetime import timedelta + +import requests +from django.http import JsonResponse +from django.utils import timezone +from django.views.decorators.http import require_http_methods + +from efile.services.filing_code_search import ( + action_rules, + current_index, + rules, + search_grouped_paths, + search_paths, + validate_path, +) +from efile.utils.jurisdiction_stuff import has_jurisdiction_login + +logger = logging.getLogger(__name__) + +RECENT_ZIP_LIMIT = 3 + + +def remember_case_zip(user, postal_code): + """Keep the ZIP a filer just found a usable code with, newest first.""" + if not user.is_authenticated or not re.fullmatch(r"[0-9]{5}(?:-[0-9]{4})?", postal_code): + return + postal_code = postal_code[:5] + recent = [postal_code, *[item for item in user.recent_case_zips or [] if item != postal_code]] + if recent[:RECENT_ZIP_LIMIT] != user.recent_case_zips: + user.recent_case_zips = recent[:RECENT_ZIP_LIMIT] + user.save(update_fields=["recent_case_zips"]) + + +@require_http_methods(["GET"]) +def filing_code_search(request): + jurisdiction = request.GET.get("jurisdiction", "").lower() + if jurisdiction not in rules()[0]["jurisdictions"]: + return JsonResponse({"error": "Choose a supported jurisdiction."}, status=400) + if not has_jurisdiction_login(request, jurisdiction): + return JsonResponse({"error": "Sign in to this jurisdiction to search filing codes."}, status=403) + initial = request.GET.get("existing_case", "no") != "yes" + index = current_index(jurisdiction) + if index is None: + return JsonResponse({"error": "Code search is not available yet. Use the court lists to continue."}, status=503) + if request.GET.get("path_id"): + try: + path_id = int(request.GET["path_id"]) + except ValueError: + return JsonResponse({"error": "Invalid filing path."}, status=400) + path = index.paths.filter(pk=path_id, initial=initial).select_related("index").first() + if path is None: + return JsonResponse({"error": "The code list has changed. Search again."}, status=409) + try: + validated = validate_path(path) + remember_case_zip(request.user, request.GET.get("zip", "").strip()) + return JsonResponse({"path": validated}) + except ValueError as error: + return JsonResponse({"error": str(error)}, status=409) + except requests.RequestException: + logger.warning("Could not revalidate filing search path %s", path.pk) + return JsonResponse({"error": "The court lists could not be checked. Try again."}, status=503) + query = request.GET.get("q", "").strip() + purpose = request.GET.get("purpose", "") + document = request.GET.get("document", "") + if purpose not in ("", "starting", "responding", "either", "unknown") or document not in ( + "", + "main", + "attachment", + "unknown", + ): + return JsonResponse({"error": "Choose a valid search filter."}, status=400) + action = request.GET.get("action", "") + if action and action not in {rule["key"] for _, rule in action_rules()}: + return JsonResponse({"error": "Choose a valid search filter."}, status=400) + role = request.GET.get("role", "") + property_kind = request.GET.get("property", "") + relief = request.GET.get("relief", "") + if ( + role not in ("", "tenant", "landlord") + or property_kind not in ("", "residential", "commercial") + or relief not in ("", "possession", "money") + ): + return JsonResponse({"error": "Choose a valid case-type filter."}, status=400) + try: + case_filters = json.loads(request.GET.get("case_filters", "{}")) + if not isinstance(case_filters, dict): + raise ValueError + except (ValueError, TypeError): + return JsonResponse({"error": "Choose valid case-type filters."}, status=400) + try: + offset = int(request.GET.get("offset", "0")) + except ValueError: + offset = -1 + try: + # The court step loads every path for one court at once, so it can ask + # about the choices they differ on; other requests page by 20. + limit = int(request.GET.get("limit", "20")) + except ValueError: + limit = 0 + if not 0 <= offset <= 10000 or not 1 <= len(query) <= 160 or not 1 <= limit <= 500: + return JsonResponse( + {"error": "Enter a search of up to 160 characters and use a valid results page."}, status=400 + ) + try: + if request.GET.get("grouped") == "true": + result = search_grouped_paths( + index, + query, + initial=initial, + offset=offset, + limit=limit, + group_key=request.GET.get("group", ""), + court=request.GET.get("court", ""), + purpose=purpose, + document=document, + context=request.GET.get("context", ""), + postal_code=request.GET.get("zip", "").strip(), + role=role, + property_kind=property_kind, + relief=relief, + case_topic=request.GET.get("case_topic", ""), + case_filters=case_filters, + action=action, + category=request.GET.get("category", "")[:200], + strong_only=request.GET.get("strong") == "true", + ) + else: + if purpose or document: + return JsonResponse({"error": "Use grouped search with filing-type filters."}, status=400) + result = search_paths(index, query, initial=initial, offset=offset) + except ValueError as error: + return JsonResponse({"error": str(error)}, status=409) + result.update( + { + "refreshed_at": index.refreshed_at.isoformat(), + "stale": index.refreshed_at < timezone.now() - timedelta(days=2), + } + ) + response = JsonResponse(result) + response["Cache-Control"] = "no-store" + return response diff --git a/efile_app/efile/api/urls.py b/efile_app/efile/api/urls.py index 0d373b35..ebc4c6ad 100644 --- a/efile_app/efile/api/urls.py +++ b/efile_app/efile/api/urls.py @@ -27,6 +27,7 @@ get_party_types, ) from .filing_availability import get_filing_availability +from .filing_code_search import filing_code_search from .filing_views import get_filings, payment_fees from .payment_views import delete_payment_account, waiver_account from .suffolk_api_views import get_party_types_from_suffolk_api, lookup_case @@ -34,6 +35,7 @@ app_name = "api" urlpatterns = [ + path("filing-code-search/", filing_code_search, name="filing_code_search"), path("get-party-types/", get_party_types_from_suffolk_api, name="get_party_types"), # Dropdown API endpoints path("dropdowns/case-categories/", get_case_categories, name="case_categories"), diff --git a/efile_app/efile/data/case_type_guidance.yaml b/efile_app/efile/data/case_type_guidance.yaml new file mode 100644 index 00000000..2d5fd52b --- /dev/null +++ b/efile_app/efile/data/case_type_guidance.yaml @@ -0,0 +1,854 @@ +# Curated guidance and discovery facets. Unknown classifications remain visible. +version: 2 +jurisdictions: +- illinois +- massachusetts +- vermont +topics: + eviction: + label: Eviction + pattern: \b(eviction|ejectment|forcible|unlawful detainer|summary process)\b + query_pattern: \b(eviction|evict\w*|ejectment|unlawful detainer|summary process|landlord|tenant|forcible)\b + text: A case about who has the right to occupy property. + source: https://www.illinoiscourts.gov/courts/additional-resources/eviction/ + facets: + - key: role + label: Your role + options: + - value: tenant + label: Tenant / occupant + pattern: ^(answer|response|objection)\b|\b(for|by) (respondent|defendant)\b + field: filing + - value: landlord + label: Landlord / owner + pattern: ^(complaint|petition)(\b|/)|\b(for|by) (petitioner|plaintiff)\b + field: filing + - key: property + label: Property + options: + - value: residential + label: Home + pattern: \bresidential\b + field: case + text: A home or apartment. + - value: commercial + label: Business + pattern: \bcommercial\b + field: case + text: Property used for a business, such as a store or office. + - key: relief + label: What the landlord is asking for + options: + - value: possession + label: Property back only + pattern: \bpossession only\b + field: case + text: The landlord seeks the property back, not unpaid rent; court costs may still be requested. + - value: money + label: Money too + pattern: \b(money|rent|damages)\b + field: case + text: Includes a money claim; check the complaint for what is requested. + jurisdictions: + illinois: + text: An eviction case asks the court to return possession of property to a landlord or owner. Some cases + also seek unpaid rent. + source: https://www.illinoiscourts.gov/courts/additional-resources/eviction/ + special: + - pattern: \bejectment\b + text: A claim to recover property based on a legal right to possess it. It can involve ownership rights; + it is not simply another name for a residential rental eviction. + source: https://my.ilga.gov/legislation/ILCS/details?ActID=2017&ActName=Code+of+Civil+Procedure.&ChapAct=735+ILCS+5%2F&Chapter=CIVIL+PROCEDURE&ChapterID=56&MajorTopic=RIGHTS+AND+REMEDIES&SeqEnd=39000000&SeqStart=34500000 + massachusetts: + text: Massachusetts calls an eviction court case “summary process.” It concerns possession of rental property + and may also include a claim for unpaid rent. + source: https://www.mass.gov/eviction-for-tenants + vermont: + text: An eviction case asks the Civil Division of the Superior Court to return possession of rented property + to a landlord. + source: https://www.vtcourts.gov/civil/eviction-process + unknown_notes: + - when: relief + if_known: property + text: The name does not say whether money is requested; check the complaint. + foreclosure: + label: Mortgage foreclosure + pattern: \b(foreclos\w*)\b + query_pattern: \b(foreclos\w*)\b + text: A lender or lien holder asks the court to sell property to pay a secured debt. + source: https://www.illinoiscourts.gov/courts/additional-resources/mortgage-foreclosure/ + facets: + - key: role + label: Your role + options: + - value: responding + label: Property owner + pattern: ^(answer|response|objection)\b|\b(for|by) (respondent|defendant)\b + field: filing + - value: requesting + label: Lender / lien holder + pattern: ^(complaint|petition)(\b|/)|\b(for|by) (petitioner|plaintiff)\b + field: filing + - key: property + label: Property + options: + - value: residential + label: Home + pattern: \bresidential\b + field: case + text: A home or apartment. + - value: commercial + label: Business + pattern: \bcommercial\b + field: case + text: Property used for a business, such as a store or office. + - key: issue + label: Type of claim + options: + - value: mortgage + label: Mortgage loan + pattern: \bmortgage\b + field: case + - value: lien + label: Other lien + pattern: \b(mechanic|assessment|condominium|lien)\b + field: case + jurisdictions: {} + replevin: + label: Return of personal property + pattern: \b(replevin|detinue)\b + query_pattern: \b(replevin|detinue)\b + text: A request to recover belongings held by someone else; this is different from recovering land or housing. + source: https://www.illinoislegalaid.org/legal-information/getting-my-property-back + facets: + - key: role + label: Your role + options: + - value: responding + label: Responding to the request + pattern: ^(answer|response|objection)\b|\b(for|by) (respondent|defendant)\b + field: filing + - value: requesting + label: Seeking property back + pattern: ^(complaint|petition)(\b|/)|\b(for|by) (petitioner|plaintiff)\b + field: filing + - key: procedure + label: Court label + options: + - value: replevin + label: Replevin + pattern: \breplevin\b + field: case + text: Replevin asks for return of specific belongings through a statutory court process. + - value: detinue + label: Detinue + pattern: \bdetinue\b + field: case + text: Detinue is another court procedure for recovery of belongings or their value. + jurisdictions: {} + property_damage: + label: Property damage + pattern: \b(property damage|damage to property)\b + query_pattern: \b(property damage|damage to property)\b + text: A claim for harm to belongings, a vehicle, or real estate, rather than injury to a person. + source: https://www.illinoislegalaid.org/legal-information/what-are-my-rights-when-car-accident-happens + facets: + - key: role + label: Your role + options: + - value: responding + label: Responding to a claim + pattern: ^(answer|response|objection)\b|\b(for|by) (respondent|defendant)\b + field: filing + - value: requesting + label: Seeking payment for damage + pattern: ^(complaint|petition)(\b|/)|\b(for|by) (petitioner|plaintiff)\b + field: filing + - key: property + label: Damaged property + options: + - value: vehicle + label: Vehicle + pattern: \b(vehicle|auto\w*)\b + field: case + - value: real_estate + label: Land or building + pattern: \b(real estate|real property|building)\b + field: case + - value: belongings + label: Other belongings + pattern: \b(personal property|belongings)\b + field: case + jurisdictions: {} + personal_injury: + label: Personal injury + # "Tort - Not Personal Injury" is not one; legal and other professional + # malpractice harm money, not people. + pattern: (?> 'code')", **extra_context) diff --git a/efile_app/efile/management/commands/process_document_extractions.py b/efile_app/efile/management/commands/process_document_extractions.py index b0ef8ded..6a3cbc1c 100644 --- a/efile_app/efile/management/commands/process_document_extractions.py +++ b/efile_app/efile/management/commands/process_document_extractions.py @@ -6,7 +6,7 @@ from django.conf import settings from django.core.management import call_command from django.core.management.base import BaseCommand -from django.db import close_old_connections +from django.db import OperationalError, close_old_connections, connection logger = logging.getLogger(__name__) @@ -69,7 +69,21 @@ def rollup_if_due(): while True: close_old_connections() rollup_if_due() - job = claim_next_extraction() + try: + job = claim_next_extraction() + except OperationalError as error: + # A large local code-index publication can outlast SQLite's + # busy timeout. Keep the supervisor alive to claim work after + # the writer finishes; other database errors still fail loudly. + if ( + options["once"] + or connection.vendor != "sqlite" + or str(error) not in {"database is locked", "database table is locked"} + ): + raise + logger.warning("Local database is busy; document extraction will retry") + time.sleep(max(0.1, options["poll_interval"])) + continue if job is None: if options["once"]: return diff --git a/efile_app/efile/management/commands/refresh_filing_code_index.py b/efile_app/efile/management/commands/refresh_filing_code_index.py new file mode 100644 index 00000000..8e87e88f --- /dev/null +++ b/efile_app/efile/management/commands/refresh_filing_code_index.py @@ -0,0 +1,188 @@ +"""Build complete code snapshots outside the request and release paths.""" + +import time +from datetime import timedelta +from pathlib import Path +from zoneinfo import ZoneInfo + +import requests +from django.conf import settings +from django.core.management.base import BaseCommand, CommandError +from django.db import DatabaseError, close_old_connections +from django.test.utils import override_settings +from django.utils import timezone + +from efile.models import FilingCodeJob +from efile.services.filing_code_copy import rebuild, run_jobs +from efile.services.filing_code_search import current_index, refresh_index, rules + +# How often the daily worker checks for staff-requested jobs between runs. +JOB_POLL_SECONDS = 30 + + +def next_daily_run(now, at, zone): + """The next moment at ``at`` (HH:MM) in ``zone`` after ``now``.""" + hour, minute = (int(part) for part in at.split(":")) + local = now.astimezone(ZoneInfo(zone)) + run = local.replace(hour=hour, minute=minute, second=0, microsecond=0) + if run <= local: + run = (local + timedelta(days=1)).replace(hour=hour, minute=minute, second=0, microsecond=0) + return run + + +class Command(BaseCommand): + help = "Sync changed court code exports, preserving previous snapshots on failure." + + def add_arguments(self, parser): + parser.add_argument("--jurisdiction", choices=rules()[0]["jurisdictions"]) + parser.add_argument("--interval", type=int, default=0, help="Repeat after this many seconds (0: run once).") + parser.add_argument( + "--daily", + action="store_true", + help="Run as the code index worker: sync once a day at FILING_CODE_SYNC_TIME " + "(FILING_CODE_SYNC_TIMEZONE), and run resync/rebuild jobs staff queue meanwhile.", + ) + parser.add_argument( + "--rebuild", + action="store_true", + help="Rebuild the search index from the local copy of the EFSP codes; contacts nothing.", + ) + parser.add_argument("--retry-interval", type=int, default=60, help="Seconds before retrying failed states.") + parser.add_argument( + "--legacy-crawl", action="store_true", help="Explicitly use the expensive full list-API crawler." + ) + parser.add_argument( + "--court", + action="append", + default=[], + help="Re-download this court even if its revision is unchanged (repeatable; needs --jurisdiction).", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Download and check changed courts and report path counts, without saving anything.", + ) + parser.add_argument( + "--cache-dir", + default=str(Path.home() / ".cache" / "litefile" / "filing-code-cache"), + help="Legacy crawler checkpoints, reused for six hours.", + ) + + def handle(self, *args, **options): + interval = options["interval"] + if interval < 0: + raise CommandError("Interval must be nonnegative") + retry_interval = options["retry_interval"] + if retry_interval <= 0: + raise CommandError("Retry interval must be positive") + if options["court"] and not options["jurisdiction"]: + raise CommandError("--court needs --jurisdiction") + if (options["court"] or options["dry_run"]) and (interval or options["daily"]): + raise CommandError("--court and --dry-run run once; drop --interval and --daily") + if options["daily"] and interval: + raise CommandError("Choose --daily or --interval") + jurisdictions = [options["jurisdiction"]] if options["jurisdiction"] else rules()[0]["jurisdictions"] + if options["rebuild"]: + return self.rebuild(jurisdictions) + if options["daily"]: + return self.daily(jurisdictions, options) + while True: + failures = [] + # The worker sleeps for hours between passes; a connection the + # database or proxy dropped meanwhile must not fail the next one. + close_old_connections() + for jurisdiction in jurisdictions: + try: + if interval: + index = current_index(jurisdiction) + if index and (timezone.now() - index.refreshed_at).total_seconds() < interval: + continue + # This dedicated command doesn't serve requests. Rendering + # and retaining SQL debug strings for millions of rows is + # costly in local development; restore DEBUG after the build. + with override_settings(DEBUG=False): + count = refresh_index( + jurisdiction, + cache_dir=options["cache_dir"], + progress=lambda court, state=jurisdiction: self.stdout.write(f"{state}: {court}") + if options["verbosity"] > 1 or options["dry_run"] + else None, + force=options["court"], + dry_run=options["dry_run"], + **({"transport": "legacy"} if options["legacy_crawl"] else {}), + ) + verb = "would synchronize" if options["dry_run"] else "synchronized" + self.stdout.write(self.style.SUCCESS(f"{jurisdiction}: {verb} {count} changed filing paths")) + # DatabaseError covers a lost connection and an overlapping run + # creating the same jurisdiction's row first (IntegrityError). + except (requests.RequestException, ValueError, DatabaseError) as error: + failures.append(jurisdiction) + self.stderr.write(f"{jurisdiction}: refresh failed; previous index kept. {error}") + if not interval: + if failures: + raise CommandError("Refresh failed: " + ", ".join(failures)) + return + delay = min(interval, retry_interval) if failures else interval + if failures: + self.stderr.write(f"Retrying failed states in {delay} seconds.") + time.sleep(delay) + + def sync(self, jurisdictions, options): + failures = [] + close_old_connections() + for jurisdiction in jurisdictions: + try: + with override_settings(DEBUG=False): + count = refresh_index( + jurisdiction, + cache_dir=options["cache_dir"], + progress=(lambda court, state=jurisdiction: self.stdout.write(f"{state}: {court}")) + if options["verbosity"] > 1 + else None, + **({"transport": "legacy"} if options["legacy_crawl"] else {}), + ) + self.stdout.write(self.style.SUCCESS(f"{jurisdiction}: synchronized {count} changed filing paths")) + except (requests.RequestException, ValueError, DatabaseError, OSError) as error: + # psycopg's errors for the EFSP database are DatabaseErrors too. + failures.append(jurisdiction) + self.stderr.write(f"{jurisdiction}: refresh failed; previous index kept. {error}") + return failures + + def rebuild(self, jurisdictions): + failures = [] + for jurisdiction in jurisdictions: + try: + with override_settings(DEBUG=False): + count = rebuild( + jurisdiction, progress=lambda line, state=jurisdiction: self.stdout.write(f"{state}: {line}") + ) + self.stdout.write(self.style.SUCCESS(f"{jurisdiction}: rebuilt {count} filing paths")) + except (ValueError, DatabaseError) as error: + failures.append(jurisdiction) + self.stderr.write(f"{jurisdiction}: rebuild failed; previous index kept. {error}") + if failures: + raise CommandError("Rebuild failed: " + ", ".join(failures)) + + def daily(self, jurisdictions, options): + at, zone = settings.FILING_CODE_SYNC_TIME, settings.FILING_CODE_SYNC_TIMEZONE + # A job a dead worker left running would otherwise stay "running" forever. + FilingCodeJob.objects.filter(status=FilingCodeJob.Status.RUNNING).update( + status=FilingCodeJob.Status.FAILED, finished_at=timezone.now(), message="Interrupted by a worker restart." + ) + # A fresh deployment, or one whose search rules changed, has nothing to + # search until the first sync: don't wait for tonight's. + pending = [j for j in jurisdictions if current_index(j) is None] + due = timezone.now() if pending else next_daily_run(timezone.now(), at, zone) + while True: + self.stdout.write(f"Next code sync at {due.astimezone(ZoneInfo(zone)):%Y-%m-%d %H:%M %Z}.") + while timezone.now() < due: + close_old_connections() + run_jobs(jurisdictions, log=self.stdout.write) + time.sleep(min(JOB_POLL_SECONDS, max(1, (due - timezone.now()).total_seconds()))) + # Failed states are retried alone, then the schedule returns to daily. + pending = self.sync(pending or jurisdictions, options) + if pending: + self.stderr.write(f"Retrying failed states in {options['retry_interval']} seconds.") + due = timezone.now() + timedelta(seconds=options["retry_interval"]) + else: + due = next_daily_run(timezone.now(), at, zone) diff --git a/efile_app/efile/migrations/0033_filing_code_search.py b/efile_app/efile/migrations/0033_filing_code_search.py new file mode 100644 index 00000000..9cfe09a5 --- /dev/null +++ b/efile_app/efile/migrations/0033_filing_code_search.py @@ -0,0 +1,56 @@ +# Generated by Django 5.2.17 on 2026-10-03 03:33 + +import django.db.models.deletion +from django.db import migrations, models + + +def add_postgres_search_index(apps, schema_editor): + if schema_editor.connection.vendor == "postgresql": + schema_editor.execute( + "CREATE INDEX filing_code_search_gin ON efile_filingcodepath " + "USING GIN (to_tsvector('simple'::regconfig, COALESCE(search_text, '')))" + ) + + +def remove_postgres_search_index(apps, schema_editor): + if schema_editor.connection.vendor == "postgresql": + schema_editor.execute("DROP INDEX IF EXISTS filing_code_search_gin") + + +class Migration(migrations.Migration): + + dependencies = [ + ('efile', '0032_inventory_owned_uploads'), + ] + + operations = [ + migrations.CreateModel( + name='FilingCodeIndex', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('jurisdiction', models.CharField(max_length=20, unique=True)), + ('source_url', models.URLField()), + ('refreshed_at', models.DateTimeField()), + ('rules_digest', models.CharField(max_length=64)), + ('vocabulary', models.JSONField(default=dict)), + ], + ), + migrations.CreateModel( + name='FilingCodePath', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('initial', models.BooleanField()), + ('court', models.JSONField()), + ('case_category', models.JSONField()), + ('case_type', models.JSONField()), + ('filing_type', models.JSONField()), + ('search_text', models.TextField()), + ('filing_terms', models.TextField()), + ('case_terms', models.TextField()), + ('explanation', models.TextField(blank=True)), + ('explanation_source', models.URLField(blank=True)), + ('index', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='paths', to='efile.filingcodeindex')), + ], + ), + migrations.RunPython(add_postgres_search_index, remove_postgres_search_index), + ] diff --git a/efile_app/efile/migrations/0034_sqlite_filing_search_index.py b/efile_app/efile/migrations/0034_sqlite_filing_search_index.py new file mode 100644 index 00000000..a988cf08 --- /dev/null +++ b/efile_app/efile/migrations/0034_sqlite_filing_search_index.py @@ -0,0 +1,51 @@ +"""Give local SQLite searches the same indexed token lookup as PostgreSQL.""" + +from django.db import migrations + + +def add_sqlite_search_index(apps, schema_editor): + if schema_editor.connection.vendor != "sqlite": + return + # Tokens are already normalized, stemmed and expanded by the application. + # Store postings only: the path table remains the source of document data. + # No positional data is needed for our AND-of-whole-token queries. Retain + # columns so jurisdiction and new/existing-case filters use postings too. + schema_editor.execute( + "CREATE VIRTUAL TABLE filing_code_search_fts USING fts5(" + "search_text, index_id, initial, content='efile_filingcodepath', content_rowid='id', " + "tokenize='ascii', detail='column', columnsize=0)" + ) + schema_editor.execute( + "CREATE TRIGGER filing_code_search_insert AFTER INSERT ON efile_filingcodepath BEGIN " + "INSERT INTO filing_code_search_fts(rowid, search_text, index_id, initial) " + "VALUES (new.id, new.search_text, new.index_id, new.initial); END" + ) + schema_editor.execute( + "CREATE TRIGGER filing_code_search_delete AFTER DELETE ON efile_filingcodepath BEGIN " + "INSERT INTO filing_code_search_fts(filing_code_search_fts, rowid, search_text, index_id, initial) " + "VALUES ('delete', old.id, old.search_text, old.index_id, old.initial); END" + ) + schema_editor.execute( + "CREATE TRIGGER filing_code_search_update AFTER UPDATE OF id, search_text, index_id, initial " + "ON efile_filingcodepath BEGIN " + "INSERT INTO filing_code_search_fts(filing_code_search_fts, rowid, search_text, index_id, initial) " + "VALUES ('delete', old.id, old.search_text, old.index_id, old.initial); " + "INSERT INTO filing_code_search_fts(rowid, search_text, index_id, initial) " + "VALUES (new.id, new.search_text, new.index_id, new.initial); END" + ) + # Triggers maintain future changes; existing installations also need their + # already-downloaded catalog indexed. No court API calls are necessary. + schema_editor.execute("INSERT INTO filing_code_search_fts(filing_code_search_fts) VALUES ('rebuild')") + + +def remove_sqlite_search_index(apps, schema_editor): + if schema_editor.connection.vendor != "sqlite": + return + for name in ("insert", "delete", "update"): + schema_editor.execute(f"DROP TRIGGER filing_code_search_{name}") + schema_editor.execute("DROP TABLE filing_code_search_fts") + + +class Migration(migrations.Migration): + dependencies = [("efile", "0033_filing_code_search")] + operations = [migrations.RunPython(add_sqlite_search_index, remove_sqlite_search_index)] diff --git a/efile_app/efile/migrations/0035_incremental_filing_catalog.py b/efile_app/efile/migrations/0035_incremental_filing_catalog.py new file mode 100644 index 00000000..2e87b95a --- /dev/null +++ b/efile_app/efile/migrations/0035_incremental_filing_catalog.py @@ -0,0 +1,42 @@ +# Generated by Django 5.2.17 on 2026-10-06 16:25 + +import efile.db_expressions +from django.db import migrations, models +from django.contrib.postgres.operations import AddIndexConcurrently + + +class AddCourtIndex(AddIndexConcurrently): + """Avoid blocking production writes while indexing an existing large catalog.""" + + def database_forwards(self, app_label, schema_editor, from_state, to_state): + if schema_editor.connection.vendor == "postgresql": + super().database_forwards(app_label, schema_editor, from_state, to_state) + else: + migrations.AddIndex.database_forwards(self, app_label, schema_editor, from_state, to_state) + + def database_backwards(self, app_label, schema_editor, from_state, to_state): + if schema_editor.connection.vendor == "postgresql": + super().database_backwards(app_label, schema_editor, from_state, to_state) + else: + migrations.AddIndex.database_backwards(self, app_label, schema_editor, from_state, to_state) + + +class Migration(migrations.Migration): + + atomic = False + + dependencies = [ + ('efile', '0034_sqlite_filing_search_index'), + ] + + operations = [ + migrations.AddField( + model_name='filingcodeindex', + name='court_snapshots', + field=models.JSONField(default=dict), + ), + AddCourtIndex( + model_name='filingcodepath', + index=models.Index(models.F('index'), efile.db_expressions.CourtCode('court'), name='filing_path_court'), + ), + ] diff --git a/efile_app/efile/migrations/0036_user_recent_case_zips.py b/efile_app/efile/migrations/0036_user_recent_case_zips.py new file mode 100644 index 00000000..23ca536a --- /dev/null +++ b/efile_app/efile/migrations/0036_user_recent_case_zips.py @@ -0,0 +1,18 @@ +# Generated by Django 5.2.17 on 2026-10-06 19:27 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('efile', '0035_incremental_filing_catalog'), + ] + + operations = [ + migrations.AddField( + model_name='userprofile', + name='recent_case_zips', + field=models.JSONField(blank=True, default=list), + ), + ] diff --git a/efile_app/efile/migrations/0037_efsp_code_copies_and_jobs.py b/efile_app/efile/migrations/0037_efsp_code_copies_and_jobs.py new file mode 100644 index 00000000..a872ebda --- /dev/null +++ b/efile_app/efile/migrations/0037_efsp_code_copies_and_jobs.py @@ -0,0 +1,48 @@ +# Generated by Django 5.2.17 on 2026-10-07 00:45 + +import django.db.models.deletion +from django.conf import settings +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('efile', '0036_user_recent_case_zips'), + ] + + operations = [ + migrations.CreateModel( + name='FilingCodeCourtCatalog', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('jurisdiction', models.CharField(max_length=20)), + ('code', models.CharField(max_length=100)), + ('name', models.CharField(max_length=255)), + ('revision', models.CharField(max_length=64)), + ('data', models.JSONField()), + ('filing_count', models.PositiveIntegerField(default=0)), + ('copied_at', models.DateTimeField()), + ], + options={ + 'constraints': [models.UniqueConstraint(fields=('jurisdiction', 'code'), name='unique_court_catalog_per_jurisdiction')], + }, + ), + migrations.CreateModel( + name='FilingCodeJob', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('kind', models.CharField(choices=[('resync', 'Resync codes'), ('rebuild', 'Rebuild code search indexes')], max_length=20)), + ('jurisdiction', models.CharField(blank=True, max_length=20)), + ('status', models.CharField(choices=[('queued', 'Queued'), ('running', 'Running'), ('succeeded', 'Succeeded'), ('failed', 'Failed')], default='queued', max_length=20)), + ('requested_at', models.DateTimeField(auto_now_add=True)), + ('started_at', models.DateTimeField(blank=True, null=True)), + ('finished_at', models.DateTimeField(blank=True, null=True)), + ('message', models.TextField(blank=True)), + ('requested_by', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to=settings.AUTH_USER_MODEL)), + ], + options={ + 'ordering': ['-requested_at'], + }, + ), + ] diff --git a/efile_app/efile/models.py b/efile_app/efile/models.py index d725d2c8..5984df40 100644 --- a/efile_app/efile/models.py +++ b/efile_app/efile/models.py @@ -4,10 +4,92 @@ from django.db import models, transaction from django.utils import timezone +from efile.db_expressions import CourtCode from efile.party_sides import PARTY_SIDE_CHOICES from efile.workflow import ExistingCase, WorkflowStepKey, get_workflow_step_choices +class FilingCodeIndex(models.Model): + """One complete, last-known-good code snapshot per jurisdiction and EFSP.""" + + jurisdiction = models.CharField(max_length=20, unique=True) + source_url = models.URLField() + refreshed_at = models.DateTimeField() + rules_digest = models.CharField(max_length=64) + vocabulary = models.JSONField(default=dict) + court_snapshots = models.JSONField(default=dict) + + +class FilingCodePath(models.Model): + """A real court/category/case-type/filing-type path, never a cross product.""" + + index = models.ForeignKey(FilingCodeIndex, on_delete=models.CASCADE, related_name="paths") + initial = models.BooleanField() + court = models.JSONField() + case_category = models.JSONField() + case_type = models.JSONField() + filing_type = models.JSONField() + search_text = models.TextField() + filing_terms = models.TextField() + case_terms = models.TextField() + explanation = models.TextField(blank=True) + explanation_source = models.URLField(blank=True) + + class Meta: + indexes = [models.Index(models.F("index"), CourtCode("court"), name="filing_path_court")] + + +class FilingCodeCourtCatalog(models.Model): + """One court's filing code tables, copied from the EFSP codes database. + + The search index is built from these copies, so rebuilding it (after a + search-rules change, say) never touches the EFSP. ``revision`` is the + EFSP's own: it changes when the court's name or code lists change. + """ + + jurisdiction = models.CharField(max_length=20) + code = models.CharField(max_length=100) + name = models.CharField(max_length=255) + revision = models.CharField(max_length=64) + data = models.JSONField() + filing_count = models.PositiveIntegerField(default=0) + copied_at = models.DateTimeField() + + class Meta: + constraints = [ + models.UniqueConstraint(fields=["jurisdiction", "code"], name="unique_court_catalog_per_jurisdiction") + ] + + +class FilingCodeJob(models.Model): + """A staff request to resync codes or rebuild indexes, run by the code index worker.""" + + class Kind(models.TextChoices): + RESYNC = "resync", "Resync codes" + REBUILD = "rebuild", "Rebuild code search indexes" + + class Status(models.TextChoices): + QUEUED = "queued", "Queued" + RUNNING = "running", "Running" + SUCCEEDED = "succeeded", "Succeeded" + FAILED = "failed", "Failed" + + kind = models.CharField(max_length=20, choices=Kind.choices) + # Blank for every jurisdiction. + jurisdiction = models.CharField(max_length=20, blank=True) + status = models.CharField(max_length=20, choices=Status.choices, default=Status.QUEUED) + requested_by = models.ForeignKey( + "efile.UserProfile", null=True, blank=True, on_delete=models.SET_NULL, related_name="+" + ) + requested_at = models.DateTimeField(auto_now_add=True) + started_at = models.DateTimeField(null=True, blank=True) + finished_at = models.DateTimeField(null=True, blank=True) + message = models.TextField(blank=True) + + class Meta: + ordering = ["-requested_at"] + + class UserProfile(AbstractUser): """ Extended user profile to store eFile registration information. @@ -32,6 +114,10 @@ class UserProfile(AbstractUser): # so this is the value a new draft is born with, not a lock on it. ai_assistance_opted_out = models.BooleanField(default=False) analytics_excluded = models.BooleanField(default=False) + # The last few ZIP codes this filer used to find a filing code, newest + # first. Filers who file for others often file far from their own address, + # so code search offers these as shortcuts, never as a prefilled value. + recent_case_zips = models.JSONField(default=list, blank=True) # Timestamps created_at = models.DateTimeField(auto_now_add=True) diff --git a/efile_app/efile/services/case_location.py b/efile_app/efile/services/case_location.py new file mode 100644 index 00000000..f90a06c6 --- /dev/null +++ b/efile_app/efile/services/case_location.py @@ -0,0 +1,134 @@ +"""Which courts serve a case ZIP: a location aid, never a jurisdiction ruling. + +A state whose court selector configures a location matcher (MACourts, +VTCourts; see ``court_location``) answers from that package's own rules, and +falls back to counties only for courts the package has no record of. Any +other state goes through counties: a ZIP maps to counties through +``data/zip_counties.json.gz`` (Census ZCTAs, built by +``scripts/build_zip_counties.py``), and a court to counties through its code or +name -- Illinois codes start with the county (``cook:cvd1``), others name a +county or a town. A court neither can place (an appeals court, a statewide +division) stays in every ZIP's results: narrowing must never hide a court that +might hear the case. +""" + +from __future__ import annotations + +import gzip +import json +import re +from functools import lru_cache +from pathlib import Path + +from django.core.cache import cache + +from efile.db_expressions import CourtCode + +DATA = Path(__file__).resolve().parents[1] / "data" +ZIP_PATTERN = re.compile(r"[0-9]{5}(?:-[0-9]{4})?") + + +def _words(text): + return re.findall(r"[a-z0-9]+", text.casefold().replace("’", "").replace("'", "")) + + +@lru_cache(maxsize=1) +def zip_table(): + with gzip.open(DATA / "zip_counties.json.gz", "rt", encoding="utf-8") as handle: + return json.load(handle) + + +def state_code(jurisdiction): + from efile.services.account_profile import default_state_code + + return default_state_code(jurisdiction).upper() + + +def zip_counties(jurisdiction, postal_code): + """Every county a valid ZIP touches, largest share first; [] when none is known.""" + if jurisdiction == "illinois": + # The state's own table also knows PO-box ZIPs that have no Census ZCTA. + from efile.utils.zip_to_county_il import COUNTY_TO_ZIPS_IL + + counties = [name for name, codes in COUNTY_TO_ZIPS_IL.items() if postal_code[:5] in codes] + if counties: + return counties + return zip_table().get(state_code(jurisdiction), {}).get("zips", {}).get(postal_code[:5], []) + + +def location_matcher(jurisdiction): + from efile.services.court_selection import selector_config + + steps = (selector_config(jurisdiction) or {}).get("steps", []) + return next((step["matcher"] for step in steps if step.get("matcher")), "") + + +def locate(index, postal_code): + """The counties and court codes for a case ZIP, or None without one. + + Raises ValueError for a ZIP that is malformed or places nowhere in this + jurisdiction, so the filer is told rather than shown every court. + """ + if not postal_code: + return None + if not ZIP_PATTERN.fullmatch(postal_code): + raise ValueError("Enter a five-digit ZIP code, or clear the ZIP filter.") + counties = zip_counties(index.jurisdiction, postal_code) + matcher = location_matcher(index.jurisdiction) + if matcher: + from efile.services.court_location import courts_for_postal_code + + found = courts_for_postal_code(matcher, postal_code[:5], index_courts(index)) + codes = found and found["matched"] | set(courts_serving(index, counties, only=found["unknown"])) + else: + codes = courts_serving(index, counties) if counties else None + if codes is None: + raise ValueError("We couldn't place this ZIP code here. Check it, or clear it to see every court.") + return {"counties": counties, "courts": sorted(codes)} + + +def _contains(haystack, needle): + return bool(needle) and any(haystack[i : i + len(needle)] == needle for i in range(len(haystack) - len(needle) + 1)) + + +@lru_cache(maxsize=4096) +def court_counties(jurisdiction, code, name): + """The counties a court's code or name places it in; empty when it names none.""" + state = zip_table().get(state_code(jurisdiction), {}) + counties = {county for names in state.get("zips", {}).values() for county in names} + code_keys = {"".join(_words(part)) for part in code.split(":")} + name_words = _words(name) + found = { + county for county in counties if "".join(_words(county)) in code_keys or _contains(name_words, _words(county)) + } + if found: + return frozenset(found) + # Courts named for a town: the last dash-separated part of the name. + town = " ".join(_words(re.split(r"\s+-+\s+", name.strip())[-1])) + return frozenset(state.get("towns", {}).get(town, [])) + + +def index_courts(index): + """Code -> name for every court in an index, saved at refresh or looked up once.""" + if index.vocabulary.get("courts"): + return index.vocabulary["courts"] + key = f"filing-index-courts-v1:{index.pk}:{index.refreshed_at.isoformat()}" + courts = cache.get(key) + if courts is None: + codes = index.paths.annotate(code=CourtCode("court")).values_list("code", flat=True).order_by().distinct() + courts = {} + for code in codes: + path = index.paths.alias(code=CourtCode("court")).filter(code=code).only("court").first() + courts[code] = path.court.get("name", code) + cache.set(key, courts, timeout=None) + return courts + + +def courts_serving(index, counties, only=None): + """Court codes in ``counties``, plus every court whose county is unknown.""" + return sorted( + code + for code, name in index_courts(index).items() + if (only is None or code in only) + and (not (placed := court_counties(index.jurisdiction, code, name)) or placed & set(counties)) + ) diff --git a/efile_app/efile/services/case_type_guidance.py b/efile_app/efile/services/case_type_guidance.py new file mode 100644 index 00000000..437cf163 --- /dev/null +++ b/efile_app/efile/services/case_type_guidance.py @@ -0,0 +1,293 @@ +"""Curated, data-driven case guidance and conservative contextual search facets.""" + +import re +from decimal import Decimal +from functools import lru_cache +from pathlib import Path + +import yaml + + +@lru_cache(maxsize=1) +def guidance_config(): + return yaml.safe_load((Path(__file__).parent.parent / "data/case_type_guidance.yaml").read_text()) + + +@lru_cache(maxsize=256) +def topic_config(jurisdiction, topic, court_name=""): + data = guidance_config() + if jurisdiction not in data["jurisdictions"] or topic not in data["topics"]: + return {} + config = data["topics"][topic] + state = config.get("jurisdictions", {}).get(jurisdiction, {}) + county = {} + for entry in state.get("counties", {}).values(): + if court_name in entry.get("court_names", []) or any( + court_name.startswith(p) for p in entry.get("court_prefixes", []) + ): + county = entry + break + court = state.get("courts", {}).get(court_name, {}) + # Local wording overrides the generated synopsis; classification rules stay + # at the state level so filtering and path descriptions cannot disagree. + local = {key: value for key, value in {**county, **court}.items() if key in ("text", "source")} + return {**config, **state, **local, "_local_text": local.get("text", "")} + + +@lru_cache(maxsize=512) +def matches(pattern, text): + text = re.sub(r"\b(?:excluding|except)\b[^)]*", "", text, flags=re.I) + return bool(re.search(pattern, text, re.I)) + + +@lru_cache(maxsize=32768) +def case_topic(jurisdiction, category, case_type): + # Procedural tracks such as small claims must survive a more specific + # case label (e.g. a contract claim inside the small-claims category). + small = topic_config(jurisdiction, "small_claims") + if small and matches(small["pattern"], f"{category} {case_type}"): + return "small_claims" + for text in (case_type, category): + for key in guidance_config()["topics"]: + config = topic_config(jurisdiction, key) + if config and matches(config["pattern"], text): + return key + return "" + + +@lru_cache(maxsize=32768) +def case_guidance(jurisdiction, category, case_type, filing_name="", court_name=""): + topic = case_topic(jurisdiction, category, case_type) + if not topic: + return {} + config = topic_config(jurisdiction, topic, court_name) + result = {"topic": topic, "source": config["source"]} + text = [] + for facet in config["facets"]: + result[facet["key"]] = "unknown" + for option in facet["options"]: + field = option.get("field", "case") + name = filing_name if field == "filing" else f"{case_type} {filing_name}" if field == "both" else case_type + if matches(option["pattern"], name): + result[facet["key"]] = option["value"] + if option.get("text"): + text.append(option["text"]) + break + for rule in config.get("special", []): + if matches(rule["pattern"], case_type): + text.append(rule["text"]) + result["source"] = rule.get("source", result["source"]) + break + for note in config.get("unknown_notes", []): + if result.get(note["when"]) == "unknown" and result.get(note["if_known"]) != "unknown": + text.append(note["text"]) + result["text"] = config.get("_local_text") or " ".join(text) or config["text"] + return result + + +OTHER = "other" + + +def matches_case_filters(guidance, role="", property_kind="", relief="", *, topic="", filters=None): + choices = filters if filters is not None else {"role": role, "property": property_kind, "relief": relief} + if topic and guidance.get("topic") != topic: + return False + # Amounts are matched against the label (see ``matches_amount``), not guidance. + choices = {key: value for key, value in choices.items() if key != "amount"} + if not any(choices.values()): + return True + # "other" keeps only the case types none of a facet's options name. + return bool(guidance) and all( + not value or guidance.get(key, "unknown") in ((value, "unknown") if value != OTHER else ("unknown",)) + for key, value in choices.items() + ) + + +# Amounts are integer cents. Parse claim brackets, never arbitrary numbers or +# filing fees. Unrecognized labels remain unclassified and are not excluded. +_MONEY = r"\$?\s*(\d[\d,]*(?:\.\d{1,2})?)\s*([kK]?)" + + +def _cents(number, suffix): + return int(Decimal(number.replace(",", "")) * (100000 if suffix else 100)) + + +@lru_cache(maxsize=8192) +def claim_range(name): + if "$" not in name and not re.search(r"\d,\d{3}", name): + return None + if match := re.search(_MONEY + r"\s*(?:to|thru|through|[-–])\s*" + _MONEY, name, re.I): + low, high = _cents(*match.groups()[:2]), _cents(*match.groups()[2:]) + return (low, high) if low <= high else None + if match := re.search(r"(?:up to|not over|not exceeding|under|less than|<=|<)\s*" + _MONEY, name, re.I): + high = _cents(*match.groups()) + exclusive = bool(re.match(r"(?:under|less than|<(?![=]))", match.group(), re.I)) + return 0, high - int(exclusive) + if match := re.search(_MONEY + r"\s*(?:or less|and under)", name, re.I): + return 0, _cents(*match.groups()) + if match := re.search(r"(?:over|more than|greater than|>)\s*" + _MONEY, name, re.I): + return _cents(*match.groups()) + 1, None + return None + + +def matches_amount(label, amount): + if not amount: + return True + bounds = claim_range(label["case_type"]) or claim_range(label["category"]) + if bounds is None: + return True + low, high = (int(part) if part else None for part in amount.split(":")) + return (high is None or bounds[0] <= high) and (bounds[1] is None or low <= bounds[1]) + + +def _money(cents): + return f"${cents / 100:,.2f}".removesuffix(".00") + + +def amount_options(labels): + ranges = set() + small_claims = set() + for label in labels: + bounds = claim_range(label["case_type"]) or claim_range(label["category"]) + if bounds is None: + continue + ranges.add(bounds) + if re.search(r"\bsmall claims?\b", f"{label['category']} {label['case_type']}", re.IGNORECASE): + small_claims.add(bounds) + if len(ranges) < 2: + return [] + cuts = sorted({0, *(low for low, _ in ranges), *(high + 1 for _, high in ranges if high is not None)}) + bands = [] + for i, low in enumerate(cuts): + high = cuts[i + 1] - 1 if i + 1 < len(cuts) else None + members = frozenset(bounds for bounds in ranges if bounds[0] <= low and (bounds[1] is None or low <= bounds[1])) + if not members: + continue + if bands and bands[-1][2] == members and bands[-1][1] == low - 1: + bands[-1] = (bands[-1][0], high, members) + else: + bands.append((low, high, members)) + # Name a band "small claims" only when it sets small claims apart from + # the others, as a civil search spanning both does. + marked = [bool(members & small_claims) for _, _, members in bands] + return [ + { + "value": f"{low}:{high if high is not None else ''}", + "label": ( + f"{_money(low)} or more" + if high is None + else f"Up to {_money(high)}" + if low == 0 + else f"{_money(low)}–{_money(high)}" + ) + + (" (small claims)" if is_small and not all(marked) else ""), + } + for (low, high, _), is_small in zip(bands, marked, strict=True) + ] + + +def _valid_amount(value): + if value == "": + return True + if not isinstance(value, str) or not re.fullmatch(r"\d{1,12}:\d{0,12}", value): + return False + low, high = value.split(":") + return not high or int(low) <= int(high) + + +def validate_case_filters(jurisdiction, topic, choices): + if not isinstance(choices, dict) or len(choices) > 8: + raise ValueError("Choose valid case-type filters.") + if topic in ("", "all"): + # A claim amount narrows any search; other details belong to a case area. + if set(choices) - {"amount"} or not _valid_amount(choices.get("amount", "")): + raise ValueError("Choose a case area before narrowing its details.") + return + config = topic_config(jurisdiction, topic) + if not config: + raise ValueError("Choose a supported case area.") + allowed = { + facet["key"]: {option["value"] for option in facet["options"]} | ({OTHER} if facet.get("other") else set()) + for facet in config["facets"] + } + for key, value in choices.items(): + if not isinstance(value, str): + raise ValueError("Choose valid case-type filters.") + if key == "amount" and _valid_amount(value): + continue + if key not in allowed or value not in allowed[key] | {""}: + raise ValueError("Choose valid case-type filters.") + + +def facet_metadata(jurisdiction, query, labels, topic="", choices=None): + available = {case_topic(jurisdiction, label["category"], label["case_type"]) for label in labels} - {""} + selected = topic + if not selected: + inferred = [key for key in available if matches(topic_config(jurisdiction, key)["query_pattern"], query)] + selected = inferred[0] if len(inferred) == 1 else next(iter(available)) if len(available) == 1 else "" + config = topic_config(jurisdiction, selected) + if config: + available.add(selected) + facets = [] + for facet in config.get("facets", []): + # Keep alternatives visible, but hide dimensions with no known labels. + known = { + case_guidance(jurisdiction, label["category"], label["case_type"], label["name"]).get( + facet["key"], "unknown" + ) + for label in labels + if case_topic(jurisdiction, label["category"], label["case_type"]) == selected + } + if known - {"unknown"} or (choices or {}).get(facet["key"]): + facets.append( + { + "key": facet["key"], + "label": facet["label"], + "options": [{"value": option["value"], "label": option["label"]} for option in facet["options"]] + + ([{"value": OTHER, "label": facet["other"]}] if facet.get("other") else []), + } + ) + # Claim amounts narrow any search whose case types differ by amount -- a + # civil complaint search spanning small claims and larger claims, say -- + # not only one already in small claims. + in_area = selected not in ("", "all") + applicable = [ + label + for label in labels + if (not in_area or case_topic(jurisdiction, label["category"], label["case_type"]) == selected) + and matches_case_filters( + case_guidance(jurisdiction, label["category"], label["case_type"], label["name"]), filters=choices or {} + ) + ] + options = amount_options(applicable) + selected_amount = (choices or {}).get("amount") + if selected_amount and not any(option["value"] == selected_amount for option in options): + low, high = selected_amount.split(":") + options.append( + { + "value": selected_amount, + "label": f"{_money(int(low))}–{_money(int(high))}" if high else f"{_money(int(low))} or more", + } + ) + small_claims = selected == "small_claims" + limit = config.get("usual_limit_cents") if small_claims else None + if limit: + for option in options: + low, high = option["value"].split(":") + if int(low) > limit or not high or int(high) > limit: + option["label"] += " (check small-claims limit)" + if options or selected_amount: + facets.append( + { + "key": "amount", + "label": "How much are you asking for?", + "options": options, + "help": config.get("amount_help", "") if small_claims else "", + "source": config["source"] if small_claims else "", + } + ) + return { + "case_topics": [{"value": key, "label": topic_config(jurisdiction, key)["label"]} for key in sorted(available)], + "case_topic": selected, + "case_facets": facets, + } diff --git a/efile_app/efile/services/court_location.py b/efile_app/efile/services/court_location.py index 5f4423a8..119536e6 100644 --- a/efile_app/efile/services/court_location.py +++ b/efile_app/efile/services/court_location.py @@ -210,3 +210,35 @@ def find_courts(matcher: str, place: str, court_types: list[str], courts: list[d seen.add(court["value"]) results.append({**court, "reason": match["reason"], "matched_name": match["name"]}) return results + + +def courts_for_postal_code(matcher: str, postal_code: str, courts: dict[str, str]) -> dict[str, set[str]] | None: + """Split ``courts`` (code -> name) by what the matcher says about a ZIP. + + ``matched`` serve it; ``unknown`` are courts the matcher has no record of -- + a statewide division, a court added to e-filing after the package's data -- + which the caller places some other way. None when the ZIP places nowhere. + MACourts identifies courts by code; Vermont's units by name. + """ + + if matcher == "macourts": + finder = _finder() + matches = finder.find_by_postal_code(postal_code) + matched = {normalize_court_code(r.tyler_code) for m in matches for r in m.records if r.tyler_code} + known = {normalize_court_code(r.tyler_code) for r in finder.catalog.records if r.tyler_code} + keys = {code: normalize_court_code(code) for code in courts} + elif matcher == "vtcourts": + from vtcourts import Location + + units = _vermont_finder().find_units(Location(postal_code=postal_code, state="Vermont")) + matched = {normalize_court_name(f"{unit.unit} Unit") for unit in units} + keys = {code: normalize_court_name(name) for code, name in courts.items()} + known = {key for key in keys.values() if key.endswith(" unit")} + else: + raise LocationLookupUnavailable(f"No court location lookup is configured for '{matcher}'.") + if not matched: + return None + return { + "matched": {code for code, key in keys.items() if key in matched}, + "unknown": {code for code, key in keys.items() if key not in known}, + } diff --git a/efile_app/efile/services/filing_code_copy.py b/efile_app/efile/services/filing_code_copy.py new file mode 100644 index 00000000..4e4326d6 --- /dev/null +++ b/efile_app/efile/services/filing_code_copy.py @@ -0,0 +1,261 @@ +"""Copy filing codes straight from the EFSP proxy's codes database, then index them. + +The proxy refreshes its codes from Tyler once a day. LITEFile reads the same +tables over a read-only connection -- the queries match the proxy's own +filing-catalog export -- and keeps a copy per court +(``FilingCodeCourtCatalog``). The search index is built from that copy, so a +search-rules change needs only a rebuild, never another trip to the EFSP. + +Every read happens inside one READ ONLY, REPEATABLE READ transaction: the +court list and every court's tables come from the same snapshot, and the +database refuses any write. (Supabase's pooler ignores connection-level +settings, so read-only is set per transaction.) +""" + +from __future__ import annotations + +import psycopg +from django.conf import settings +from django.db import transaction +from django.utils import timezone +from psycopg.rows import dict_row + +from efile.models import FilingCodeCourtCatalog, FilingCodeIndex +from efile.services.court_selection import is_non_filing_court +from efile.services.filing_code_sync import synchronize_index, validate_export + +# Each court's revision changes when its name or the installed version of any of +# its three code lists does: the proxy's own manifest query. +COURTS_SQL = """ +SELECT l.code, l.name, (l.initial ILIKE 'true' OR l.subsequent ILIKE 'true') AS fileable, + count(v.installedversion) AS versions, + md5(jsonb_build_array(l.name, jsonb_object_agg(v.codelist, v.installedversion) + FILTER (WHERE v.codelist IS NOT NULL))::text) AS revision +FROM location l LEFT JOIN installedversion v + ON v.jurisdiction=l.jurisdiction AND v.location=l.code + AND v.codelist IN ('casecategorycodes.zip', 'casetypecodes.zip', 'filingcodes.zip') + AND btrim(v.installedversion) <> '' +WHERE l.jurisdiction=%s +GROUP BY l.code, l.name, l.initial, l.subsequent ORDER BY l.code +""" +CATEGORIES_SQL = """ +SELECT code, name FROM casecategory WHERE jurisdiction=%s AND location=%s + AND ecfcasetype != 'CriminalCase' ORDER BY code, name +""" +CASE_TYPES_SQL = """ +SELECT code, name, casecategory AS case_category, coalesce(lower(initial)='true', false) AS initial +FROM casetype WHERE jurisdiction=%s AND location=%s ORDER BY casecategory, code, name +""" +FILING_TYPES_SQL = """ +SELECT code, name, casecategory AS case_category, casetypeid AS case_type, filingtype AS timing +FROM filing WHERE jurisdiction=%s AND location=%s AND iscourtuseonly='False' + AND filingtype IN ('Initial', 'Subsequent', 'Both') +ORDER BY code, casecategory, casetypeid, filingtype, name +""" + + +class EfspCodesDatabase: + """A read-only snapshot of one jurisdiction's codes in the EFSP database.""" + + def __init__(self, jurisdiction): + if not settings.EFSP_CODES_DATABASE_URL: + raise ValueError("Set EFSP_CODES_DATABASE_URL to copy codes from the EFSP database.") + self.jurisdiction = jurisdiction + + def __enter__(self): + self.connection = psycopg.connect( + settings.EFSP_CODES_DATABASE_URL, + connect_timeout=30, + application_name="litefile-code-copy", + row_factory=dict_row, + ) + try: + self.connection.read_only = True + self.connection.isolation_level = psycopg.IsolationLevel.REPEATABLE_READ + self.snapshot = self.connection.transaction() + self.snapshot.__enter__() + with self.connection.cursor() as cursor: + cursor.execute("SHOW transaction_read_only") + if (cursor.fetchone() or {}).get("transaction_read_only") != "on": + raise ValueError("The EFSP codes database connection is not read only; refusing to continue.") + except BaseException: + # __exit__ never runs when __enter__ fails: close here. + self.connection.close() + raise + return self + + def __exit__(self, *exc_info): + try: + self.snapshot.__exit__(*exc_info) + finally: + self.connection.close() + + def _rows(self, sql, *params): + with self.connection.cursor() as cursor: + cursor.execute(sql, (self.jurisdiction, *params)) + return cursor.fetchall() + + def manifest(self): + courts = {} + for row in self._rows(COURTS_SQL): + # Tyler-internal rows (Illinois' "System", flagged fileable for + # existing cases but holding no code lists) never block a copy. + if is_non_filing_court(row["name"]): + continue + if row["versions"] != 3: + if row["fileable"]: + # As the proxy's export does: a filing court with missing code + # lists means Tyler's update is mid-way. Copy nothing. + raise ValueError(f"Incomplete installed filing catalog for court {row['code']}; try again later.") + continue + courts[row["code"]] = {key: row[key] for key in ("code", "name", "revision")} + if not courts: + raise ValueError("No filing courts in the EFSP codes database; keeping the previous copy.") + return courts + + def court(self, court): + data = { + "court": court, + "categories": self._rows(CATEGORIES_SQL, court["code"]), + "case_types": self._rows(CASE_TYPES_SQL, court["code"]), + "filing_types": self._rows(FILING_TYPES_SQL, court["code"]), + } + validate_export(data) + return data + + +def copy_codes(jurisdiction, *, progress=None, force=(), dry_run=False): + """Copy courts whose EFSP revision changed (plus ``force``); drop courts the EFSP no longer has. + + Each court is saved as it is read, so memory holds one court at a time and + the local database is never locked for the whole copy. A court that + suddenly has no filing types almost always means a half-finished EFSP + update, so the copy stops and keeps the old one. + """ + existing = { + row["code"]: row + for row in FilingCodeCourtCatalog.objects.filter(jurisdiction=jurisdiction).values( + "code", "revision", "filing_count" + ) + } + changed = 0 + with EfspCodesDatabase(jurisdiction) as source: + courts = source.manifest() + if unknown := set(force) - set(courts): + raise ValueError("Not filing courts in the EFSP codes database: " + ", ".join(sorted(unknown))) + for code, court in courts.items(): + old = existing.get(code) + if old and old["revision"] == court["revision"] and code not in force: + continue + data = source.court(court) + count = len(data["filing_types"]) + if old and old["filing_count"] and not count and code not in force: + raise ValueError( + f"{court['name']} ({code}) now has no filing types (had {old['filing_count']}); keeping the " + "previous copy. If that's expected, resync with this court forced." + ) + if progress: + progress(f"{court['name']} ({code}): {old['filing_count'] if old else 'new'} -> {count} filing types") + changed += 1 + if dry_run: + continue + FilingCodeCourtCatalog.objects.update_or_create( + jurisdiction=jurisdiction, + code=code, + defaults={ + "name": court["name"], + "revision": court["revision"], + "data": data, + "filing_count": count, + "copied_at": timezone.now(), + }, + ) + removed = set(existing) - set(courts) + if removed and not dry_run: + FilingCodeCourtCatalog.objects.filter(jurisdiction=jurisdiction, code__in=removed).delete() + if progress: + progress(f"{changed} changed courts, {len(removed)} removed, {len(courts) - changed} unchanged") + return {"courts": len(courts), "changed": changed, "removed": len(removed)} + + +class LocalCopyCatalog: + """The copied courts, in the shape ``synchronize_index`` reads.""" + + def __init__(self, jurisdiction): + self.jurisdiction = jurisdiction + + def manifest(self): + courts = { + row["code"]: row + for row in FilingCodeCourtCatalog.objects.filter(jurisdiction=self.jurisdiction) + .order_by("code") + .values("code", "name", "revision") + } + if not courts: + raise ValueError("No copied codes yet for this jurisdiction. Resync codes first.") + return courts + + def court(self, court): + return FilingCodeCourtCatalog.objects.get(jurisdiction=self.jurisdiction, code=court["code"]).data + + def close(self): + pass + + +def resync(jurisdiction, *, progress=None, force=(), dry_run=False): + """Copy changed courts from the EFSP, then update the index for those courts.""" + copied = copy_codes(jurisdiction, progress=progress, force=force, dry_run=dry_run) + if dry_run: + return copied["changed"] + # An index built from older rules is replaced in full; otherwise only changed courts are. + return synchronize_index(jurisdiction, catalog=LocalCopyCatalog(jurisdiction), progress=progress, force=force) + + +def rebuild(jurisdiction, *, progress=None): + """Rebuild the whole search index from the local copy, without contacting the EFSP.""" + return synchronize_index(jurisdiction, catalog=LocalCopyCatalog(jurisdiction), progress=progress, full=True) + + +def status(jurisdiction): + """What the staff page shows for one jurisdiction.""" + copies = FilingCodeCourtCatalog.objects.filter(jurisdiction=jurisdiction) + index = FilingCodeIndex.objects.filter(jurisdiction=jurisdiction).first() + latest = copies.order_by("-copied_at").values_list("copied_at", flat=True).first() + return { + "jurisdiction": jurisdiction, + "copied_courts": copies.count(), + "copied_at": latest, + "indexed_at": index.refreshed_at if index else None, + "indexed_paths": sum(item.get("count", 0) for item in (index.court_snapshots or {}).values()) if index else 0, + } + + +def run_jobs(jurisdictions, *, log=print): + """Run queued staff requests, oldest first, until none are left.""" + from efile.models import FilingCodeJob + + while True: + job = FilingCodeJob.objects.filter(status=FilingCodeJob.Status.QUEUED).order_by("requested_at").first() + if job is None: + return + # Claim it with a conditional update, which works on SQLite and Postgres alike. + claimed = FilingCodeJob.objects.filter(pk=job.pk, status=FilingCodeJob.Status.QUEUED).update( + status=FilingCodeJob.Status.RUNNING, started_at=timezone.now() + ) + if not claimed: + continue + lines, failed = [], [] + for jurisdiction in [job.jurisdiction] if job.jurisdiction else jurisdictions: + try: + count = (rebuild if job.kind == FilingCodeJob.Kind.REBUILD else resync)(jurisdiction) + lines.append(f"{jurisdiction}: {count} filing paths saved") + except Exception as error: # Report every failure on the job; keep the worker alive. + failed.append(jurisdiction) + lines.append(f"{jurisdiction}: failed; previous index kept. {error}") + log(lines[-1]) + with transaction.atomic(): + FilingCodeJob.objects.filter(pk=job.pk).update( + status=FilingCodeJob.Status.FAILED if failed else FilingCodeJob.Status.SUCCEEDED, + finished_at=timezone.now(), + message="\n".join(lines), + ) diff --git a/efile_app/efile/services/filing_code_search.py b/efile_app/efile/services/filing_code_search.py new file mode 100644 index 00000000..109bc839 --- /dev/null +++ b/efile_app/efile/services/filing_code_search.py @@ -0,0 +1,1177 @@ +"""Deterministic search of complete EFSP paths, with portable local storage. + +Snowball stems and YAML concepts are materialized when refreshing the index. +PostgreSQL uses an indexed simple tsvector; SQLite uses an FTS5 inverted index. +No external service is called while searching. Selection is revalidated live. +""" + +from __future__ import annotations + +import gzip +import hashlib +import json +import re +import time +import unicodedata +from concurrent.futures import ThreadPoolExecutor +from contextlib import closing, nullcontext +from difflib import get_close_matches +from functools import lru_cache +from itertools import islice +from pathlib import Path +from tempfile import NamedTemporaryFile, TemporaryDirectory, TemporaryFile +from threading import Lock, local +from urllib.parse import quote + +import requests +import snowballstemmer +import yaml +from django.conf import settings +from django.contrib.postgres.search import SearchQuery, SearchVector +from django.core.cache import cache +from django.db import connection, transaction +from django.db.models import Case, Count, IntegerField, Max, Q, Value, When +from django.db.models.expressions import RawSQL +from django.utils import timezone +from requests.adapters import HTTPAdapter +from urllib3.util.retry import Retry + +from efile.models import FilingCodeIndex, FilingCodePath +from efile.services.case_location import locate +from efile.services.case_type_guidance import ( + _money, + case_guidance, + claim_range, + facet_metadata, + matches_amount, + matches_case_filters, + validate_case_filters, +) +from efile.services.case_type_guidance import case_topic as topic_of +from efile.services.court_selection import is_non_filing_court +from efile.services.filing_availability import filing_unavailable_message +from efile.services.glossary import glossary_for + +FACETS = ("court", "case_category", "case_type", "filing_type") +RULES_PATH = Path(__file__).resolve().parents[1] / "data" / "filing_code_search.yaml" +DEFINITIONS_PATH = RULES_PATH.with_name("filing_concept_definitions.yaml") +_stemmer_local = local() + + +@lru_cache(maxsize=1) +def rules(): + raw = RULES_PATH.read_bytes() + config = yaml.safe_load(raw) + if config.get("version") != 1 or config.get("stemming") != "english": + raise ValueError("Unsupported filing search rules version or stemmer") + # Bump this prefix whenever tokenization changes incompatibly. + return config, hashlib.sha256(b"filing-search-v1\n" + raw).hexdigest() + + +def words(text): + text = unicodedata.normalize("NFKD", str(text)).encode("ascii", "ignore").decode().lower() + return re.findall(r"[a-z0-9]+", text) + + +def stemmer(): + # Snowball keeps mutable cursor state. ASGI runs Django views in threads. + if not hasattr(_stemmer_local, "english"): + _stemmer_local.english = snowballstemmer.stemmer("english") + return _stemmer_local.english + + +@lru_cache(maxsize=32768) +def stems(text): + return stemmer().stemWords(words(text)) + + +@lru_cache(maxsize=8) +def concept_phrases(jurisdiction): + config, _ = rules() + phrases = [] + for name, concept in config["concepts"].items(): + if jurisdiction not in concept.get("jurisdictions", config["jurisdictions"]): + continue + for term in concept["terms"]: + phrases.append((stems(term), "concept" + re.sub(r"[^a-z0-9]", "", name))) + return sorted(phrases, key=lambda pair: -len(pair[0])) + + +@lru_cache(maxsize=32768) +def search_tokens(text, jurisdiction, *, query=False): + tokens = stems(text) + concepts = set() + consumed = set() + for phrase, concept in concept_phrases(jurisdiction): + for start in range(len(tokens) - len(phrase) + 1): + positions = set(range(start, start + len(phrase))) + if query and consumed & positions: + continue + if tokens[start : start + len(phrase)] == phrase: + concepts.add(concept) + consumed.update(positions) + stop = set(stems(" ".join(rules()[0]["stop_words"]))) + return concepts | { + token for i, token in enumerate(tokens) if token not in stop and (not query or i not in consumed) + } + + +def packed(tokens): + return " " + " ".join(sorted(tokens)) + " " + + +@lru_cache(maxsize=1) +def concept_definitions(): + config = yaml.safe_load(DEFINITIONS_PATH.read_bytes()) + if config.get("version") != 1: + raise ValueError("Unsupported concept definitions version") + return config + + +def concept_meaning(concept, jurisdiction, court_name=""): + """Resolve display text only, with explicit local scope and precedence.""" + config = concept_definitions() + definition = config.get("concepts", {}).get(concept, {}) + result = { + "key": concept, + "label": definition.get("label", concept.replace("_", " ").capitalize()), + "text": "", + "source": "", + "scope": "general", + "scope_name": "", + } + + def apply(override, scope, name=""): + if "text" in override: + # A local definition must not silently inherit a statewide source + # that may not support the local meaning. + result.update(text=override["text"], source=override.get("source", ""), scope=scope, scope_name=name) + if "label" in override: + result["label"] = override["label"] + + apply(definition.get("default", {}), "general") + state = definition.get("jurisdictions", {}).get(jurisdiction, {}) + apply(state, "state", jurisdiction.replace("_", " ").title()) + if court_name: + normalized = court_name.casefold().strip() + for county, selectors in config.get("county_courts", {}).get(jurisdiction, {}).items(): + exact = any(normalized == name.casefold().strip() for name in selectors.get("names", [])) + prefix = any(normalized.startswith(name.casefold()) for name in selectors.get("prefixes", [])) + if exact or prefix: + apply(state.get("counties", {}).get(county, {}), "county", county) + break + for name, override in state.get("courts", {}).items(): + if normalized == name.casefold().strip(): + apply(override, "court", court_name) + break + return result + + +def search_context(jurisdiction, query, labels, *, court_name="", corrected=(), resolved_query=None): + """Explain discovery, without claiming a matched topic defines a document. + + A group contains filing labels only; a path also provides its case/court + labels. Query concepts can match that associated context, not the document. + """ + query_tokens = search_tokens(resolved_query if resolved_query is not None else query, jurisdiction, query=True) + name_tokens = set().union(*(search_tokens(name, jurisdiction, query=True) for _, name in labels)) + related = [] + meanings = [] + for key, concept in rules()[0]["concepts"].items(): + if jurisdiction not in concept.get("jurisdictions", rules()[0]["jurisdictions"]): + continue + token = "concept" + re.sub(r"[^a-z0-9]", "", key) + if token in query_tokens | name_tokens: + meanings.append(concept_meaning(key, jurisdiction, court_name)) + if token in query_tokens: + related.append(key) + normalized_query = " ".join(words(query)) + filing_names = [name for facet, name in labels if facet == "filing_type"] + title = filing_names[0] if filing_names else "" + literal = normalized_query and normalized_query in " ".join(words(title)) + query_words = set(stems(query)) - set(stems(" ".join(rules()[0]["stop_words"]))) + title_words = set(stems(title)) + overlap = bool(query_words) and len(query_words & title_words) / len(query_words) >= 0.6 + reason = None + if query and not literal and not overlap: + if any(normalized_query and normalized_query in " ".join(words(name)) for name in filing_names[1:]): + kind = "alternate_name" + elif related: + kind = "related_concept" + elif corrected: + kind = "spelling" + else: + kind = "context" + reason = {"kind": kind, "query": query, "concepts": related, "corrected_terms": list(corrected)} + return {"concepts": meanings, "match_reason": reason} + + +@lru_cache(maxsize=8) +def explanation_rules(jurisdiction, initial): + compiled = [] + facets = set() + for rule in rules()[0]["explanations"]: + if jurisdiction not in rule["jurisdictions"]: + continue + if "initial" in rule and rule["initial"] != initial: + continue + constraints = {facet: {tuple(words(label)) for label in rule[facet]} for facet in FACETS if facet in rule} + facets.update(constraints) + compiled.append((constraints, rule["text"], rule["source"])) + return tuple(facet for facet in FACETS if facet in facets), compiled + + +@lru_cache(maxsize=32768) +def _explanation_for_names(jurisdiction, initial, names): + facets, compiled = explanation_rules(jurisdiction, initial) + normalized = dict(zip(facets, (tuple(words(name)) for name in names), strict=True)) + for constraints, text, source in compiled: + if all(normalized[facet] in labels for facet, labels in constraints.items()): + return text, source + return "", "" + + +def explanation_for(jurisdiction, path, initial): + # Catalogs repeat the same filing labels across thousands of valid paths. + # Cache only the facets that rules actually constrain, preserving exact + # context-specific matches without re-normalizing every rule for every row. + facets, _ = explanation_rules(jurisdiction, initial) + return _explanation_for_names(jurisdiction, initial, tuple(path[facet]["name"] for facet in facets)) + + +def source_url(): + return settings.EFSP_URL.rstrip("/") + + +class CodeCatalog: + """Strict API reader: a failed branch must not publish a partial index.""" + + def __init__(self, jurisdiction, *, timeout=(10, 60)): + if jurisdiction not in rules()[0]["jurisdictions"]: + raise ValueError("Unsupported jurisdiction") + self.base = f"{source_url()}/jurisdictions/{jurisdiction}/codes/courts/" + self._local = local() + self._sessions = [] + self._session_lock = Lock() + self.timeout = timeout + + @property + def session(self): + # Keep cookies and connection pools private to each catalog thread. + if not hasattr(self._local, "session"): + session = requests.Session() + retry = Retry( + total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504], allowed_methods=["GET"] + ) + session.mount("https://", HTTPAdapter(max_retries=retry)) + session.mount("http://", HTTPAdapter(max_retries=retry)) + self._local.session = session + with self._session_lock: + self._sessions.append(session) + return self._local.session + + def close(self): + for session in self._sessions: + session.close() + + def get(self, suffix="", **params): + # Adapter retries don't cover failures while Requests consumes a + # streamed response body. Retry that whole read as well. + for attempt in range(4): + response = None + try: + response = self.session.get(self.base + suffix, params=params, timeout=self.timeout) + response.raise_for_status() + data = response.json() + break + except (requests.Timeout, requests.ConnectionError): + if attempt == 3: + raise + time.sleep(2**attempt) + finally: + if response is not None: + response.close() + if not isinstance(data, list): + raise ValueError("Court code API did not return a list") + result = {} + for item in data: + if not isinstance(item, dict) or item.get("code") is None or not item.get("name"): + raise ValueError("Court code API returned an invalid option") + code = str(item["code"]) + if not code: + raise ValueError("Court code API returned an empty code") + result[code] = {"code": code, "name": str(item["name"])} + return list(result.values()) + + def courts(self): + return [c for c in self.get(fileable_only="false", with_names="true") if not is_non_filing_court(c["name"])] + + def categories(self, court, initial=True): + return self.get( + quote(court, safe="") + "/categories", + fileable_only="true", + timing="Initial" if initial else "Subsequent", + ) + + def types(self, court, category, initial=True): + return self.get( + quote(court, safe="") + "/case_types/", + category_id=category, + timing="Initial" if initial else "Subsequent", + ) + + def filings(self, court, category, case_type, initial): + return self.get( + quote(court, safe="") + "/filing_types/", + category_id=category, + type_id=case_type, + initial=str(initial).lower(), + ) + + +def catalog_entries(catalog, jurisdiction, progress, cache_dir=None): + # Stage courts concurrently so a slow court does not idle the whole build. + # A shared pool bounds filing-type requests across all four court readers. + # Per-court files keep memory bounded even when results finish out of order. + # Every branch still must succeed before publishing the snapshot. + digest = hashlib.sha256(f"{source_url()}:{rules()[1]}:{jurisdiction}".encode()).hexdigest() + location = Path(cache_dir) / digest if cache_dir else None + with nullcontext(location) if location else TemporaryDirectory(prefix="filing-codes-") as directory: + Path(directory).mkdir(parents=True, exist_ok=True) + with ThreadPoolExecutor(max_workers=8) as filings, ThreadPoolExecutor(max_workers=4) as courts: + + def stage(item): + _, court = item + path = Path(directory) / (hashlib.sha256(court["code"].encode()).hexdigest() + ".json.gz") + if location and path.exists() and time.time() - path.stat().st_mtime < 21600: + if progress: + progress(f"{court['name']} (resuming completed court)") + return path + if progress: + progress(court["name"]) + with NamedTemporaryFile(dir=directory, delete=False) as temporary: + temporary_path = Path(temporary.name) + try: + with gzip.open(temporary_path, "wt", encoding="utf-8", compresslevel=1) as output: + for entry in _court_entries(catalog, jurisdiction, court, filings): + output.write(json.dumps(entry) + "\n") + temporary_path.replace(path) + finally: + temporary_path.unlink(missing_ok=True) + return path + + for path in courts.map(stage, enumerate(catalog.courts())): + with gzip.open(path, "rt", encoding="utf-8") as snapshot: + for line in snapshot: + yield json.loads(line) + if not location: + path.unlink() + + +def _court_entries(catalog, jurisdiction, court, executor): + for initial in (True, False): + for category in catalog.categories(court["code"], initial): + + def fetch_filings(case_type, court=court, category=category, initial=initial): + return case_type, catalog.filings(court["code"], category["code"], case_type["code"], initial) + + # Consume the whole map before advancing these parent facets. + for case_type, filings in executor.map( + fetch_filings, catalog.types(court["code"], category["code"], initial) + ): + for filing in filings: + path = dict(zip(FACETS, (court, category, case_type, filing), strict=True)) + terms = {facet: search_tokens(path[facet]["name"], jurisdiction) for facet in FACETS} + all_terms = set.union(*terms.values()) + all_terms.update(t for facet in FACETS for t in stems(path[facet]["code"])) + explanation, source = explanation_for(jurisdiction, path, initial) + yield dict( + **path, + initial=initial, + search_text=packed(all_terms), + filing_terms=packed(terms["filing_type"]), + case_terms=packed(terms["case_category"] | terms["case_type"]), + explanation=explanation, + explanation_source=source, + ) + + +def refresh_index(jurisdiction, *, progress=None, cache_dir=None, transport=None, force=(), dry_run=False): + """Synchronize changed court exports; the expensive old crawler requires explicit opt-in.""" + mode = transport or getattr(settings, "FILING_CODE_SYNC_MODE", "bulk") + if mode == "database": + from efile.services.filing_code_copy import resync + + return resync(jurisdiction, progress=progress, force=force, dry_run=dry_run) + if mode == "bulk": + from efile.services.filing_code_sync import synchronize_index + + return synchronize_index(jurisdiction, progress=progress, force=force, dry_run=dry_run) + if mode != "legacy": + raise ValueError("FILING_CODE_SYNC_MODE must be 'database', 'bulk', or 'legacy'") + if force or dry_run: + raise ValueError("Forcing courts and dry runs need the bulk sync") + return _refresh_index_legacy(jurisdiction, progress=progress, cache_dir=cache_dir) + + +def _refresh_index_legacy(jurisdiction, *, progress=None, cache_dir=None): + """Replace a jurisdiction atomically after *every* branch was fetched. + + No database transaction is held open during the slow network traversal. + A later-started refresh cannot be overwritten by an older overlapping run. + """ + started = timezone.now() + vocabulary = set() + indexed_terms = set() + for concept in rules()[0]["concepts"].values(): + if jurisdiction in concept.get("jurisdictions", rules()[0]["jurisdictions"]): + vocabulary.update(word for term in concept["terms"] for word in words(term)) + count = 0 + courts = {} + # Large catalogs share many names but still have many distinct paths. Stage + # on disk, then insert bounded batches: memory does not grow with row count. + with closing(CodeCatalog(jurisdiction)) as catalog, TemporaryFile(mode="w+b") as staging: + with gzip.open(staging, "wt", encoding="utf-8", compresslevel=1) as output: + for entry in catalog_entries(catalog, jurisdiction, progress, cache_dir): + output.write(json.dumps(entry) + "\n") + indexed_terms.update(entry["search_text"].split()) + courts.setdefault(entry["court"]["code"], entry["court"]["name"]) + vocabulary.update(word for facet in FACETS for word in words(entry[facet]["name"])) + count += 1 + if not count: + raise ValueError("No filing paths returned; keeping the previous index") + if progress: + progress(f"Saving {count} filing paths") + staging.seek(0) + with gzip.open(staging, "rt", encoding="utf-8") as snapshot, transaction.atomic(): + index, _ = FilingCodeIndex.objects.get_or_create( + jurisdiction=jurisdiction, + defaults={"source_url": source_url(), "refreshed_at": started, "rules_digest": rules()[1]}, + ) + index = FilingCodeIndex.objects.select_for_update().get(pk=index.pk) + if index.refreshed_at > started: + return 0 + index.paths.all().delete() + saved = 0 + while batch := [FilingCodePath(index=index, **json.loads(line)) for line in islice(snapshot, 500)]: + FilingCodePath.objects.bulk_create(batch, batch_size=500) + saved += len(batch) + if progress and saved % 100000 == 0: + progress(f"Saved {saved} of {count} filing paths") + index.source_url = source_url() + index.refreshed_at = started + index.rules_digest = rules()[1] + index.vocabulary = {"words": sorted(vocabulary), "tokens": sorted(indexed_terms), "courts": courts} + index.court_snapshots = {} + index.save() + return count + + +def current_index(jurisdiction): + return FilingCodeIndex.objects.filter( + jurisdiction=jurisdiction, + source_url=source_url(), + rules_digest=rules()[1], + ).first() + + +def case_description(path): + """Use existing case-specific explanations or normalized case meanings.""" + jurisdiction = path.index.jurisdiction + for rule in rules()[0]["explanations"]: + if "case_type" not in rule or jurisdiction not in rule["jurisdictions"]: + continue + if "initial" in rule and rule["initial"] != path.initial: + continue + if all( + tuple(words(getattr(path, facet)["name"])) in {tuple(words(name)) for name in rule[facet]} + for facet in FACETS + if facet in rule + ): + return rule["text"] + guidance = case_guidance( + jurisdiction, path.case_category["name"], path.case_type["name"], path.filing_type["name"], path.court["name"] + ) + if guidance.get("text"): + return guidance["text"] + meanings = search_context(jurisdiction, "", [("case_type", path.case_type["name"])], court_name=path.court["name"])[ + "concepts" + ] + return next((meaning["text"] for meaning in meanings if meaning.get("text")), "") + + +JURY_SIZE = re.compile(r"\(\s*jury\s*-\s*(\d+)\s*\)", re.IGNORECASE) +NON_JURY = re.compile(r"\bnon\s*-?\s*jury\b", re.IGNORECASE) +JURY = re.compile(r"\bjury\b", re.IGNORECASE) +SELF_REPRESENTED = re.compile(r"self\s*-?\s*represented|\bSRL\b|\bpro se\b", re.IGNORECASE) +GOVERNMENT = re.compile(r"\(\s*govn'?t\s*\)|\bgovernment(al)? (filer|entity|agency)\b", re.IGNORECASE) + + +def path_qualifiers(category, case_type, filing_name): + """Choices some courts code as separate case or filing types, in plain values. + + Cook County, for one, has a separate code for each mix of jury size, + government filer, self-represented litigant, and claim amount. The court + step asks only the ones its paths actually differ on. + """ + case_text = f"{category} {case_type}" + if size := JURY_SIZE.search(filing_name): + jury = size.group(1) + elif NON_JURY.search(case_text) or NON_JURY.search(filing_name): + jury = "none" + elif JURY.search(case_text) or JURY.search(filing_name): + jury = "jury" + else: + jury = "none" + bounds = claim_range(case_type) or claim_range(category) + amount = None + if bounds: + low, high = bounds + amount = { + "value": f"{low}:{high if high is not None else ''}", + "label": f"{_money(low)} to {_money(high)}" if high is not None else f"More than {_money(low - 1)}", + } + return { + "jury": jury, + "representation": "self" if SELF_REPRESENTED.search(case_text) else "lawyer", + "government": "yes" if GOVERNMENT.search(filing_name) else "no", + "amount": amount, + } + + +def serialize_path(path, *, query="", corrected=(), resolved_query=None): + return { + "qualifiers": path_qualifiers(path.case_category["name"], path.case_type["name"], path.filing_type["name"]), + "filing_label": unqualified_label(path.filing_type["name"]), + "glossary": glossary_for(path.index.jurisdiction, [path.filing_type["name"], path.case_type["name"]]), + "id": path.pk, + **{facet: getattr(path, facet) for facet in FACETS}, + "initial": path.initial, + "explanation": path.explanation, + "explanation_source": path.explanation_source, + "facets": filing_facets(path.filing_type["name"], path.index.jurisdiction), + "case_description": case_description(path), + "case_guidance": case_guidance( + path.index.jurisdiction, + path.case_category["name"], + path.case_type["name"], + path.filing_type["name"], + path.court["name"], + ), + "case_context": case_context_label(path.case_category["name"], path.case_type["name"]), + **search_context( + path.index.jurisdiction, + query, + [(facet, getattr(path, facet)["name"]) for facet in ("filing_type", "case_type", "case_category", "court")], + court_name=path.court["name"], + corrected=corrected, + resolved_query=resolved_query, + ), + "unavailable": filing_unavailable_message( + path.index.jurisdiction, + path.court["code"], + case_category=path.case_category["name"], + case_type=path.case_type["name"], + filing_types=[path.filing_type["name"]], + ), + } + + +def corrected_search_query(index, query): + vocabulary = set(index.vocabulary["words"]) + indexed_terms = set(index.vocabulary["tokens"]) + corrected = [] + query_words = words(query) + # Correct before stemming and phrase expansion. A typo in "eviction" can + # prevent Snowball from recognizing its suffix; the full word must be in + # the vocabulary even if this state's labels only say "summary process". + for position, word in enumerate(query_words): + if len(word) < 5 or not word.isalpha() or word in vocabulary or set(stems(word)) <= indexed_terms: + continue + matches = get_close_matches(word, sorted(vocabulary), n=1, cutoff=0.86) + if matches: + query_words[position] = matches[0] + corrected.append(matches[0]) + return " ".join(query_words), corrected + + +def matching_paths(index, query, initial): + resolved_query, corrected = corrected_search_query(index, query) + tokens = search_tokens(resolved_query, index.jurisdiction, query=True) + if not tokens: + return index.paths.none().annotate(score=Value(0)), [] + paths = index.paths.filter(initial=initial) + if connection.vendor == "postgresql": + search = SearchQuery(" ".join(sorted(tokens)), config="simple") + paths = paths.annotate(document=SearchVector("search_text", config="simple")).filter(document=search) + elif connection.vendor == "sqlite": + # Look up candidate row IDs before ranking/grouping. LIKE '% token %' + # scanned every path in the jurisdiction (millions for Illinois). + # Quote each normalized token and bind the MATCH expression as data; + # user punctuation must never become FTS operators or SQL syntax. + terms = " AND ".join('"' + token.replace('"', '""') + '"' for token in sorted(tokens)) + expression = f'index_id : "{int(index.pk)}" AND initial : "{int(initial)}" AND search_text : ({terms})' + paths = paths.filter( + pk__in=RawSQL( + "SELECT rowid FROM filing_code_search_fts WHERE filing_code_search_fts MATCH %s", + (expression,), + ) + ) + else: + for token in sorted(tokens): + paths = paths.filter(search_text__contains=f" {token} ") + scores = { + "filing_terms": Value(0, output_field=IntegerField()), + "case_terms": Value(0, output_field=IntegerField()), + } + for token in sorted(tokens): + for field, weight in (("filing_terms", 8), ("case_terms", 3)): + clause = Q(**{f"{field}__contains": f" {token} "}) + scores[field] += Case(When(clause, then=Value(weight)), default=Value(0), output_field=IntegerField()) + paths = paths.annotate(name_score=scores["filing_terms"]) + return paths.annotate(score=scores["filing_terms"] + scores["case_terms"]), corrected + + +def search_paths(index, query, *, initial=True, offset=0, limit=20, names=None, court=None, contexts=None, courts=()): + paths, corrected = matching_paths(index, query, initial) + resolved_query = corrected_search_query(index, query)[0] if corrected else query + if names is not None: + paths = paths.filter(filing_type__name__in=names) + if court: + paths = paths.filter(court__code=court) + if courts: + paths = paths.filter(court__code__in=courts) + if contexts is not None: + paths = paths.filter(context_filter(contexts)) + total = paths.count() + paths = paths.order_by("-score", "court__name", "case_type__name", "filing_type__name", "pk") + return { + "results": [ + serialize_path(path, query=query, corrected=corrected, resolved_query=resolved_query) + for path in paths.select_related("index")[offset : offset + limit] + ], + "total": total, + "corrected_terms": corrected, + } + + +def clean_filing_label(name): + name = re.sub(r"^\s*e[\s-]?filed\s*[-:–—]?\s*", "", name, flags=re.IGNORECASE) + return re.sub(r"\s+filed[.\s]*$", "", name, flags=re.IGNORECASE).strip(" .-") + + +# Qualifiers some courts code as separate filings of the same document, such as +# Cook County's "Complaint / Petition - Fraud (Jury - 12) (Govn't) - Fee". The +# filer still picks the exact one in the court step; search shows it once. +FILING_QUALIFIERS = re.compile( + r"\((?:jury[^)]*|govn'?t|government|no fee|fee)\)|\s+-\s+(?:no\s+)?fee\s*$", + re.IGNORECASE, +) + + +def unqualified_label(name): + previous = None + name = clean_filing_label(name) + while previous != name: + previous, name = name, clean_filing_label(FILING_QUALIFIERS.sub("", name)) + return re.sub(r"\s{2,}", " ", name) + + +def filing_group_key(name, jurisdiction): + # Group discovery results using the same curated synonyms and stemming as + # search. These are related labels, not interchangeable court codes: the + # filer still chooses and validates an exact court/category/type path. + label = " ".join(words(unqualified_label(name))) + aliases = { + "answer to complaint": "answer", + "response to complaint": "answer", + "civil complaint": "complaint", + "complaint civil": "complaint", + "motion to the court": "motion", + } + label = aliases.get(label, label) + terms = search_tokens(label, jurisdiction, query=True) or set(words(name)) + parts = words(label) + # Preserve the document's role: "Motion to dismiss" is not a "Dismissal + # of motion", even though a bag of search words would make them identical. + # Allow subject-before-document labels like "Eviction complaint". + role = parts[0] if parts else "" + document_roles = { + "complaint", + "petition", + "answer", + "motion", + "order", + "notice", + "affidavit", + "dismissal", + "application", + "summons", + } + if ( + role not in document_roles + and parts + and parts[-1] in {"complaint", "petition", "answer", "order", "affidavit", "summons"} + ): + role = parts[-1] + if role not in document_roles: + role = "" + return hashlib.sha256((role + ":" + " ".join(sorted(terms))).encode()).hexdigest()[:24] + + +@lru_cache(maxsize=1) +def facet_rules(): + config = yaml.safe_load(RULES_PATH.with_name("filing_code_facets.yaml").read_bytes()) + if config.get("version") != 1: + raise ValueError("Unsupported filing facet rules version") + return [(re.compile(rule["pattern"], re.IGNORECASE), rule) for rule in config["rules"]] + + +@lru_cache(maxsize=1) +def action_rules(): + config = yaml.safe_load(RULES_PATH.with_name("filing_code_facets.yaml").read_bytes()) + return [(re.compile(rule["pattern"], re.IGNORECASE), rule) for rule in config.get("actions", [])] + + +@lru_cache(maxsize=65536) +def filing_action(name): + """What a filing does, in plain words, guessed from its name; "" when unknown.""" + label = " ".join(words(clean_filing_label(name))) + return next((rule["key"] for pattern, rule in action_rules() if pattern.search(label)), "") + + +def action_options(groups): + counts = {} + for group in groups: + for action in {filing_action(label["name"]) for label in group["_labels"]} - {""}: + counts[action] = counts.get(action, 0) + 1 + return [ + {"value": rule["key"], "label": rule["label"], "count": counts[rule["key"]]} + for _, rule in action_rules() + if rule["key"] in counts + ] + + +def category_options(groups): + """Case categories (Cook's "Civil", "Probate", ...) the groups are filed under, most used first.""" + counts = {} + for group in groups: + for name in {label["category"] for label in group["_labels"]}: + key = tuple(words(name)) + counts.setdefault(key, [name, 0])[1] += 1 + return [ + {"value": name, "label": name, "count": count} + for name, count in sorted(counts.values(), key=lambda item: (-item[1], item[0].casefold())) + ] + + +def strong_matches(groups): + """How many leading groups match by filing name rather than only by case type. + + Groups sort by name score first, so these lead. A name match has to be at + least half as strong as the best one, so a two-word search does not count a + one-word partial match beside an exact one. + """ + top = max((group["name_rank"] for group in groups), default=0) + return sum(1 for group in groups if top and group["name_rank"] >= top / 2) + + +def filing_facets(name, jurisdiction): + label = " ".join(words(clean_filing_label(name))) + result = {"purpose": "unknown", "document": "unknown"} + for pattern, rule in facet_rules(): + if jurisdiction not in rule.get("jurisdictions", rules()[0]["jurisdictions"]): + continue + if pattern.search(label): + for facet in result: + if facet in rule: + result[facet] = rule[facet] + return result + return result + + +def case_context_label(category, case_type): + return case_type if words(category) == words(case_type) else f"{category} › {case_type}" + + +# Parts of a case-type name that only say jury, self-representation, or the +# small-claims band: the court step asks about those, so step 1 counts and +# shows "Personal Injury Complaint" once instead of once per variant. +CASE_QUALIFIERS = re.compile( + r"\s*-?\s*\b(?:non\s*-?\s*)?jury(?: demand)?\b" + r"|\s*-?\s*\b(?:self\s*-?\s*represented litigant|srl)\b" + r"|\s*-?\s*\bsmall claims\b\s*-?\s*\$[\d,]+\s*to\s*\$[\d,]+", + re.IGNORECASE, +) + + +def unqualified_case_type(case_type): + return re.sub(r"\s*-\s*$", "", CASE_QUALIFIERS.sub("", case_type)).strip() or case_type + + +def display_case_context(category, case_type): + return case_context_label(category, unqualified_case_type(case_type)) + + +def case_context_key(category, case_type): + return hashlib.sha256(json.dumps([words(category), words(case_type)]).encode()).hexdigest()[:24] + + +def context_filter(labels): + condition = Q(pk__in=[]) + for category, case_type in {(label["category"], label["case_type"]) for label in labels}: + condition |= Q(case_category__name=category, case_type__name=case_type) + return condition + + +def case_contexts(labels): + contexts = {} + for label in labels: + category, case_type = label["category"], label["case_type"] + key = (tuple(words(category)), tuple(words(unqualified_case_type(case_type)))) + context = contexts.setdefault(key, {"label": display_case_context(category, case_type), "count": 0, "rank": 0}) + context["count"] += label["path_count"] + context["rank"] = max(context["rank"], label["rank"]) + return [ + item["label"] + for item in sorted( + contexts.values(), key=lambda item: (-item["rank"], -item["count"], item["label"].casefold()) + ) + ] + + +def filter_filing_groups( + groups, + jurisdiction, + purpose="", + document="", + role="", + property_kind="", + relief="", + case_topic="", + case_filters=None, + action="", + category="", +): + selected = [] + for group in groups: + labels = [] + for label in group["_labels"]: + guidance = case_guidance(jurisdiction, label["category"], label["case_type"], label["name"]) + if not matches_case_filters( + guidance, role, property_kind, relief, topic=case_topic, filters=case_filters + ) or not matches_amount(label, (case_filters or {}).get("amount", "")): + continue + facets = filing_facets(label["name"], jurisdiction) + if action and filing_action(label["name"]) != action: + continue + if category and words(label["category"]) != words(category): + continue + if (not purpose or facets["purpose"] == purpose) and (not document or facets["document"] == document): + labels.append(label) + if not labels: + continue + names = sorted( + {label["name"] for label in labels}, key=lambda name: (len(name), name.isupper(), name.casefold()) + ) + distinct = {} + for name in names: + distinct.setdefault(tuple(words(unqualified_label(name))), unqualified_label(name)) + contexts = case_contexts(labels) + selected.append( + { + **group, + "_labels": labels, + "_names": names, + "variants": list(distinct.values()), + "name": unqualified_label(names[0]), + "path_count": sum(label["path_count"] for label in labels), + "rank": max(label["rank"] for label in labels), + "name_rank": max(label["name_rank"] for label in labels), + "case_contexts": contexts[:8], + "case_context_count": len(contexts), + "case_context_options": list( + { + case_context_key(label["category"], label["case_type"]): { + "key": case_context_key(label["category"], label["case_type"]), + "label": display_case_context(label["category"], label["case_type"]), + } + for label in labels + if display_case_context(label["category"], label["case_type"]) in contexts[:8] + }.values() + ), + "facets": { + facet: values.pop() + if len(values := {filing_facets(name, jurisdiction)[facet] for name in names}) == 1 + else "unknown" + for facet in ("purpose", "document") + }, + } + ) + return sorted( + selected, + key=lambda group: ( + -group["name_rank"], + -group["rank"], + -group["path_count"], + len(search_tokens(group["name"], jurisdiction, query=True)), + group["name"].casefold(), + ), + ) + + +def filing_groups(index, query, initial, courts=()): + fingerprint = hashlib.sha256(f"{index.pk}:{index.refreshed_at}:{initial}:{query}:{courts}".encode()).hexdigest() + key = f"filing-groups-v7:{fingerprint}" + cached = cache.get(key) + if cached is not None: + return cached + paths, corrected = matching_paths(index, query, initial) + if courts: + paths = paths.filter(court__code__in=courts) + labels = ( + paths.order_by() + .values("filing_type__name", "case_category__name", "case_type__name") + .annotate(path_count=Count("pk"), rank=Max("score"), name_rank=Max("name_score")) + ) + groups = {} + for label in labels: + name = label["filing_type__name"] + group_key = filing_group_key(name, index.jurisdiction) + group = groups.setdefault( + group_key, {"key": group_key, "variants": [], "path_count": 0, "rank": 0, "name_rank": 0, "_labels": []} + ) + group["_labels"].append( + { + "name": name, + "path_count": label["path_count"], + "rank": label["rank"], + "name_rank": label["name_rank"], + "category": label["case_category__name"], + "case_type": label["case_type__name"], + } + ) + group["variants"].append(name) + group["path_count"] += label["path_count"] + group["rank"] = max(group["rank"], label["rank"]) + group["name_rank"] = max(group["name_rank"], label["name_rank"]) + for group in groups.values(): + group["variants"] = sorted( + set(group["variants"]), key=lambda name: (len(name), name.isupper(), name.casefold()) + ) + group["_names"] = group["variants"][:] + distinct = {} + for name in group["variants"]: + distinct.setdefault(tuple(words(unqualified_label(name))), unqualified_label(name)) + group["variants"] = list(distinct.values()) + group["name"] = group["variants"][0] + result = ( + sorted( + groups.values(), + key=lambda group: ( + -group["name_rank"], + -group["rank"], + -group["path_count"], + len(search_tokens(group["name"], index.jurisdiction, query=True)), + group["name"].casefold(), + ), + ), + corrected, + ) + cache.set(key, result, timeout=300) + return result + + +def count_groups(index, query, initial, courts, *, purpose, document, topic, case_filters, action, category): + """How many filing types the same search and answers find in the other case stage.""" + groups = filing_groups(index, query, initial, courts)[0] + groups = filter_filing_groups(groups, index.jurisdiction, purpose, document) + groups = filter_filing_groups(groups, index.jurisdiction, case_topic=topic, case_filters=case_filters) + if action: + groups = filter_filing_groups(groups, index.jurisdiction, action=action) + if category: + groups = filter_filing_groups(groups, index.jurisdiction, category=category) + return len(groups) + + +def search_grouped_paths( + index, + query, + *, + initial=True, + offset=0, + limit=20, + group_key="", + court="", + purpose="", + document="", + context="", + postal_code="", + role="", + property_kind="", + relief="", + case_topic="", + case_filters=None, + action="", + category="", + strong_only=False, +): + location = locate(index, postal_code) or {"counties": [], "courts": []} + counties, courts = location["counties"], tuple(location["courts"]) + groups, corrected = filing_groups(index, query, initial, courts) + resolved_query = corrected_search_query(index, query)[0] if corrected else query + case_filters = case_filters or {} + if any((role, property_kind, relief)): + case_topic = "eviction" + case_filters = { + key: value for key, value in {"role": role, "property": property_kind, "relief": relief}.items() if value + } + validate_case_filters(index.jurisdiction, case_topic, case_filters) + groups = filter_filing_groups(groups, index.jurisdiction, purpose, document) + metadata = facet_metadata( + index.jurisdiction, query, [label for group in groups for label in group["_labels"]], case_topic, case_filters + ) + selected_topic = "" if metadata["case_topic"] == "all" else metadata["case_topic"] + groups = filter_filing_groups(groups, index.jurisdiction, case_topic=selected_topic, case_filters=case_filters) + topic_counts = {} + for group in filing_groups(index, query, initial, courts)[0] if metadata["case_topics"] else []: + for topic in { + topic_of(index.jurisdiction, label["category"], label["case_type"]) for label in group["_labels"] + }: + topic_counts[topic] = topic_counts.get(topic, 0) + 1 + for item in metadata["case_topics"]: + item["count"] = topic_counts.get(item["value"], 0) + metadata["case_topics"].sort(key=lambda item: (-item["count"], item["label"].casefold())) + actions = action_options(groups) + if action: + groups = filter_filing_groups(groups, index.jurisdiction, action=action) + categories = category_options(groups) + if category: + groups = filter_filing_groups(groups, index.jurisdiction, category=category) + if not group_key: + strong = strong_matches(groups) + shown = groups[:strong] if strong_only and strong else groups + # Sealing an eviction, say, is filed in the existing case: a filer + # searching new-case filings is told the other list has matches. + other_stage_total = ( + count_groups( + index, + query, + not initial, + courts, + purpose=purpose, + document=document, + topic=selected_topic, + case_filters=case_filters, + action=action, + category=category, + ) + if not offset + else None + ) + return { + "groups": [ + { + **{key: value for key, value in group.items() if not key.startswith("_")}, + # Only the names shown: filing and case-type names without the + # qualifiers step 1 drops, and not categories (Cook files every + # Law-division injury under "Personal Injury/Wrongful Death"). + "glossary": glossary_for( + index.jurisdiction, + [ + group["name"], + *sorted({unqualified_case_type(label["case_type"]) for label in group["_labels"]}), + ], + ), + **search_context( + index.jurisdiction, + query, + [("filing_type", name) for name in [group["name"], *group["variants"]]], + corrected=corrected, + resolved_query=resolved_query, + ), + } + for group in shown[offset : offset + limit] + ], + **metadata, + "actions": actions, + "categories": categories, + "strong_total": strong, + "other_stage_total": other_stage_total, + "eviction_facets": selected_topic == "eviction", + "location_counties": counties, + "total": len(groups), + "corrected_terms": corrected, + } + group = next((group for group in groups if group["key"] == group_key), None) + if group is None: + raise ValueError("These results have changed. Search again.") + contexts = group["_labels"] if selected_topic or case_filters else None + if context: + contexts = [ + label for label in group["_labels"] if case_context_key(label["category"], label["case_type"]) == context + ] + if not contexts: + raise ValueError("This case type is no longer in these results. Search again.") + if court: + return search_paths( + index, + query, + initial=initial, + offset=offset, + limit=limit, + names=group["_names"], + court=court, + contexts=contexts, + courts=courts, + ) + paths, _ = matching_paths(index, query, initial) + if courts: + paths = paths.filter(court__code__in=courts) + if contexts is not None: + paths = paths.filter(context_filter(contexts)) + courts = ( + paths.filter(filing_type__name__in=group["_names"]) + .order_by("court__name") + .values("court__code", "court__name") + .distinct() + ) + return { + "courts": [{"code": row["court__code"], "name": row["court__name"]} for row in courts], + "corrected_terms": corrected, + } + + +def validate_path(path): + """Re-fetch each parent-child relationship; names come from today's API.""" + catalog = CodeCatalog(path.index.jurisdiction, timeout=(5, 8)) + + def find(options, facet): + chosen = next((o for o in options if o["code"] == getattr(path, facet)["code"]), None) + if chosen is None: + raise ValueError("This filing path is no longer offered. Search again or use the court lists.") + return chosen + + try: + options = {"court": catalog.courts()} + court = find(options["court"], "court") + options["case_category"] = catalog.categories(court["code"], path.initial) + category = find(options["case_category"], "case_category") + options["case_type"] = catalog.types(court["code"], category["code"], path.initial) + case_type = find(options["case_type"], "case_type") + options["filing_type"] = catalog.filings(court["code"], category["code"], case_type["code"], path.initial) + filing = find(options["filing_type"], "filing_type") + finally: + catalog.close() + for facet, option in zip(FACETS, (court, category, case_type, filing), strict=True): + setattr(path, facet, option) + path.explanation, path.explanation_source = explanation_for( + path.index.jurisdiction, + {facet: getattr(path, facet) for facet in FACETS}, + path.initial, + ) + result = serialize_path(path) + result["options"] = options + if result["unavailable"]: + raise ValueError(result["unavailable"]) + return result diff --git a/efile_app/efile/services/filing_code_sync.py b/efile_app/efile/services/filing_code_sync.py new file mode 100644 index 00000000..382f6400 --- /dev/null +++ b/efile_app/efile/services/filing_code_sync.py @@ -0,0 +1,286 @@ +"""Incremental public-catalog imports across the proxy's HTTP service boundary. + +One small manifest identifies changed courts. A compact export contains their +code tables and relationships; expansion and search rules belong to LITEFile. +Downloads and validation finish before a transaction replaces changed courts. +""" + +import gzip +import hashlib +import json +from contextlib import closing +from itertools import islice +from pathlib import Path +from tempfile import TemporaryDirectory +from urllib.parse import quote + +from django.db import connection, transaction +from django.utils import timezone +from psycopg import Cursor +from psycopg.types.json import Jsonb + +from efile.db_expressions import CourtCode +from efile.models import FilingCodeIndex, FilingCodePath +from efile.services.court_selection import is_non_filing_court +from efile.services.filing_code_search import ( + FACETS, + CodeCatalog, + explanation_for, + packed, + rules, + search_tokens, + source_url, + stems, + words, +) + + +class BulkCodeCatalog(CodeCatalog): + def __init__(self, jurisdiction): + super().__init__(jurisdiction) + self.jurisdiction = jurisdiction + self.base = f"{source_url()}/jurisdictions/{jurisdiction}/codes/filing_catalog" + + def read(self, suffix=""): + with self.session.get(self.base + suffix, timeout=self.timeout) as response: + if response.status_code == 404 and not suffix: + raise ValueError("Deploy the proxy filing_catalog API first, or explicitly use --legacy-crawl.") + response.raise_for_status() + data = response.json() + if not isinstance(data, dict) or data.get("version") != 1 or data.get("jurisdiction") != self.jurisdiction: + raise ValueError("Unsupported or wrong-jurisdiction filing catalog export") + return data + + def manifest(self): + data = self.read() + if not isinstance(data.get("courts"), list) or not data["courts"]: + raise ValueError("Empty or invalid filing catalog manifest; keeping the previous index") + courts = {} + for court in data["courts"]: + _option(court) + revision = court.get("revision") + if not isinstance(revision, str) or not revision: + raise ValueError("Invalid court catalog revision") + if court["code"] in courts: + raise ValueError("Duplicate court in filing catalog manifest") + if not is_non_filing_court(court["name"]): + courts[court["code"]] = court + if not courts: + raise ValueError("No filing courts in catalog manifest; keeping the previous index") + return courts + + def court(self, court): + data = self.read(f"/courts/{quote(court['code'], safe='')}") + if data.get("court") != court: + raise ValueError("Court export does not match its manifest revision") + validate_export(data) + return data + + +def _option(value): + if not isinstance(value, dict) or any( + not isinstance(value.get(key), str) or not value[key] for key in ("code", "name") + ): + raise ValueError("Invalid option in filing catalog export") + + +def validate_export(data): + for table in ("categories", "case_types", "filing_types"): + if not isinstance(data.get(table), list): + raise ValueError("Missing code table in court export") + for row in data[table]: + _option(row) + if table != "categories" and ( + "case_category" not in row + or row["case_category"] is not None + and not isinstance(row["case_category"], str) + ): + raise ValueError("Invalid category relationship in court export") + if table == "case_types" and type(row.get("initial")) is not bool: + raise ValueError("Invalid case-type timing in court export") + if table == "filing_types" and ( + "case_type" not in row + or row["case_type"] is not None + and not isinstance(row["case_type"], str) + or row.get("timing") not in ("Initial", "Subsequent", "Both") + ): + raise ValueError("Invalid filing-type relationship or timing in court export") + + +def export_entries(data, jurisdiction): + """Match the existing proxy filters, including its specific-before-generic fallback.""" + court = {key: data["court"][key] for key in ("code", "name")} + categories = {row["code"]: row for row in data["categories"]} + case_types = {(row["case_category"], row["code"]): row for row in data["case_types"]} + for initial in (True, False): + # The proxy's category endpoint gates each timing on at least one case + # type with that initial flag. Subsequent type lists then include all + # case types in those categories, matching its existing SQL exactly. + offered_categories = {row["case_category"] for row in case_types.values() if row["initial"] == initial} + by_category, by_type, generic = {}, {}, [] + for filing in data["filing_types"]: + if filing["timing"] not in (("Initial", "Both") if initial else ("Subsequent", "Both")): + continue + if filing["case_type"] == "": + if filing["case_category"] == "": + generic.append(filing) + elif filing["case_category"] is not None: + by_category.setdefault(filing["case_category"], []).append(filing) + elif filing["case_type"] is not None: + by_type.setdefault(filing["case_type"], []).append(filing) + for case_type in case_types.values(): + category = categories.get(case_type["case_category"]) + if category is None or category["code"] not in offered_categories or initial and not case_type["initial"]: + continue + # Filtered list APIs deduplicate options by code after choosing specific or generic lists. + selected = by_category.get(category["code"], []) + by_type.get(case_type["code"], []) + filings = {row["code"]: row for row in selected or generic} + for filing in filings.values(): + path = dict(zip(FACETS, (court, category, case_type, filing), strict=True)) + path = {facet: {key: value[key] for key in ("code", "name")} for facet, value in path.items()} + terms = {facet: search_tokens(path[facet]["name"], jurisdiction) for facet in FACETS} + all_terms = set.union(*terms.values()) + all_terms.update(token for facet in FACETS for token in stems(path[facet]["code"])) + explanation, source = explanation_for(jurisdiction, path, initial) + yield dict( + **path, + initial=initial, + search_text=packed(all_terms), + filing_terms=packed(terms["filing_type"]), + case_terms=packed(terms["case_category"] | terms["case_type"]), + explanation=explanation, + explanation_source=source, + ) + + +def _stage(data, jurisdiction, path): + vocabulary, tokens, count = set(), set(), 0 + with gzip.open(path, "wt", encoding="utf-8", compresslevel=1) as stream: + for entry in export_entries(data, jurisdiction): + stream.write(json.dumps(entry) + "\n") + tokens.update(entry["search_text"].split()) + vocabulary.update(word for facet in FACETS for word in words(entry[facet]["name"])) + count += 1 + return {"count": count, "words": sorted(vocabulary), "tokens": sorted(tokens)} + + +def _insert_paths(index, snapshot, count, progress): + """COPY on Postgres; bounded ORM batches on SQLite. IDs come from each database's sequence.""" + saved = 0 + if connection.vendor == "postgresql": + fields = [field for field in FilingCodePath._meta.concrete_fields if not field.primary_key] + columns = ", ".join(connection.ops.quote_name(field.column) for field in fields) + table = connection.ops.quote_name(FilingCodePath._meta.db_table) + # Django defaults to psycopg's ClientCursor, which does not implement COPY. + with Cursor(connection.connection) as cursor, cursor.copy(f"COPY {table} ({columns}) FROM STDIN") as copy: + for line in snapshot: + entry = json.loads(line) + entry["index"] = index.pk + copy.write_row( + [ + Jsonb(entry[field.name]) if field.get_internal_type() == "JSONField" else entry[field.name] + for field in fields + ] + ) + saved += 1 + if progress and saved % 100000 == 0: + progress(f"Saved {saved} of {count} changed filing paths") + else: + while batch := [FilingCodePath(index=index, **json.loads(line)) for line in islice(snapshot, 500)]: + FilingCodePath.objects.bulk_create(batch, batch_size=500) + saved += len(batch) + if progress and saved % 100000 == 0: + progress(f"Saved {saved} of {count} changed filing paths") + + +def synchronize_index(jurisdiction, *, progress=None, force=(), dry_run=False, catalog=None, full=False): + """Re-index courts whose revision changed, plus any court codes in ``force``. + + ``catalog`` supplies courts (the proxy's HTTP export by default, or the local + copy of the EFSP database); ``full`` re-indexes every court. ``dry_run`` + reads and checks everything, reports each changed court, and saves nothing. + """ + started = timezone.now() + baseline = FilingCodeIndex.objects.filter(jurisdiction=jurisdiction).first() + previous = ( + baseline if baseline and baseline.source_url == source_url() and baseline.rules_digest == rules()[1] else None + ) + snapshots = previous.court_snapshots if previous and not full else {} + # Stage every changed court before publishing anything. A failure preserves the whole old index. + catalog = catalog or BulkCodeCatalog(jurisdiction) + with closing(catalog), TemporaryDirectory(prefix="filing-sync-") as directory: + courts = catalog.manifest() + if unknown := set(force) - set(courts): + raise ValueError("Not filing courts in the proxy's catalog manifest: " + ", ".join(sorted(unknown))) + next_snapshots, staged = {}, {} + downloaded = False + for code, court in courts.items(): + old = snapshots.get(code) + if old and old["revision"] == court["revision"] and code not in force: + next_snapshots[code] = old + continue + if progress: + progress(f"Downloading changed court: {court['name']}") + data = catalog.court(court) + downloaded = True + path = Path(directory) / f"{hashlib.sha256(code.encode()).hexdigest()}.json.gz" + metadata = _stage(data, jurisdiction, path) + # A court that loses every path almost always means the proxy served a half-loaded + # court. Keep the old index rather than publishing an empty court. + if old and old["count"] and not metadata["count"] and code not in force: + raise ValueError( + f"{court['name']} ({code}) now has no filing paths (had {old['count']}); keeping the " + f"previous index. If that's expected, re-run with --court {code}." + ) + if progress: + progress(f"{court['name']} ({code}): {old['count'] if old else 'new'} -> {metadata['count']} paths") + next_snapshots[code] = dict(metadata, revision=court["revision"]) + staged[code] = path + if (downloaded or set(snapshots) - set(courts)) and catalog.manifest() != courts: + raise ValueError("Proxy catalog changed during synchronization; keeping the previous index and retrying") + if not sum(item["count"] for item in next_snapshots.values()): + raise ValueError("No filing paths returned; keeping the previous index") + vocabulary = set().union(*(set(item["words"]) for item in next_snapshots.values())) + for concept in rules()[0]["concepts"].values(): + if jurisdiction in concept.get("jurisdictions", rules()[0]["jurisdictions"]): + vocabulary.update(word for term in concept["terms"] for word in words(term)) + tokens = set().union(*(set(item["tokens"]) for item in next_snapshots.values())) + count = sum(next_snapshots[code]["count"] for code in staged) + if progress: + progress( + f"{len(staged)} changed courts; {len(courts) - len(staged)} unchanged; saving {count} filing paths" + ) + if dry_run: + return count + with transaction.atomic(): + index, created = FilingCodeIndex.objects.get_or_create( + jurisdiction=jurisdiction, + defaults={"source_url": source_url(), "refreshed_at": started, "rules_digest": rules()[1]}, + ) + index = FilingCodeIndex.objects.select_for_update().get(pk=index.pk) + if ( + index.refreshed_at > started + or not created + and (baseline is None or index.refreshed_at != baseline.refreshed_at) + ): + return 0 + if previous is None or not snapshots: + # One-time bootstrap from legacy indexes, or a different source/rules revision. + index.paths.all().delete() + else: + for code in set(snapshots) - set(courts) | set(staged): + index.paths.alias(catalog_court_code=CourtCode("court")).filter(catalog_court_code=code).delete() + for code, path in staged.items(): + with gzip.open(path, "rt", encoding="utf-8") as snapshot: + _insert_paths(index, snapshot, next_snapshots[code]["count"], progress) + index.source_url = source_url() + index.rules_digest = rules()[1] + index.refreshed_at = started + index.court_snapshots = next_snapshots + index.vocabulary = { + "words": sorted(vocabulary), + "tokens": sorted(tokens), + "courts": {code: court["name"] for code, court in courts.items()}, + } + index.save() + return count diff --git a/efile_app/efile/services/filing_path.py b/efile_app/efile/services/filing_path.py index 0a155e67..9c88479e 100644 --- a/efile_app/efile/services/filing_path.py +++ b/efile_app/efile/services/filing_path.py @@ -22,7 +22,7 @@ from django.db import transaction -from efile.models import FilingDocument, FilingDraft, sync_primary_filing_type +from efile.models import FilingDocument, FilingDraft, FilingParty, sync_primary_filing_type from efile.services.fee_quotes import invalidate_fee_quote from efile.workflow import ExistingCase, normalize_existing_case @@ -139,6 +139,51 @@ def change_filing_path(draft: FilingDraft, new_path: str) -> FilingPathChange: } +@transaction.atomic +def clear_changed_classification(draft, court_code, case_category_code, case_type_code): + """Codes and fees below a changed parent cannot carry over to another path.""" + previous = (draft.court_code, draft.case_category_code, draft.case_type_code) + if not any(previous) or previous == (court_code, case_category_code, case_type_code): + return False + FilingDocument.objects.filter(draft=draft).update( + filing_type_code="", + filing_type_name="", + document_type_code="", + document_type_name="", + document_type_confirmed=False, + filing_component_code="", + filing_component_name="", + requested_optional_services=[], + filing_requires_amount_in_controversy=False, + ) + FilingParty.objects.filter(draft=draft).update(party_type="", party_type_name="") + draft.case_subtype_code = "" + draft.case_subtype_name = "" + draft.document_type_code = "" + draft.document_type_name = "" + draft.previous_case_id = "" + draft.case_title = "" + draft.optional_services = [] + invalidate_fee_quote(draft, save=False) + draft.save( + update_fields=[ + "case_subtype_code", + "case_subtype_name", + "document_type_code", + "document_type_name", + "previous_case_id", + "case_title", + "optional_services", + "quoted_fee_total", + "quoted_fee_breakdown", + "quoted_fee_fingerprint", + "updated_at", + ] + ) + sync_primary_filing_type(draft) + return True + + def describe_path_change(change: FilingPathChange) -> str: """What changing the path kept and what it undid, in the filer's terms.""" diff --git a/efile_app/efile/services/glossary.py b/efile_app/efile/services/glossary.py new file mode 100644 index 00000000..c336b607 --- /dev/null +++ b/efile_app/efile/services/glossary.py @@ -0,0 +1,53 @@ +"""Plain-language meanings for court terms in filing and case-type names. + +Terms live in ``data/legal_glossary.yaml``: general ones, plus a state's own +under ``jurisdictions``, which replace the general entry with the same key. +""" + +from __future__ import annotations + +import re +from functools import lru_cache +from pathlib import Path + +import yaml + +GLOSSARY_PATH = Path(__file__).resolve().parents[1] / "data" / "legal_glossary.yaml" + + +@lru_cache(maxsize=1) +def _config(): + config = yaml.safe_load(GLOSSARY_PATH.read_bytes()) + if config.get("version") != 1: + raise ValueError("Unsupported glossary version") + return config + + +@lru_cache(maxsize=16) +def _terms(jurisdiction): + config = _config() + terms = {**config.get("terms", {}), **config.get("jurisdictions", {}).get(jurisdiction, {}).get("terms", {})} + compiled = [] + for key, term in terms.items(): + words = "|".join(re.escape(word).replace(r"\ ", r"[\s-]+") for word in term["match"]) + compiled.append((key, re.compile(rf"(?/", self.admin_view(self.privacy_request), name="privacy_request"), path("analytics/", self.admin_view(self.analytics), name="analytics"), path("authenticator//", self.admin_view(self.authenticator), name="authenticator"), + path("filing-codes/", self.admin_view(self.filing_codes), name="filing_codes"), ] return extra + super().get_urls() @@ -355,6 +357,43 @@ def analytics(self, request): export_enabled=settings.LITEFILE_ANALYTICS_EXPORT_ENABLED, ) + def filing_codes(self, request): + """Resync codes from the EFSP, or rebuild search indexes; the code index worker runs them.""" + if not request.user.is_superuser: + raise PermissionDenied + from efile.services.filing_code_copy import status + from efile.services.filing_code_search import rules + + jurisdictions = rules()[0]["jurisdictions"] + if request.method == "POST": + kind = request.POST.get("kind", "") + jurisdiction = request.POST.get("jurisdiction", "") + if kind not in FilingCodeJob.Kind.values or jurisdiction not in ["", *jurisdictions]: + messages.error(request, "Choose a valid action and jurisdiction.") + elif FilingCodeJob.objects.filter( + kind=kind, jurisdiction=jurisdiction, status=FilingCodeJob.Status.QUEUED + ).exists(): + messages.info(request, "That is already queued.") + else: + FilingCodeJob.objects.create(kind=kind, jurisdiction=jurisdiction, requested_by=request.user) + audit(request.user, jurisdiction, f"filing_codes_{kind}", "queued") + messages.success( + request, + f"{FilingCodeJob.Kind(kind).label} queued for {jurisdiction.capitalize() or 'every jurisdiction'}. " + "The code index worker starts it within a minute.", + ) + return redirect("litefile_staff:filing_codes") + return self.page( + request, + "filing_codes", + title="Filing codes", + jurisdictions=jurisdictions, + statuses=[status(jurisdiction) for jurisdiction in jurisdictions], + jobs=FilingCodeJob.objects.select_related("requested_by")[:15], + schedule=f"{settings.FILING_CODE_SYNC_TIME} {settings.FILING_CODE_SYNC_TIMEZONE}", + source=settings.FILING_CODE_SYNC_MODE, + ) + def authenticator(self, request, account_id): if not request.user.is_superuser: raise PermissionDenied diff --git a/efile_app/efile/static/css/components/filing-code-search.css b/efile_app/efile/static/css/components/filing-code-search.css new file mode 100644 index 00000000..8ad5b480 --- /dev/null +++ b/efile_app/efile/static/css/components/filing-code-search.css @@ -0,0 +1,578 @@ +.code-search-invitation { + margin-block: 1.5rem; + padding: 1.25rem; + background: #edf5ff; + border-radius: 0.75rem; +} + +.code-search { + width: min(48rem, calc(100% - 2rem)); + height: min(60rem, calc(100dvh - 2rem)); + max-height: calc(100dvh - 2rem); + padding: 0; + border: 1px solid #cbd5e1; + border-radius: 1rem; + color: #48576c; + overflow: hidden; +} + +.code-search::backdrop { + background: rgb(20 40 70 / 35%); +} + +.code-search[open] { + display: flex; + flex-direction: column; +} + +.code-search h2, +.code-search h3, +.code-search legend, +.code-search label, +.code-search dd { + color: #1e3a5f; +} + +.code-search__heading { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 1rem; + padding: 1.5rem 1.5rem 0; +} + +.code-search__heading h2 { + margin: 0; + font-size: 1.5rem; + font-weight: 600; +} + +.code-search__heading h2:focus { + outline: none; +} + +.code-search__heading h2:focus-visible { + outline: 3px solid #b9cdeb; + outline-offset: 2px; +} + +.code-search__step { + margin: 0 0 0.25rem; + font-size: 0.875rem; + color: #616c82; +} + +#code-search-help { + margin: 0.375rem 1.5rem 0; +} + +.code-search__body { + flex: 1 1 auto; + min-height: 0; + padding: 1.25rem 1.5rem 1rem; + overflow-y: auto; + overscroll-behavior: contain; +} + +.code-search__panel { + display: flex; + flex-direction: column; + gap: 1rem; +} + +.code-search__panel[hidden] { + display: none; +} + +.code-search__actions { + display: flex; + align-items: center; + justify-content: space-between; + gap: 0.75rem; + padding: 1rem 1.5rem 1.25rem; + border-block-start: 1px solid #e3e8ef; +} + +.code-search__actions .btn { + min-height: 2.75rem; +} + +.code-search__actions .btn-primary { + margin-inline-start: auto; +} + +.code-search__link { + padding: 0; + border: 0; + background: none; + font: inherit; + color: #2c5aa0; + text-decoration: underline; + text-underline-offset: 3px; + cursor: pointer; +} + +@media (hover: hover) { + .code-search__link:hover { + color: #1e3a5f; + } +} + +.code-search__meta { + margin: 0; + font-size: 0.875rem; + color: #616c82; +} + +/* Step 1: what you're filing */ + +.code-search__lookup { + display: grid; + grid-template-columns: minmax(0, 1fr) 10rem; + gap: 0.75rem; + align-items: end; +} + +.code-search__lookup:has(#code-search-zip-filter[hidden]) { + grid-template-columns: minmax(0, 1fr); +} + +.code-search__lookup label { + display: block; + margin-block-end: 0.375rem; + font-size: 0.875rem; + font-weight: 600; +} + +.code-search__lookup .form-control { + min-height: 3.25rem; + font-size: 1.0625rem; + border-radius: 0.625rem; +} + +#code-search-zip { + letter-spacing: 0.04em; +} + +.code-search__context-line { + display: flex; + flex-wrap: wrap; + align-items: flex-start; + justify-content: space-between; + gap: 0.5rem 1rem; + margin-block-start: -0.25rem; + font-size: 0.875rem; +} + +.code-search__context-line p { + margin: 0; +} + +.code-search__zip-help { + display: flex; + flex-direction: column; + gap: 0.25rem; +} + +.code-search__zip-help p:empty, +.code-search__zip-shortcuts[hidden] { + display: none; +} + +.code-search__stage { + margin-inline-start: auto; +} + +/* Short questions: one row each, label beside its chips. */ +.code-search__narrow { + display: flex; + flex-direction: column; + gap: 0.5rem; + padding: 0.625rem 0.875rem; + border: 1px solid #dce3ec; + border-radius: 0.625rem; + background: #f9fafc; +} + +.code-search__narrow[hidden] { + display: none; +} + +.code-search__question { + display: grid; + grid-template-columns: 11rem minmax(0, 1fr); + gap: 0.25rem 0.75rem; + align-items: baseline; +} + +.code-search__question-label { + font-size: 0.8125rem; + font-weight: 600; + color: #1e3a5f; +} + +.code-search__chips { + display: flex; + flex-wrap: wrap; + gap: 0.375rem; + min-width: 0; +} + +.code-search__chip { + display: inline-flex; + align-items: center; + gap: 0.375rem; + min-height: 2rem; + padding: 0 0.75rem; + border: 1px solid #c9d3e0; + border-radius: 999px; + background: #fff; + font: inherit; + font-size: 0.8125rem; + color: #1e3a5f; + cursor: pointer; +} + +@media (hover: hover) { + .code-search__chip:hover { + border-color: #9fb4d6; + } +} + +.code-search__chip[aria-pressed="true"] { + border-color: #2c5aa0; + background: #2c5aa0; + color: #fff; +} + +.code-search__chip:focus-visible { + outline: 3px solid #b9cdeb; + outline-offset: 2px; +} + +.code-search__chip-count { + font-size: 0.75rem; + color: #48576c; +} + +.code-search__chip[aria-pressed="true"] .code-search__chip-count { + color: #fff; +} + +.code-search__amount-help { + margin-inline-start: 11.75rem; + font-size: 0.8125rem; +} + +#code-search-status { + margin: 0; + font-size: 0.875rem; + color: #616c82; +} + +#code-search-suggestions h3 { + font-size: 1.125rem; +} + +.code-search__suggestions { + display: flex; + flex-wrap: wrap; + gap: 0.75rem; +} + +.code-search__results, +#code-search-paths { + display: flex; + flex-direction: column; + gap: 0.625rem; + min-width: 0; +} + +.code-search__results:empty { + display: none; +} + +.code-search__result { + cursor: pointer; + padding: 0.875rem 1rem; + border: 1px solid #dce3ec; + border-radius: 0.625rem; + background: #fff; + overflow-wrap: anywhere; +} + +@media (hover: hover) { + .code-search__result:hover { + border-color: #9fb4d6; + } +} + +.code-search__result:has(input:checked) { + border-color: #2c5aa0; + background: #f2f6fc; + box-shadow: inset 0 0 0 1px #2c5aa0; +} + +.code-search__choice { + display: flex; + align-items: baseline; + gap: 0.75rem; + margin: 0; + font-size: 1rem; + font-weight: 600; + cursor: pointer; +} + +.code-search__choice input { + flex-shrink: 0; + accent-color: #2c5aa0; +} + +.code-search__result>p, +.code-search__result>details { + margin: 0.25rem 0 0 1.75rem; +} + +.code-search__result>.code-search__meta { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.code-search__result details { + font-size: 0.875rem; +} + +.code-search__result details p, +.code-search__result details ul { + margin-block: 0.375rem; +} + +.code-search__variant, +.code-search__case-description { + font-size: 0.875rem; +} + +/* Step 2: court and case type */ + +.code-search__summary { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 1rem; + padding: 1rem 1.125rem; + border-radius: 0.625rem; + background: #f6f8fb; + font-size: 0.9375rem; +} + +.code-search__summary dl { + display: grid; + grid-template-columns: 7.5rem minmax(0, 1fr); + gap: 0.375rem 0.75rem; + margin: 0; +} + +.code-search__summary dt { + font-weight: 400; + color: #616c82; +} + +.code-search__summary dd { + margin: 0; +} + +.code-search__summary .code-search__link { + font-size: 0.875rem; + white-space: nowrap; +} + +.code-search__courts, +.code-search__case-types { + display: flex; + flex-direction: column; + gap: 0.625rem; + min-width: 0; + margin: 0; + padding: 0; + border: 0; +} + +.code-search__courts legend, +.code-search__courts>label, +.code-search__case-types legend { + margin: 0; + font-size: 0.9375rem; + font-weight: 600; +} + +.code-search__case-types[hidden] { + display: none; +} + +.code-search__court-cards { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 0.625rem; +} + +.code-search__card { + display: flex; + align-items: baseline; + gap: 0.75rem; + padding: 0.875rem 1rem; + border: 1px solid #dce3ec; + border-radius: 0.625rem; + font-weight: 600; + cursor: pointer; +} + +@media (hover: hover) { + .code-search__card:hover { + border-color: #9fb4d6; + } +} + +.code-search__card:has(input:checked) { + border-color: #2c5aa0; + background: #f2f6fc; + box-shadow: inset 0 0 0 1px #2c5aa0; +} + +.code-search__card input { + flex-shrink: 0; + accent-color: #2c5aa0; +} + +#code-search-path-status { + margin: 0; + font-size: 0.875rem; + color: #616c82; +} + +/* Step 3: check and use */ + +.code-search__confirm { + margin: 0; + border: 1px solid #dce3ec; + border-radius: 0.75rem; +} + +.code-search__confirm-row { + display: grid; + grid-template-columns: 9rem minmax(0, 1fr); + gap: 1rem; + align-items: baseline; + padding: 1rem 1.25rem; +} + +.code-search__confirm-row+.code-search__confirm-row { + border-block-start: 1px solid #e3e8ef; +} + +.code-search__confirm-row dt { + font-size: 0.875rem; + font-weight: 400; + color: #616c82; +} + +.code-search__confirm-row dd { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: 1rem; + margin: 0; +} + +.code-search__confirm-value { + display: flex; + flex-direction: column; + gap: 0.25rem; +} + +.code-search__confirm-row .code-search__link { + font-size: 0.875rem; +} + +.code-search__explanation { + display: flex; + flex-direction: column; + gap: 0.375rem; + padding: 1rem 1.125rem; + border-radius: 0.625rem; + background: #f6f8fb; + font-size: 0.9375rem; +} + +.code-search__explanation p { + margin: 0; +} + +.code-search__explanation[hidden] { + display: none; +} + +#code-search-apply-status { + margin: 0; +} + +@media (width <=600px) { + + .code-search__heading, + .code-search__body, + .code-search__actions { + padding-inline: 1rem; + } + + #code-search-help { + margin-inline: 1rem; + } + + .code-search__lookup, + .code-search__question { + grid-template-columns: minmax(0, 1fr); + } + + .code-search__amount-help { + margin-inline-start: 0; + } + + .code-search__stage { + margin-inline-start: 0; + } + + .code-search__court-cards { + grid-template-columns: minmax(0, 1fr); + } + + .code-search__summary dl, + .code-search__confirm-row { + grid-template-columns: minmax(0, 1fr); + gap: 0.25rem; + } +} + +.code-search__name-prefix.is-shared { + font-weight: 400; + color: #616c82; +} + +.code-search__chip--more { + border-style: dashed; +} + +.code-search__glossary { + font-size: 0.875rem; +} + +.code-search__glossary dl { + margin: 0.375rem 0 0; +} + +.code-search__glossary dt { + font-weight: 600; +} + +.code-search__glossary dd { + margin: 0 0 0.375rem; +} \ No newline at end of file diff --git a/efile_app/efile/static/js/court-selector.js b/efile_app/efile/static/js/court-selector.js index ea725e3e..28604109 100644 --- a/efile_app/efile/static/js/court-selector.js +++ b/efile_app/efile/static/js/court-selector.js @@ -251,6 +251,7 @@ // The newest request whose answer has been drawn (or failed). Anything // below `latest` is still on its way, whatever order older ones return in. let settledRequest = 0; + let latestStart = 0; // orders start() calls against each other let lastSteps = []; let expanded = ""; // the answered question the filer reopened let lastRender = { @@ -558,7 +559,14 @@ return { /** Draw the first questions, picking up a court already chosen. */ - async start(courtCode, guessedCourt) { + async start(courtCode, guessedCourt, { + requireMatch = false + } = {}) { + // Nothing in flight is dropped until this lands: a court that + // cannot be matched (or a failed fetch) leaves the questions, + // and any answer still loading for them, exactly as they were. + const token = ++latestStart; + const before = latest; const query = new URLSearchParams({ jurisdiction }); @@ -570,7 +578,16 @@ }, }); const result = await response.json(); + // A newer start, or an answer the filer gave meanwhile, wins. + if (token !== latestStart || before !== latest) return false; if (!response.ok || !result.success || !result.data.available) return false; + if (requireMatch && result.data.selected?.value !== courtCode) return false; + latest += 1; + settledRequest = latest; + container.removeAttribute("aria-busy"); + editing = null; + expanded = ""; + Object.keys(answers).forEach(key => delete answers[key]); (result.data.steps || []).forEach((step) => { if (step.answer) answers[step.id] = step.answer; }); diff --git a/efile_app/efile/static/js/extraction-review.js b/efile_app/efile/static/js/extraction-review.js index 27813f54..bb53a9f7 100644 --- a/efile_app/efile/static/js/extraction-review.js +++ b/efile_app/efile/static/js/extraction-review.js @@ -251,6 +251,7 @@ params: () => ({ court: fields.court.select.value, guessed_case_category: guesses["case category"] || "", + existing_case: existingCaseWire(), }), }, case_type: { @@ -261,6 +262,7 @@ court: fields.court.select.value, parent: fields.case_category.select.value, guessed_case_type: guesses["case type"] || "", + existing_case: existingCaseWire(), }), }, filing_type: { @@ -522,8 +524,52 @@ filing_type: loadFilerRoles, }; + let applyingSearch = false; + + async function applySearchPath(path) { + applyingSearch = true; + try { + if (courtSelector && !await courtSelector.start(path.court.code, "", { + requireMatch: true + })) { + throw new Error(gettext("This court could not be selected. Try again or use the court questions.")); + } + const stage = form.querySelector(`input[name="existing_case"][value="${path.initial ? "new" : "existing"}"]`); + if (stage && !stage.checked) { + stage.checked = true; + stage.dispatchEvent(new Event("change", { + bubbles: true + })); + // Make the changed answer visible beside the saved summary. + if (pathQuestion) pathQuestion.hidden = false; + changePath?.setAttribute("aria-expanded", "true"); + } + // All four lists were validated together on the server. Publish + // them together so old cascading requests cannot undo this choice. + supersede("court"); + ORDER.forEach(key => { + const field = fields[key]; + field.select.replaceChildren(new Option(gettext("Choose an option"), "")); + path.options[key].forEach(item => field.select.add(new Option(item.name, item.code))); + field.select.value = path[key].code; + field.select.disabled = false; + field.loaded = true; + finishLoading(key); + remember(key); + field.hint.textContent = field.defaultHint; + setMode(key, field.guided ? "edit" : "found"); + }); + await loadFilerRoles(); + availability.check(); + announce(gettext("Filing path updated. Check the choices, then confirm and continue to save them.")); + } finally { + applyingSearch = false; + } + } + ORDER.forEach((key) => { fields[key].select.addEventListener("change", () => { + if (applyingSearch) return; remember(key); availability.check(fields[key].root); fields[key].hint.textContent = fields[key].defaultHint; @@ -533,7 +579,8 @@ form.querySelectorAll('input[name="existing_case"]').forEach((radio) => { radio.addEventListener("change", () => { - if (fields.case_type.select.value) loadList("filing_type"); + if (applyingSearch) return; + if (fields.court.select.value) loadList("case_category"); }); }); @@ -725,5 +772,13 @@ // The guard blocks until a check runs. A guided selector with no court // chosen yet fires no change, so check once the courts are in place. - loadCourts().finally(() => availability.check()); + loadCourts().finally(() => { + availability.check(); + window.filingCodeSearch?.mount({ + jurisdiction: context.jurisdiction, + existingCase: existingCaseWire, + applyPath: applySearchPath, + zipShortcuts: context.zip_shortcuts || [], + }); + }); })(); \ No newline at end of file diff --git a/efile_app/efile/static/js/filing-code-search.js b/efile_app/efile/static/js/filing-code-search.js new file mode 100644 index 00000000..dfca27ac --- /dev/null +++ b/efile_app/efile/static/js/filing-code-search.js @@ -0,0 +1,1127 @@ +(function() { + const STEPS = { + filing: { + number: 1, + title: () => gettext("What are you filing?") + }, + path: { + number: 2, + title: () => gettext("Choose your court and case type") + }, + confirm: { + number: 3, + title: () => gettext("Check your choices") + }, + }; + // Up to this many courts show as cards; more fall back to a dropdown. + const COURT_CARD_LIMIT = 6; + // "What do you want to do?" is asked only once results are this many. + const ACTION_QUESTION_MIN = 8; + // A question with more choices than this shows the most used ones first. + const CHIPS_SHOWN = 5; + // Choices a court may code as separate case types, asked in this order and + // only when the court's paths differ on them. See path_qualifiers(). + const PATH_QUESTIONS = [{ + key: "amount", + label: () => gettext("How much are you asking for?"), + options: {} + }, { + key: "representation", + label: () => gettext("Do you have a lawyer?"), + options: { + self: () => gettext("No, I'm representing myself"), + lawyer: () => gettext("Yes") + } + }, { + key: "jury", + label: () => gettext("Do you want a jury?"), + options: { + none: () => gettext("No jury"), + jury: () => gettext("Jury"), + 6: () => gettext("Jury of 6"), + 12: () => gettext("Jury of 12") + } + }, { + key: "government", + label: () => gettext("Filing for a government body?"), + options: { + no: () => gettext("No"), + yes: () => gettext("Yes") + }, + // Almost no one filing here is; the answer stays visible to change. + assume: "no" + }]; + const ZIP_PATTERN = /^\d{5}(?:-\d{4})?$/; + + function mount({ + jurisdiction, + existingCase, + applyPath, + zipShortcuts = [] + }) { + const dialog = document.getElementById("code-search-dialog"); + const open = document.getElementById("open-code-search"); + if (!dialog || !open) return; + const byId = id => document.getElementById(id); + const title = byId("code-search-title"); + const stepLabel = byId("code-search-step"); + const panels = { + filing: byId("code-search-step-filing"), + path: byId("code-search-step-path"), + confirm: byId("code-search-step-confirm"), + }; + const query = byId("code-search-query"); + const status = byId("code-search-status"); + const results = byId("code-search-results"); + const more = byId("code-search-more"); + const otherStage = byId("code-search-other-stage"); + const suggestions = byId("code-search-suggestions"); + const postalCode = byId("code-search-zip"); + const zipStatus = byId("code-search-zip-status"); + const zipShortcutList = byId("code-search-zip-shortcuts"); + const stageLabel = byId("code-search-stage-label"); + const stageToggle = byId("code-search-stage-toggle"); + const moreFilters = byId("code-search-more-filters"); + const filterPanel = byId("code-search-filters"); + const caseFilters = byId("code-search-case-filters"); + const summary = byId("code-search-summary"); + const courtsBox = byId("code-search-courts"); + const pathStatus = byId("code-search-path-status"); + const pathQuestions = byId("code-search-path-questions"); + const caseTypes = byId("code-search-case-types"); + const paths = byId("code-search-paths"); + const morePaths = byId("code-search-more-paths"); + const confirmList = byId("code-search-confirm"); + const explanation = byId("code-search-explanation"); + const applyStatus = byId("code-search-apply-status"); + const cancel = byId("cancel-code-search"); + const back = byId("code-search-back"); + const next = byId("code-search-next"); + const apply = byId("apply-code-search"); + + let step = "filing"; + let stage = "no"; + let documentKind = ""; + let action = ""; + let category = ""; + const expanded = new Set(); + let showAll = false; + let strongTotal = 0; + let activeZip = ""; + let caseTopic = ""; + let caseChoices = {}; + let caseFilterSchema = ""; + let lastFacets = null; + let lastCounties = []; + let generation = 0; + let controller; + let timer; + let offset = 0; + let chosen = null; + let selected = null; + let pathGeneration = 0; + let pathController; + let loadedFor = null; + let court = ""; + let pathOffset = 0; + let courtPaths = []; + let pathAnswers = {}; + let applying = false; + + function element(tag, text, className) { + const node = document.createElement(tag); + if (text) node.textContent = text; + if (className) node.className = className; + return node; + } + + function link(text, href) { + const node = element("a", text); + node.href = href; + node.target = "_blank"; + node.rel = "noopener noreferrer"; + return node; + } + + function chip(text, pressed, data) { + const button = element("button", text, "code-search__chip"); + button.type = "button"; + button.setAttribute("aria-pressed", String(pressed)); + Object.assign(button.dataset, data); + return button; + } + + function pressOnly(button) { + for (const sibling of button.parentElement.querySelectorAll(".code-search__chip")) { + sibling.setAttribute("aria-pressed", String(sibling === button)); + } + } + + function list(items) { + return items.length > 1 ? + interpolate(gettext("%(first)s and %(last)s"), { + first: items.slice(0, -1).join(", "), + last: items.at(-1) + }, true) : items[0] || ""; + } + + async function request(params, signal) { + const response = await fetch(`/api/filing-code-search/?${new URLSearchParams({ + jurisdiction, zip: activeZip, existing_case: stage, document: documentKind, grouped: "true", + case_topic: caseTopic, case_filters: JSON.stringify(caseChoices), action, category, strong: String(!showAll), ...params, + })}`, { + signal, + headers: { + Accept: "application/json" + } + }).catch(error => { + if (error.name === "AbortError") throw error; + const unavailable = new Error(gettext("Code search could not be loaded. Try again shortly.")); + unavailable.retryable = true; + throw unavailable; + }); + if (!response.headers.get("content-type")?.includes("application/json")) { + const error = new Error(gettext("Code search could not be loaded. Refresh the page and try again.")); + error.retryable = response.status >= 500; + throw error; + } + const data = await response.json(); + if (!response.ok) { + const error = new Error(data.error || gettext("Code search could not be loaded. Try again.")); + error.retryable = response.status === 503; + throw error; + } + return data; + } + + // Steps + + function showStep(name, { + focus = true + } = {}) { + step = name; + for (const [key, panel] of Object.entries(panels)) panel.hidden = key !== name; + stepLabel.textContent = interpolate(gettext("Step %(number)s of 3"), { + number: STEPS[name].number + }, true); + title.textContent = STEPS[name].title(); + cancel.hidden = name !== "filing"; + back.hidden = name === "filing"; + next.hidden = name === "confirm"; + apply.hidden = name !== "confirm"; + updateNext(); + if (focus) title.focus(); + } + + function updateNext() { + next.textContent = step === "path" || chosen?.path ? + gettext("Next: check and use") : gettext("Next: court and case type"); + next.disabled = step === "filing" ? !chosen : !selected; + apply.disabled = !selected || applying; + } + + function goForward() { + if (step === "filing" && chosen?.path) { + selected = chosen.path; + renderConfirm(); + showStep("confirm"); + } else if (step === "filing" && chosen) { + showStep("path"); + if (loadedFor !== chosen) loadCourts(); + } else if (step === "path" && selected) { + renderConfirm(); + showStep("confirm"); + } + } + + function goBack() { + if (step === "confirm" && !chosen?.path) showStep("path"); + else showStep("filing"); + } + + // Step 1: search, ZIP, and case questions + + function invalidate() { + generation += 1; + controller?.abort(); + controller = null; + clearTimeout(timer); + chosen = null; + resetPaths(); + more.hidden = true; + updateNext(); + } + + function resetPaths() { + pathGeneration += 1; + pathController?.abort(); + pathController = null; + loadedFor = null; + selected = null; + } + + function clearResults() { + results.replaceChildren(); + } + + // "What does “subrogation” mean?", from the state's glossary. + function addGlossary(terms, target) { + if (!terms?.length) return; + const details = element("details", "", "code-search__glossary"); + details.append(element("summary", terms.length === 1 ? + interpolate(gettext("What does “%(term)s” mean?"), { + term: terms[0].label.toLowerCase() + }, true) : gettext("What do these terms mean?"))); + const definitions = element("dl"); + for (const term of terms) { + const meaning = element("dd", term.text); + if (term.source) meaning.append(" ", link(gettext("Learn more"), term.source)); + definitions.append(element("dt", term.label), meaning); + } + details.append(definitions); + target.append(details); + } + + function addMeanings(data, target) { + addGlossary(data.glossary, target); + const meanings = (data.concepts || []).filter(item => item.text); + const variants = data.variants || []; + if (!meanings.length && variants.length < 2) return; + const details = element("details", "", "code-search__concept-info"); + details.append(element("summary", gettext("About this filing type"))); + for (const concept of meanings) { + const definition = element("div", "", "code-search__meaning"); + let scope = gettext("General meaning"); + if (concept.scope === "county") scope = interpolate(gettext("Meaning in %(name)s County"), { + name: concept.scope_name + }, true); + else if (concept.scope !== "general") scope = interpolate(gettext("Meaning in %(name)s"), { + name: concept.scope_name + }, true); + definition.append(element("p", `${concept.label} — ${scope}: ${concept.text}`)); + if (concept.source) definition.append(link(gettext("About this concept"), concept.source)); + details.append(definition); + } + if (variants.length > 1) details.append(element("p", gettext("Also listed as:") + " " + variants.slice(1).join("; "))); + target.append(details); + } + + // Only when the search found this through a different word, such as + // "unlawful detainer" finding an eviction filing. + function relatedTerms(data) { + const reason = data.match_reason?.kind === "related_concept" ? data.match_reason : null; + if (!reason) return ""; + const asked = words(reason.query); + const labels = (data.concepts || []) + .filter(item => reason.concepts.includes(item.key) && words(item.label) !== asked) + .map(item => item.label.toLowerCase()); + return labels.length ? interpolate(gettext("Related to %(terms)s"), { + terms: list(labels) + }, true) : ""; + } + + function words(text) { + return String(text || "").toLowerCase().replace(/[^a-z0-9]+/g, " ").trim(); + } + + function contextSummary(data) { + const contexts = data.case_contexts || []; + if (!contexts.length) return ""; + return data.case_context_count > 1 ? + interpolate(gettext("%(count)s case types, including %(context)s"), { + context: contexts[0], + count: data.case_context_count + }, true) : contexts[0]; + } + + function choiceRow(name, onChoose, disabled = false) { + const row = element("div", "", "code-search__result"); + const label = element("label", "", "code-search__choice"); + const radio = element("input"); + radio.type = "radio"; + radio.name = "code-search-filing"; + radio.disabled = disabled; + // A leading "Complaint / Petition - " that many results share is + // muted (see markSharedPrefixes) so the words that differ stand out. + const [, prefix, rest] = name.match(/^(.+?\s-\s)(.+)$/) || [null, "", name]; + const text = element("span", "", "code-search__name"); + if (prefix) { + const muted = element("span", prefix, "code-search__name-prefix"); + muted.dataset.prefix = prefix; + text.append(muted); + } + text.append(rest); + label.append(radio, text); + row.append(label); + radio.addEventListener("change", onChoose); + // The whole card chooses it, except the "About" disclosure and links. + row.addEventListener("click", event => { + if (!event.target.closest("details, a, label, input")) radio.click(); + }); + return { + row, + radio + }; + } + + function addGroup(group, searchText) { + const choice = { + group, + query: searchText + }; + const { + row + } = choiceRow(group.name, () => { + chosen = choice; + updateNext(); + }); + row.classList.add("code-search__group"); + const meta = [contextSummary(group), relatedTerms(group)].filter(Boolean).join(" · "); + if (meta) { + const summaryLine = element("p", meta, "code-search__meta"); + summaryLine.title = meta; + row.append(summaryLine); + } + addMeanings(group, row); + results.append(row); + } + + // Ungrouped results are already full paths, so they skip the court step. + function addPath(path) { + const { + row + } = choiceRow(path.filing_type.name, () => { + chosen = { + path + }; + updateNext(); + }, Boolean(path.unavailable)); + row.append(element("p", `${path.case_context || path.case_type.name} · ${path.court.name}`, "code-search__meta")); + if (path.unavailable) row.append(element("p", path.unavailable)); + results.append(row); + } + + function topicLabel(value) { + return lastFacets?.case_topics.find(item => item.value === value)?.label || ""; + } + + // Only rows ranked by how many results they hold collapse; a fixed + // question's choices always show. + function question(key, label, options, pressed, collapsible = false) { + const row = element("div", "", "code-search__question"); + row.setAttribute("role", "group"); + const name = element("span", label, "code-search__question-label"); + name.id = `code-search-question-${key}`; + row.setAttribute("aria-labelledby", name.id); + const chips = element("div", "", "code-search__chips"); + // "Not sure" (last) and the current answer always stay visible. + const collapse = collapsible && options.length > CHIPS_SHOWN + 2 && !expanded.has(key); + const shown = collapse ? + options.filter((option, index) => index < CHIPS_SHOWN || index === options.length - 1 || option.value === pressed) : + options; + for (const option of shown) { + const button = chip(option.label, option.value === pressed, { + caseFacet: key, + value: option.value + }); + button.id = `code-search-chip-${key}-${option.value || "any"}`; + if (option.count) button.append(element("span", String(option.count), "code-search__chip-count")); + chips.append(button); + } + if (collapse) { + const moreChips = element("button", interpolate(gettext("%(count)s more"), { + count: options.length - shown.length + }, true), "code-search__chip code-search__chip--more"); + moreChips.type = "button"; + moreChips.id = `code-search-chip-${key}-more`; + moreChips.dataset.expand = key; + moreChips.setAttribute("aria-label", interpolate(gettext("Show %(count)s more choices for %(question)s"), { + count: options.length - shown.length, + question: label + }, true)); + chips.insertBefore(moreChips, chips.lastElementChild); + } + row.append(name, chips); + caseFilters.append(row); + return row; + } + + function addFacetHelp(facet, open) { + const facetHelp = element("details", "", "code-search__amount-help"); + const toggle = element("summary", gettext("About claim amounts")); + facetHelp.id = `code-search-help-${facet.key}`; + toggle.id = `${facetHelp.id}-toggle`; + facetHelp.open = open; + facetHelp.append(toggle, element("p", facet.help)); + if (facet.source) facetHelp.append(link(gettext("Court guidance"), facet.source)); + caseFilters.append(facetHelp); + } + + function renderCaseFilters(data) { + lastFacets = data; + caseTopic = data.case_topic || ""; + const actions = data.actions || []; + const askAction = action || (data.total >= ACTION_QUESTION_MIN && actions.length > 1); + const askTopic = data.case_topics.length > 1; + const categories = data.categories || []; + const askCategory = category || categories.length > 1; + const schema = JSON.stringify([caseTopic, caseChoices, action, category, [...expanded], askAction && actions, askCategory && categories, askTopic, data.case_topics, data.case_facets]); + caseFilters.hidden = !askAction && !askTopic && !askCategory && !data.case_facets.length; + if (schema === caseFilterSchema) return; + caseFilterSchema = schema; + const focused = caseFilters.contains(document.activeElement) ? document.activeElement.id : ""; + const opened = new Set(Array.from(caseFilters.querySelectorAll("details[open]"), item => item.id)); + caseFilters.replaceChildren(); + const notSure = gettext("Not sure"); + if (askAction) question("action", gettext("What do you want to do?"), [...actions, { + value: "", + label: notSure + }], action); + if (askTopic) question("topic", gettext("Kind of case"), [...data.case_topics, { + value: "all", + label: notSure + }], caseTopic || "all", true); + if (askCategory) question("category", gettext("Case category"), [...categories, { + value: "", + label: notSure + }], category, true); + for (const facet of data.case_facets) { + question(facet.key, facet.label, [...facet.options, { + value: "", + label: notSure + }], caseChoices[facet.key] || ""); + if (facet.help) addFacetHelp(facet, opened.has(`code-search-help-${facet.key}`)); + } + if (focused) byId(focused)?.focus(); + } + + function resetCaseFilters() { + caseTopic = ""; + caseChoices = {}; + caseFilterSchema = ""; + action = ""; + category = ""; + expanded.clear(); + lastFacets = null; + caseFilters.hidden = true; + } + + function showZipStatus(counties) { + lastCounties = counties; + const typed = postalCode.value.trim(); + zipStatus.textContent = ""; + if (typed && !ZIP_PATTERN.test(typed)) zipStatus.textContent = gettext("Enter all 5 digits."); + else if (counties.length) zipStatus.textContent = interpolate(ngettext("%(zip)s is in %(names)s County.", "%(zip)s is in %(names)s counties.", counties.length), { + zip: typed.slice(0, 5), + names: list(counties) + }, true); + renderZipShortcuts(); + } + + function renderZipShortcuts() { + const typed = postalCode.value.trim().slice(0, 5); + const options = zipShortcuts.filter(item => item.zip !== typed); + zipShortcutList.hidden = !options.length; + zipShortcutList.replaceChildren(); + const own = options.find(item => item.kind === "address"); + const recent = options.filter(item => item.kind === "recent"); + const shortcut = (text, zip, label) => { + const button = element("button", text, "code-search__link"); + button.type = "button"; + button.dataset.zipShortcut = zip; + if (label) button.setAttribute("aria-label", label); + return button; + }; + if (own) { + zipShortcutList.append(shortcut(interpolate(gettext("Use your ZIP (%(zip)s)"), { + zip: own.zip + }, true), own.zip)); + } + if (recent.length) { + if (own) zipShortcutList.append(" · "); + zipShortcutList.append(gettext("Recent:") + " "); + recent.forEach((item, index) => { + if (index) zipShortcutList.append(", "); + zipShortcutList.append(shortcut(item.zip, item.zip, interpolate(gettext("Use recent ZIP %(zip)s"), { + zip: item.zip + }, true))); + }); + } + } + + function zipChanged() { + const typed = postalCode.value.trim(); + if (typed && !ZIP_PATTERN.test(typed)) { + showZipStatus([]); + return; + } + if (typed === activeZip) return showZipStatus(lastCounties); + activeZip = typed; + invalidate(); + clearResults(); + showZipStatus([]); + timer = setTimeout(() => search(), 350); + } + + function showOtherStage(count) { + otherStage.hidden = !count; + otherStage.textContent = stage === "yes" ? + interpolate(ngettext("%(count)s result in “new case” filings", "%(count)s results in “new case” filings", count), { + count + }, true) : + interpolate(ngettext("%(count)s result in “existing case” filings", "%(count)s results in “existing case” filings", count), { + count + }, true); + } + + function markSharedPrefixes() { + const prefixes = Array.from(results.querySelectorAll("[data-prefix]")); + const counts = {}; + for (const node of prefixes) counts[node.dataset.prefix] = (counts[node.dataset.prefix] || 0) + 1; + for (const node of prefixes) node.classList.toggle("is-shared", counts[node.dataset.prefix] >= 3); + } + + function showSearchSummary(data, text) { + markSharedPrefixes(); + if (data.case_topics || data.categories) renderCaseFilters({ + case_topics: [], + case_facets: [], + ...data + }); + showZipStatus(data.location_counties || []); + strongTotal = data.strong_total || 0; + const narrowed = !showAll && strongTotal && strongTotal < data.total; + const shownTotal = narrowed ? strongTotal : data.total; + if (!data.total) status.textContent = gettext("No matching filing types. Try fewer words or another term."); + else if (narrowed) status.textContent = interpolate(ngettext("%(count)s filing type matches “%(query)s”", "%(count)s filing types match “%(query)s”", strongTotal), { + count: strongTotal, + query: text + }, true); + else status.textContent = interpolate(ngettext("%(total)s filing type", "%(total)s filing types", data.total), { + total: data.total + }, true); + if (data.corrected_terms.length) status.textContent += ". " + gettext("Similar spellings were included."); + if (data.stale) status.textContent += ". " + gettext("These lists may be out of date."); + // Only the first page carries it; later pages keep the link as is. + if (data.other_stage_total !== undefined && data.other_stage_total !== null) showOtherStage(data.other_stage_total); + const remaining = data.total - offset; + more.hidden = offset >= 10000 || remaining <= 0; + more.textContent = offset < shownTotal ? gettext("Show more") : interpolate(ngettext("Show %(count)s more filing type used in these case types", "Show %(count)s more filing types used in these case types", remaining), { + count: remaining + }, true); + } + + async function search(append = false) { + const text = query.value.trim(); + if (!append) { + invalidate(); + offset = 0; + showAll = false; + clearResults(); + } + suggestions.hidden = Boolean(text); + results.hidden = !text; + if (!append) otherStage.hidden = true; + if (!text) { + status.textContent = ""; + return; + } + const token = generation; + controller ||= new AbortController(); + more.hidden = true; + status.textContent = gettext("Searching filing types…"); + try { + const data = await request({ + q: text, + offset + }, controller.signal); + if (token !== generation || !dialog.open) return; + if (data.groups) data.groups.forEach(group => addGroup(group, text)); + else data.results.forEach(path => addPath(path)); + offset += (data.groups || data.results).length; + showSearchSummary(data, text); + } catch (error) { + if (token === generation && error.name !== "AbortError") { + status.textContent = error.message; + if (error.retryable) { + status.textContent += " " + gettext("We will try again shortly."); + timer = setTimeout(() => search(append), 10000); + } + } + } + } + + function showStage() { + stageLabel.textContent = stage === "yes" ? gettext("Existing case") : gettext("New case"); + } + + // Step 2: court, then case type + + function answers() { + const chosenLabels = []; + const chosenAction = lastFacets?.actions?.find(item => item.value === action); + if (chosenAction) chosenLabels.push(chosenAction.label); + if (category) chosenLabels.push(category); + if (caseTopic && caseTopic !== "all") chosenLabels.push(topicLabel(caseTopic)); + for (const facet of lastFacets?.case_facets || []) { + const option = facet.options.find(item => item.value === caseChoices[facet.key]); + if (option) chosenLabels.push(option.label); + } + if (documentKind) chosenLabels.push(filterPanel.querySelector(`[data-document="${documentKind}"]`).textContent.trim()); + return chosenLabels.filter(Boolean); + } + + function renderSummary() { + summary.replaceChildren(); + const add = (term, value) => { + if (!value) return; + summary.append(element("dt", term), element("dd", value)); + }; + add(gettext("Filing"), chosen.group.name); + add(gettext("Case"), stage === "yes" ? gettext("Existing case") : gettext("New case")); + if (activeZip) { + add(gettext("Case ZIP"), lastCounties.length ? interpolate(gettext("%(zip)s · %(names)s"), { + zip: activeZip, + names: list(lastCounties) + }, true) : activeZip); + } + add(gettext("Your answers"), answers().join(" · ")); + } + + async function loadCourts() { + resetPaths(); + const target = chosen; + loadedFor = target; + const token = pathGeneration; + pathController = new AbortController(); + renderSummary(); + courtsBox.replaceChildren(); + paths.replaceChildren(); + caseTypes.hidden = true; + morePaths.hidden = true; + updateNext(); + pathStatus.textContent = gettext("Loading courts…"); + try { + const data = await request({ + q: target.query, + group: target.group.key + }, pathController.signal); + if (token !== pathGeneration || !dialog.open) return; + const preferred = court || byId("court_code")?.value; + const courts = [...data.courts].sort((a, b) => (b.code === preferred) - (a.code === preferred)); + if (courts.some(item => item.code === preferred)) court = preferred; + else court = courts.length === 1 ? courts[0].code : ""; + if (!courts.length) { + pathStatus.textContent = gettext("No court accepts this filing with your answers. Go back and change them."); + return; + } + renderCourts(courts); + await loadPaths(); + } catch (error) { + if (token === pathGeneration && error.name !== "AbortError") pathStatus.textContent = error.message; + } + } + + function renderCourts(courts) { + const question = gettext("Which court will hear the case?"); + if (courts.length <= COURT_CARD_LIMIT) { + const fieldset = element("fieldset", "", "code-search__courts"); + fieldset.append(element("legend", question)); + const grid = element("div", "", "code-search__court-cards"); + for (const item of courts) { + const label = element("label", "", "code-search__card"); + const radio = element("input"); + radio.type = "radio"; + radio.name = "code-search-court"; + radio.value = item.code; + radio.checked = item.code === court; + radio.addEventListener("change", () => { + court = item.code; + loadPaths(); + }); + label.append(radio, element("span", item.name)); + grid.append(label); + } + fieldset.append(grid); + courtsBox.append(fieldset); + } else { + const wrapper = element("div", "", "code-search__courts"); + const label = element("label", question); + const select = element("select", "", "form-select"); + select.id = "code-search-court"; + label.htmlFor = select.id; + const blank = element("option", gettext("Choose a court")); + blank.value = ""; + select.append(blank); + for (const item of courts) { + const option = element("option", item.name); + option.value = item.code; + select.append(option); + } + select.value = court; + select.addEventListener("change", () => { + court = select.value; + loadPaths(); + }); + wrapper.append(label, select); + courtsBox.append(wrapper); + } + } + + function addCaseType(path) { + const row = element("div", "", "code-search__result code-search__path-choice"); + const label = element("label", "", "code-search__choice"); + const radio = element("input"); + radio.type = "radio"; + radio.name = "code-search-result"; + radio.value = path.id; + radio.disabled = Boolean(path.unavailable); + label.append(radio, element("span", path.case_context || path.case_type.name)); + row.append(label); + const described = []; + if ((path.filing_label || path.filing_type.name).toLowerCase() !== chosen.group.name.toLowerCase()) { + const variant = element("p", interpolate(gettext("This court calls it “%(name)s”."), { + name: path.filing_type.name + }, true), "code-search__variant"); + variant.id = `code-search-variant-${path.id}`; + described.push(variant.id); + row.append(variant); + } + if (path.case_description) { + const description = element("p", path.case_description, "code-search__case-description"); + description.id = `code-search-description-${path.id}`; + described.push(description.id); + row.append(description); + } + if (described.length) radio.setAttribute("aria-describedby", described.join(" ")); + if (path.unavailable) row.append(element("p", path.unavailable)); + radio.addEventListener("change", () => { + selected = path; + updateNext(); + }); + paths.append(row); + } + + function qualifierValue(path, key) { + const value = path.qualifiers?.[key]; + return key === "amount" ? value?.value || "" : value || ""; + } + + function qualifierChoices(questionSpec, pathList) { + const values = new Map(); + for (const path of pathList) { + const value = qualifierValue(path, questionSpec.key); + if (!value) continue; + const label = questionSpec.key === "amount" ? path.qualifiers.amount.label : questionSpec.options[value]?.(); + values.set(value, label || value); + } + return values; + } + + // Each question's choices, from the paths the earlier answers leave. + function askedQuestions() { + let remaining = courtPaths; + const asked = []; + for (const questionSpec of PATH_QUESTIONS) { + const values = qualifierChoices(questionSpec, remaining); + if (values.size < 2) continue; + let answer = pathAnswers[questionSpec.key] ?? questionSpec.assume ?? ""; + if (!values.has(answer)) answer = ""; + asked.push({ + ...questionSpec, + answer, + choices: [...values].sort(([a], [b]) => String(a).localeCompare(String(b), undefined, { + numeric: true + })) + }); + // A path that doesn't say stays in, as with step 1's questions. + if (answer) remaining = remaining.filter(path => [answer, ""].includes(qualifierValue(path, questionSpec.key))); + } + return { + asked, + remaining + }; + } + + function renderCaseTypes() { + const { + asked, + remaining + } = askedQuestions(); + const focused = pathQuestions.contains(document.activeElement) ? document.activeElement.id : ""; + pathQuestions.replaceChildren(); + for (const item of asked) { + const row = element("div", "", "code-search__question"); + row.setAttribute("role", "group"); + const name = element("span", item.label(), "code-search__question-label"); + name.id = `code-search-path-question-${item.key}`; + row.setAttribute("aria-labelledby", name.id); + const chips = element("div", "", "code-search__chips"); + for (const [value, label] of [...item.choices, ["", gettext("Not sure")]]) { + const button = chip(label, value === item.answer, { + pathQuestion: item.key, + value + }); + button.id = `code-search-path-chip-${item.key}-${value || "any"}`; + chips.append(button); + } + row.append(name, chips); + pathQuestions.append(row); + } + pathQuestions.hidden = !asked.length; + if (focused) byId(focused)?.focus(); + if (selected && !remaining.includes(selected)) selected = null; + paths.replaceChildren(); + const terms = new Map(remaining.flatMap(path => path.glossary || []).map(term => [term.key, term])); + addGlossary([...terms.values()], paths); + remaining.forEach(addCaseType); + for (const radio of paths.querySelectorAll("input")) radio.checked = String(selected?.id) === radio.value; + caseTypes.hidden = !remaining.length; + pathStatus.textContent = remaining.length ? + interpolate(ngettext("%(count)s case type fits", "%(count)s case types fit", remaining.length), { + count: remaining.length + }, true) : gettext("No matching case types in this court."); + updateNext(); + } + + async function loadPaths(append = false) { + const token = ++pathGeneration; + pathController?.abort(); + pathController = new AbortController(); + if (!append) { + pathOffset = 0; + courtPaths = []; + paths.replaceChildren(); + pathQuestions.hidden = true; + selected = null; + updateNext(); + } + morePaths.hidden = true; + if (!court) { + caseTypes.hidden = true; + pathStatus.textContent = gettext("Choose a court to see its case types."); + return; + } + pathStatus.textContent = gettext("Loading case types…"); + try { + const data = await request({ + q: chosen.query, + group: chosen.group.key, + court, + offset: pathOffset, + limit: 500 + }, pathController.signal); + if (token !== pathGeneration || !dialog.open) return; + courtPaths = courtPaths.concat(data.results); + pathOffset += data.results.length; + renderCaseTypes(); + morePaths.hidden = pathOffset >= data.total || pathOffset >= 10000; + } catch (error) { + if (token === pathGeneration && error.name !== "AbortError") pathStatus.textContent = error.message; + } + } + + // Step 3: check and use + + function renderConfirm() { + confirmList.replaceChildren(); + applyStatus.textContent = ""; + const row = (term, value, note, changeStep, changeLabel) => { + const item = element("div", "", "code-search__confirm-row"); + const detail = element("dd"); + const text = element("span", "", "code-search__confirm-value"); + text.append(element("span", value)); + if (note) text.append(element("span", note, "code-search__meta")); + detail.append(text); + if (changeStep) { + const change = element("button", gettext("Change"), "code-search__link"); + change.type = "button"; + change.dataset.codeSearchStep = changeStep; + change.append(element("span", " " + changeLabel, "visually-hidden")); + detail.append(change); + } + item.append(element("dt", term), detail); + confirmList.append(item); + }; + const viaGroup = !chosen?.path; + row(gettext("Court"), selected.court.name, "", viaGroup && "path", gettext("court")); + row(gettext("Case category"), selected.case_category.name); + row(gettext("Case type"), selected.case_type.name, selected.case_description, viaGroup && "path", gettext("case type")); + const renamed = viaGroup && selected.filing_type.name.toLowerCase() !== chosen.group.name.toLowerCase(); + row(gettext("Filing type"), selected.filing_type.name, renamed ? interpolate(gettext("This court's name for “%(name)s”"), { + name: chosen.group.name + }, true) : "", "filing", gettext("filing type")); + explanation.replaceChildren(); + addGlossary(selected.glossary, explanation); + if (selected.explanation) explanation.append(element("p", selected.explanation)); + if (selected.case_guidance?.source) explanation.append(link(gettext("About this case type"), selected.case_guidance.source)); + if (selected.explanation_source) explanation.append(link(gettext("Court information"), selected.explanation_source)); + explanation.hidden = !explanation.children.length; + } + + // Events + + open.disabled = false; + open.addEventListener("click", () => { + stage = existingCase(); + showStage(); + showStep("filing", { + focus: false + }); + showZipStatus(lastCounties); + dialog.showModal(); + query.focus(); + search(); + }); + query.addEventListener("input", () => { + resetCaseFilters(); + invalidate(); + clearResults(); + status.textContent = gettext("Searching filing types…"); + if (!query.value.trim()) search(); + else timer = setTimeout(() => search(), 250); + }); + query.addEventListener("keydown", event => { + if (event.key === "Enter") { + event.preventDefault(); + search(); + } + }); + postalCode.addEventListener("input", zipChanged); + zipShortcutList.addEventListener("click", event => { + const button = event.target.closest("[data-zip-shortcut]"); + if (!button) return; + postalCode.value = button.dataset.zipShortcut; + postalCode.focus(); + zipChanged(); + }); + + function switchStage() { + stage = stage === "yes" ? "no" : "yes"; + showStage(); + search(); + } + stageToggle.addEventListener("click", switchStage); + otherStage.addEventListener("click", () => { + switchStage(); + query.focus(); + }); + + function showFilterToggle() { + moreFilters.setAttribute("aria-expanded", String(!filterPanel.hidden)); + if (!filterPanel.hidden) moreFilters.textContent = gettext("Fewer filters"); + else moreFilters.textContent = documentKind ? gettext("More filters (1)") : gettext("More filters"); + } + moreFilters.addEventListener("click", () => { + filterPanel.hidden = !filterPanel.hidden; + showFilterToggle(); + }); + filterPanel.addEventListener("click", event => { + const button = event.target.closest("[data-document]"); + if (!button) return; + documentKind = button.dataset.document; + pressOnly(button); + showFilterToggle(); + search(); + }); + caseFilters.addEventListener("click", event => { + const expand = event.target.closest("[data-expand]"); + if (expand) { + expanded.add(expand.dataset.expand); + caseFilterSchema = ""; + renderCaseFilters(lastFacets); + // Focus the first choice that was hidden. + const chips = byId(`code-search-question-${expand.dataset.expand}`).nextElementSibling.children; + chips[Math.min(CHIPS_SHOWN, chips.length - 1)]?.focus(); + return; + } + const button = event.target.closest("[data-case-facet]"); + if (!button) return; + const key = button.dataset.caseFacet; + pressOnly(button); + if (key === "action") action = button.dataset.value; + else if (key === "category") category = button.dataset.value; + else if (key === "topic") { + caseTopic = button.dataset.value; + caseChoices = {}; + } else if (button.dataset.value) caseChoices[key] = button.dataset.value; + else delete caseChoices[key]; + search(); + }); + suggestions.addEventListener("click", event => { + const button = event.target.closest("[data-code-query]"); + if (!button) return; + query.value = button.dataset.codeQuery; + query.focus(); + search(); + }); + more.addEventListener("click", () => { + if (offset >= strongTotal) showAll = true; + search(true); + }); + morePaths.addEventListener("click", () => loadPaths(true)); + pathQuestions.addEventListener("click", event => { + const button = event.target.closest("[data-path-question]"); + if (!button) return; + pathAnswers[button.dataset.pathQuestion] = button.dataset.value; + renderCaseTypes(); + }); + next.addEventListener("click", goForward); + back.addEventListener("click", goBack); + dialog.addEventListener("click", event => { + const button = event.target.closest("[data-code-search-step]"); + if (button && !applying) showStep(button.dataset.codeSearchStep); + }); + + function close() { + if (!applying) dialog.close(); + } + ["close-code-search", "cancel-code-search"].forEach(id => byId(id).addEventListener("click", close)); + dialog.addEventListener("cancel", event => { + if (applying) event.preventDefault(); + }); + dialog.addEventListener("keydown", event => { + if (event.key !== "Tab") return; + const controls = Array.from(dialog.querySelectorAll("button, input, select, a[href], summary, [tabindex]:not([tabindex='-1'])")) + .filter(node => node.tabIndex >= 0 && !node.matches(":disabled") && node.getClientRects().length); + const first = controls[0]; + const last = controls[controls.length - 1]; + if (event.shiftKey && document.activeElement === first) { + event.preventDefault(); + last?.focus(); + } else if (!event.shiftKey && document.activeElement === last) { + event.preventDefault(); + first?.focus(); + } + }); + dialog.addEventListener("close", () => { + invalidate(); + open.focus(); + }); + apply.addEventListener("click", async () => { + if (!selected || applying) return; + applying = true; + updateNext(); + back.disabled = true; + panels.confirm.inert = true; + applyStatus.textContent = gettext("Checking this path with the court…"); + try { + const data = await request({ + path_id: selected.id + }); + await applyPath(data.path); + dialog.close(); + } catch (error) { + applyStatus.textContent = error.message; + } finally { + applying = false; + back.disabled = false; + panels.confirm.inert = false; + updateNext(); + } + }); + } + window.filingCodeSearch = { + mount + }; +})(); \ No newline at end of file diff --git a/efile_app/efile/templates/efile/components/filing_code_search.html b/efile_app/efile/templates/efile/components/filing_code_search.html new file mode 100644 index 00000000..f66898f5 --- /dev/null +++ b/efile_app/efile/templates/efile/components/filing_code_search.html @@ -0,0 +1,186 @@ +{% load i18n %} +
+ {% translate "Not sure which code to use?" %} +

{% translate "Search in plain language across courts, case categories, case types, and filing types." %}

+ +
+{% comment %} +Three steps, each answering one question: what is being filed (search, ZIP, +short questions), where and under which case type (court, then case type), +and a last check of the full path before it fills the form. +{% endcomment %} + +
+
+

{% translate "Step 1 of 3" %}

+

{% translate "What are you filing?" %}

+
+ +
+
+
+
+
+ + +
+
+ + +
+
+
+
+

+ +
+

+ {% translate "New case" %} + · + + · + +

+
+ + +

+ +
+ + +
+ + +
+
+ + + + +
+ diff --git a/efile_app/efile/templates/efile/extraction_review.html b/efile_app/efile/templates/efile/extraction_review.html index b1686352..7889675b 100644 --- a/efile_app/efile/templates/efile/extraction_review.html +++ b/efile_app/efile/templates/efile/extraction_review.html @@ -5,6 +5,10 @@ {% block title %} {% translate "Review what we found" %} {% endblock title %} +{% block extra_css %} + +{% endblock extra_css %} {% block workflow_content %}
{% translate "Confirm case" %}
@@ -246,6 +250,7 @@

{% translate "Tell us about your case" %}

{% translate "We will use this to find your case in the court's records." %} + {% include "efile/components/filing_code_search.html" %}
{{ court_label }} @@ -520,6 +525,7 @@

{% translate "Tell us about your case" %}

{{ extraction_context|json_script:"extraction-context" }} {% endblock workflow_content %} {% block extra_js %} + diff --git a/efile_app/efile/templates/efile/staff/filing_codes.html b/efile_app/efile/templates/efile/staff/filing_codes.html new file mode 100644 index 00000000..5839f688 --- /dev/null +++ b/efile_app/efile/templates/efile/staff/filing_codes.html @@ -0,0 +1,72 @@ +{% extends "efile/staff/base.html" %} +{% block staff_content %} +

+ Codes are copied from the EFSP ({{ source }}) every day at {{ schedule }}, then indexed for code search. + Resync codes copies any courts that changed and updates their part of the index. + Rebuild code search indexes rebuilds the whole index from the copy already here, without contacting the EFSP. Use it after changing search rules. +

+ + + + + + + + + + + + {% for item in statuses %} + + + + + + + + {% endfor %} + +
JurisdictionCourts copiedLast copiedFiling paths indexedLast indexed
{{ item.jurisdiction|capfirst }}{{ item.copied_courts }}{{ item.copied_at|default:"Never" }}{{ item.indexed_paths }}{{ item.indexed_at|default:"Never" }}
+

Run now

+
+ {% csrf_token %} + + + + +
+

Recent runs

+ + + + + + + + + + + + + + {% for job in jobs %} + + + + + + + + + + {% empty %} + + + + {% endfor %} + +
RequestedActionJurisdictionByStatusFinishedResult
{{ job.requested_at }}{{ job.get_kind_display }}{{ job.jurisdiction|capfirst|default:"Every jurisdiction" }}{{ job.requested_by|default:"" }}{{ job.get_status_display }}{{ job.finished_at|default:"" }}{{ job.message|linebreaksbr }}
No runs requested yet.
+{% endblock %} diff --git a/efile_app/efile/templates/efile/staff/index.html b/efile_app/efile/templates/efile/staff/index.html index 128ef7f5..ef078f49 100644 --- a/efile_app/efile/templates/efile/staff/index.html +++ b/efile_app/efile/templates/efile/staff/index.html @@ -5,6 +5,7 @@

Staff tools

{{ block.super }} diff --git a/efile_app/efile/tests/test_case_type_guidance.py b/efile_app/efile/tests/test_case_type_guidance.py new file mode 100644 index 00000000..c8962c49 --- /dev/null +++ b/efile_app/efile/tests/test_case_type_guidance.py @@ -0,0 +1,215 @@ +"""Case facets distinguish real labels without guessing missing information.""" + +import pytest + +from efile.services.case_type_guidance import ( + amount_options, + case_guidance, + case_topic, + claim_range, + facet_metadata, + guidance_config, + matches_amount, + matches_case_filters, + topic_config, + validate_case_filters, +) + +EXAMPLES = [ + ("eviction", "Eviction", "Residential - Possession Only", "property", "residential"), + ("foreclosure", "Foreclosure", "Residential Mortgage Foreclosure", "property", "residential"), + ("debt", "Civil", "Credit card debt", "issue", "card"), + ("small_claims", "Small Claims", "Contract ($250.01 to $1,000)", "issue", "contract"), + ("contract", "Civil", "Contract - Construction", "issue", "services"), + ("personal_injury", "Personal Injury", "Motor Vehicle", "issue", "vehicle"), + ("property_damage", "Property Damage", "Vehicle Property Damage", "property", "vehicle"), + ("replevin", "Civil", "Replevin", "procedure", "replevin"), + ("divorce", "Dissolution", "Dissolution with children", "children", "with"), + ("parentage", "Family", "Establish Parentage", "issue", "establish"), + ("parenting", "Family", "Parenting time", "issue", "time"), + ("child_support", "Family", "Interstate child support", "issue", "interstate"), + ("protection", "Protection", "Stalking no contact", "issue", "stalking"), + ("adult_guardianship", "Adult Guardianship", "Guardianship of the estate", "scope", "financial"), + ("minor_guardianship", "Minor Guardianship", "Temporary guardianship of minor", "duration", "temporary"), + ("probate", "Probate", "Will contest", "issue", "contest"), + ("name_change", "Miscellaneous", "Adult name change", "person", "adult"), + ("adoption", "Adoption", "Related adult adoption", "relationship", "related"), + ("administrative_review", "Administrative Review", "Unemployment", "agency", "benefits"), + ("employment", "Civil", "Unpaid wages", "issue", "wages"), +] + + +@pytest.mark.parametrize("jurisdiction", ["illinois", "massachusetts", "vermont"]) +@pytest.mark.parametrize(("topic", "category", "name", "facet", "value"), EXAMPLES) +def test_twenty_case_families_have_working_contextual_facets(jurisdiction, topic, category, name, facet, value): + guidance = case_guidance(jurisdiction, category, name, "Complaint") + assert guidance["topic"] == topic + assert guidance[facet] == value + assert guidance["text"] and guidance["source"].startswith("https://") + label = {"category": category, "case_type": name, "name": "Complaint"} + metadata = facet_metadata(jurisdiction, "", [label], topic) + assert any(item["key"] == facet for item in metadata["case_facets"]) + assert matches_case_filters(guidance, topic=topic, filters={facet: value}) + assert not matches_case_filters(guidance, topic=topic, filters={facet: "different"}) + + +def test_catalog_is_complete_and_exclusions_are_not_positive_matches(): + assert set(guidance_config()["topics"]) == {row[0] for row in EXAMPLES} + assert case_topic("illinois", "Small Claims", "Contract - Employment") == "small_claims" + assert ( + case_topic("illinois", "Civil", "Contract - Other (Excluding Business/Employment Dispute and Debt Collection)") + == "contract" + ) + assert ( + case_guidance("illinois", "Employment", "Employment - Other (Excluding Discrimination)")["issue"] == "unknown" + ) + assert case_guidance("illinois", "Contracts", "Commercial")["topic"] == "contract" + assert case_guidance("illinois", "Unknown category", "Other") == {} + + +@pytest.mark.parametrize( + ("name", "expected"), + [ + ("Small Claims ($2,500.01 to $10K)", (250001, 1000000)), + ("Small claims (Up to $2,500)", (0, 250000)), + ("Small Claims $1,001 to $5,000", (100100, 500000)), + ("Small claims less than $500", (0, 49999)), + ("Small claims $500 or less", (0, 50000)), + ("Small claims over $5,000", (500001, None)), + ("Small Claims", None), + ("Filing fee $120", None), + ("Claim 12-34", None), + ], +) +def test_claim_range_parsing(name, expected): + assert claim_range(name) == expected + + +def test_amount_bands_are_disjoint_and_preserve_unknown_codes(): + labels = [ + {"category": "Small Claims", "case_type": name, "name": "Complaint"} + for name in [ + "Contract (Up to $2,500)", + "Contract ($2,500.01 to $10K)", + "Other", + ] + ] + assert amount_options(labels) == [ + {"value": "0:250000", "label": "Up to $2,500"}, + {"value": "250001:1000000", "label": "$2,500.01–$10,000"}, + ] + assert matches_amount(labels[0], "250000:250000") + assert not matches_amount(labels[0], "250001:1000000") + assert matches_amount(labels[1], "250001:1000000") + assert matches_amount(labels[2], "250001:1000000") + assert amount_options(labels[:1]) == [] + + +def test_amount_metadata_is_state_specific_and_only_shown_when_useful(): + labels = [ + {"category": "Small Claims", "case_type": name, "name": "Complaint"} + for name in ["Up to $1,000", "$1,000.01 to $5,000"] + ] + for jurisdiction, text in [("illinois", "$10,000"), ("massachusetts", "$7,000"), ("vermont", "medical debt")]: + data = facet_metadata(jurisdiction, "small claims", labels) + amount = next(facet for facet in data["case_facets"] if facet["key"] == "amount") + assert text in amount["help"] + assert all( + facet["key"] != "amount" for facet in facet_metadata("illinois", "small claims", labels[:1])["case_facets"] + ) + + +@pytest.mark.parametrize( + ("topic", "filters"), + [ + ("small_claims", {"amount": "x"}), + ("small_claims", {"amount": "500:100"}), + ("", {"amount": "500:100"}), + ("eviction", {"children": "with"}), + ("no_such_topic", {}), + ("", {"role": "tenant"}), + ("all", {"role": "tenant"}), + ("eviction", []), + ("eviction", {"role": ["tenant"]}), + ], +) +def test_invalid_contextual_filters_are_rejected(topic, filters): + with pytest.raises(ValueError): + validate_case_filters("illinois", topic, filters) + + +def test_local_description_precedence(monkeypatch): + config = guidance_config()["topics"]["name_change"] + monkeypatch.setitem( + config["jurisdictions"], + "illinois", + { + "text": "State wording.", + "counties": { + "Example": {"court_names": ["Example Court", "Example Other Court"], "text": "County wording."} + }, + "courts": {"Example Court": {"text": "Court wording."}}, + }, + ) + topic_config.cache_clear() + case_guidance.cache_clear() + try: + assert case_guidance("illinois", "Civil", "Name change", court_name="Example Court")["text"] == "Court wording." + assert ( + case_guidance("illinois", "Civil", "Name change", court_name="Example Other Court")["text"] + == "County wording." + ) + assert case_guidance("illinois", "Civil", "Name change", court_name="Elsewhere")["text"] == "State wording." + finally: + topic_config.cache_clear() + case_guidance.cache_clear() + + +def test_civil_searches_spanning_amounts_ask_for_one_and_name_small_claims(): + labels = [ + {"category": "Civil", "case_type": "Contract - Small Claims - $0 to $10,000 - Jury", "name": "Complaint"}, + { + "category": "Civil - Amount Claimed Greater Than $10,000", + "case_type": "Contract - Jury", + "name": "Complaint", + }, + {"category": "Civil", "case_type": "Contract - Jury", "name": "Complaint"}, + ] + data = facet_metadata("illinois", "civil complaint", labels) + amount = next(facet for facet in data["case_facets"] if facet["key"] == "amount") + assert amount["label"] == "How much are you asking for?" + assert [option["label"] for option in amount["options"]] == ["Up to $10,000 (small claims)", "$10,000.01 or more"] + assert not amount["help"] + validate_case_filters("illinois", "", {"amount": amount["options"][0]["value"]}) + + +@pytest.mark.parametrize( + ("category", "case_type", "topic", "issue"), + [ + ("Civil", "Personal Injury Complaint - Jury", "personal_injury", "unknown"), + ("Civil", "Tort - Not Personal Injury - Jury", "", None), + ("Other Actions", "Legal Malpractice", "", None), + ("Personal Injury/Wrongful Death", "Medical Malpractice", "personal_injury", "medical"), + ("Personal Injury/Wrongful Death", "Other Personal Injury/Wrongful Death", "personal_injury", "unknown"), + ("Personal Injury/Wrongful Death", "Asbestos - Jury Demand", "personal_injury", "exposure"), + ("Personal Injury/Wrongful Death", "Product Liability", "personal_injury", "product"), + ], +) +def test_injury_case_types_are_classified_without_false_matches(category, case_type, topic, issue): + guidance = case_guidance("illinois", category, case_type, "Complaint / Petition") + assert guidance.get("topic", "") == topic + assert guidance.get("issue") == issue + + +def test_something_else_keeps_only_case_types_no_option_names(): + general = case_guidance("illinois", "Civil", "Personal Injury Complaint", "Complaint") + vehicle = case_guidance("illinois", "Civil", "Personal Injury - Motor Vehicle", "Complaint") + assert matches_case_filters(general, filters={"issue": "other"}) + assert not matches_case_filters(vehicle, filters={"issue": "other"}) + assert matches_case_filters(general, filters={"issue": "vehicle"}) + validate_case_filters("illinois", "personal_injury", {"issue": "other"}) + with pytest.raises(ValueError): + validate_case_filters("illinois", "eviction", {"role": "other"}) + labels = [{"category": "Civil", "case_type": "Personal Injury - Motor Vehicle", "name": "Complaint"}] + issue = next(f for f in facet_metadata("illinois", "", labels)["case_facets"] if f["key"] == "issue") + assert issue["options"][-1] == {"value": "other", "label": "Something else"} diff --git a/efile_app/efile/tests/test_filing_code_copy.py b/efile_app/efile/tests/test_filing_code_copy.py new file mode 100644 index 00000000..9597aa35 --- /dev/null +++ b/efile_app/efile/tests/test_filing_code_copy.py @@ -0,0 +1,188 @@ +"""Copying codes from the EFSP database, indexing the copy, scheduling, and staff requests.""" + +from contextlib import contextmanager +from datetime import UTC, datetime +from unittest.mock import MagicMock, patch + +import pytest +from django.urls import reverse + +from efile.management.commands.refresh_filing_code_index import next_daily_run +from efile.models import FilingCodeCourtCatalog, FilingCodeJob, StaffAudit +from efile.services import filing_code_copy +from efile.services.filing_code_copy import EfspCodesDatabase, copy_codes, rebuild, resync, run_jobs +from efile.services.filing_code_search import current_index, search_grouped_paths +from efile.tests.test_staff_tools import verified_client + +pytestmark = pytest.mark.django_db + + +def court_export(code, name, filings=("Complaint", "Answer")): + return { + "court": {"code": code, "name": name, "revision": f"{code}-{len(filings)}"}, + "categories": [{"code": "civ", "name": "Civil"}], + "case_types": [{"code": "ct", "name": "Eviction", "case_category": "civ", "initial": True}], + "filing_types": [ + {"code": f"f{i}", "name": filing, "case_category": "civ", "case_type": "", "timing": "Both"} + for i, filing in enumerate(filings) + ], + } + + +class FakeEfsp: + """Stands in for the EFSP database: courts by code, and which ones were read.""" + + def __init__(self, *exports): + self.exports = {export["court"]["code"]: export for export in exports} + self.read = [] + + @contextmanager + def __call__(self, jurisdiction): + source = MagicMock() + source.manifest.side_effect = lambda: {code: export["court"] for code, export in self.exports.items()} + source.court.side_effect = lambda court: self.read.append(court["code"]) or self.exports[court["code"]] + yield source + + +@pytest.fixture +def efsp(): + fake = FakeEfsp(court_export("adams", "Adams County"), court_export("cook:cvd1", "Cook County")) + with patch.object(filing_code_copy, "EfspCodesDatabase", fake): + yield fake + + +def test_copy_reads_only_changed_courts_and_drops_removed_ones(efsp): + assert copy_codes("illinois") == {"courts": 2, "changed": 2, "removed": 0} + assert copy_codes("illinois")["changed"] == 0 + efsp.exports["adams"] = court_export("adams", "Adams County", ("Complaint", "Answer", "Motion")) + del efsp.exports["cook:cvd1"] + efsp.read.clear() + assert copy_codes("illinois") == {"courts": 1, "changed": 1, "removed": 1} + assert efsp.read == ["adams"] + assert list(FilingCodeCourtCatalog.objects.values_list("code", "filing_count")) == [("adams", 3)] + + +def test_a_court_that_suddenly_has_no_filings_keeps_its_old_copy(efsp): + copy_codes("illinois") + efsp.exports["adams"] = court_export("adams", "Adams County", ()) + with pytest.raises(ValueError, match="no filing types"): + copy_codes("illinois") + assert FilingCodeCourtCatalog.objects.get(code="adams").filing_count == 2 + copy_codes("illinois", force=["adams"]) + assert FilingCodeCourtCatalog.objects.get(code="adams").filing_count == 0 + + +def test_dry_run_reads_and_reports_without_saving(efsp): + assert copy_codes("illinois", dry_run=True)["changed"] == 2 + assert not FilingCodeCourtCatalog.objects.exists() + + +def test_resync_indexes_the_copy_and_rebuild_needs_no_efsp(efsp): + assert resync("illinois") > 0 + index = current_index("illinois") + assert search_grouped_paths(index, "complaint")["total"] == 1 + assert index.vocabulary["courts"] == {"adams": "Adams County", "cook:cvd1": "Cook County"} + indexed = sum(item["count"] for item in index.court_snapshots.values()) + with patch.object(filing_code_copy, "EfspCodesDatabase", side_effect=AssertionError("contacted the EFSP")): + assert rebuild("illinois") == indexed + assert search_grouped_paths(current_index("illinois"), "answer")["total"] == 1 + + +def test_rebuild_without_a_copy_says_to_resync_first(): + with pytest.raises(ValueError, match="Resync codes first"): + rebuild("illinois") + + +def test_the_efsp_connection_must_be_read_only(settings): + settings.EFSP_CODES_DATABASE_URL = "postgresql://reader@efsp.example/postgres" + connection = MagicMock() + connection.cursor.return_value.__enter__.return_value.fetchone.return_value = {"transaction_read_only": "off"} + with patch("psycopg.connect", return_value=connection), pytest.raises(ValueError, match="not read only"): + with EfspCodesDatabase("illinois"): + pass + assert connection.read_only is True + connection.close.assert_called_once() + settings.EFSP_CODES_DATABASE_URL = "" + with pytest.raises(ValueError, match="EFSP_CODES_DATABASE_URL"): + EfspCodesDatabase("illinois") + + +def test_an_incomplete_efsp_update_copies_nothing(settings): + settings.EFSP_CODES_DATABASE_URL = "postgresql://reader@efsp.example/postgres" + source = EfspCodesDatabase("illinois") + rows = [{"code": "adams", "name": "Adams County", "fileable": True, "versions": 2, "revision": "r"}] + with patch.object(EfspCodesDatabase, "_rows", return_value=rows), pytest.raises(ValueError, match="Incomplete"): + source.manifest() + # Tyler's internal "System" row is fileable but has no code lists; it is skipped, not fatal. + rows = [ + {"code": "0", "name": "System", "fileable": True, "versions": 0, "revision": "s"}, + { + "code": "cook:dv", + "name": "Cook County - Domestic Violence", + "fileable": False, + "versions": 2, + "revision": "d", + }, + {"code": "adams", "name": "Adams County", "fileable": True, "versions": 3, "revision": "a"}, + ] + with patch.object(EfspCodesDatabase, "_rows", return_value=rows): + assert list(source.manifest()) == ["adams"] + + +def test_queued_jobs_run_once_and_record_the_outcome(efsp): + resync_job = FilingCodeJob.objects.create(kind="resync", jurisdiction="illinois") + rebuild_job = FilingCodeJob.objects.create(kind="rebuild", jurisdiction="vermont") + run_jobs(["illinois", "vermont"], log=lambda line: None) + resync_job.refresh_from_db() + rebuild_job.refresh_from_db() + assert resync_job.status == "succeeded" and "illinois:" in resync_job.message + assert rebuild_job.status == "failed" and "Resync codes first" in rebuild_job.message + assert resync_job.started_at and resync_job.finished_at + + +@pytest.mark.parametrize( + ("now", "expected"), + [ + (datetime(2026, 10, 6, 20, 0, tzinfo=UTC), "2026-10-06 21:30 EDT"), # 16:00 Eastern: tonight + (datetime(2026, 10, 7, 2, 0, tzinfo=UTC), "2026-10-07 21:30 EDT"), # 22:00 Eastern: tomorrow + (datetime(2026, 11, 1, 3, 0, tzinfo=UTC), "2026-11-01 21:30 EST"), # across the DST change + ], +) +def test_daily_runs_land_at_the_configured_eastern_time(now, expected): + assert f"{next_daily_run(now, '21:30', 'America/New_York'):%Y-%m-%d %H:%M %Z}" == expected + + +@pytest.fixture +def superuser(django_user_model): + return django_user_model.objects.create_user( + username="root", password="a-strong-local-password", is_staff=True, is_superuser=True + ) + + +def test_staff_can_queue_a_resync_or_rebuild_once(client, superuser): + client = verified_client(client, superuser) + url = reverse("litefile_staff:filing_codes") + page = client.get(url) + assert page.status_code == 200 + assert b"Resync codes" in page.content and b"Rebuild code search indexes" in page.content + client.post(url, {"kind": "resync", "jurisdiction": "illinois"}) + client.post(url, {"kind": "resync", "jurisdiction": "illinois"}) + client.post(url, {"kind": "rebuild", "jurisdiction": ""}) + assert list(FilingCodeJob.objects.order_by("pk").values_list("kind", "jurisdiction", "status")) == [ + ("resync", "illinois", "queued"), + ("rebuild", "", "queued"), + ] + assert StaffAudit.objects.filter(action="filing_codes_resync", jurisdiction="illinois").count() == 1 + client.post(url, {"kind": "drop tables", "jurisdiction": "illinois"}) + assert FilingCodeJob.objects.count() == 2 + + +def test_only_superusers_reach_the_filing_codes_page(client, django_user_model): + from efile.models import StaffRoleGrant + + staff = django_user_model.objects.create_user(username="staff", password="a-strong-local-password", is_staff=True) + StaffRoleGrant.objects.create(user=staff, jurisdiction="illinois", role="accounts") + client = verified_client(client, staff) + assert client.get(reverse("litefile_staff:filing_codes")).status_code == 403 + assert client.post(reverse("litefile_staff:filing_codes"), {"kind": "resync"}).status_code == 403 + assert not FilingCodeJob.objects.exists() diff --git a/efile_app/efile/tests/test_filing_code_search.py b/efile_app/efile/tests/test_filing_code_search.py new file mode 100644 index 00000000..b1c51744 --- /dev/null +++ b/efile_app/efile/tests/test_filing_code_search.py @@ -0,0 +1,1176 @@ +"""Search preserves real paths, jurisdiction boundaries and changing catalogs.""" + +from copy import deepcopy +from importlib import import_module +from unittest.mock import Mock, patch + +import pytest +import requests +from django.core.management import call_command +from django.core.management.base import CommandError +from django.db import connection, transaction +from django.urls import reverse + +from efile.models import FilingDocument, FilingDraft, FilingParty +from efile.services.filing_code_search import ( + CodeCatalog, + concept_definitions, + concept_meaning, + current_index, + explanation_for, + filing_facets, + filing_group_key, + matching_paths, + refresh_index, + search_context, + search_grouped_paths, + search_paths, + search_tokens, + validate_path, +) +from efile.services.filing_path import clear_changed_classification + +pytestmark = pytest.mark.django_db + + +@pytest.fixture(autouse=True) +def legacy_catalog_transport(settings): + # These tests exercise the legacy reader and shared search/validation behavior. + # The bulk transport and incremental imports have separate contract tests. + settings.FILING_CODE_SYNC_MODE = "legacy" + + +def option(code, name): + return {"code": code, "name": name} + + +@pytest.fixture +def catalog(): + with patch("efile.services.filing_code_search.CodeCatalog") as factory: + api = factory.return_value + api.courts.return_value = [ + option("housing", "Boston Housing Court"), + option("district", "Boston District Court"), + ] + api.categories.return_value = [option("civil", "Civil")] + api.types.return_value = [option("sp", "Summary Process"), option("debt", "Debt collection")] + + def filings(court, category, case_type, initial): + assert category == "civil" + if case_type == "debt": + return [option("complaint", "Complaint")] if initial else [option("motion", "Motion")] + return ( + [option("spcomplaint", "Summary Process Complaint")] + if initial + else [ + option("answer", "Answer to Summary Process Complaint"), + option("dismiss", "Motion to Dismiss"), + ] + ) + + api.filings.side_effect = filings + yield api + + +@pytest.fixture +def index(catalog): + refresh_index("massachusetts") + return current_index("massachusetts") + + +@pytest.mark.parametrize("jurisdiction", ["illinois", "massachusetts", "vermont"]) +def test_synonyms_search_other_courts_and_all_facets(catalog, jurisdiction): + refresh_index(jurisdiction) + index = current_index(jurisdiction) + for query in ("eviction", "evicted", "unlawful detainer", "summary process", "forcible entry and detainer"): + data = search_paths(index, query) + assert data["total"] == 2 + assert {row["court"]["code"] for row in data["results"]} == {"housing", "district"} + data = search_paths(index, "district civil eviction complaint") + assert data["total"] == 1 + row = data["results"][0] + assert row["court"]["code"] == "district" + assert row["case_category"]["code"] == "civil" + assert row["case_type"]["code"] == "sp" + assert row["filing_type"]["code"] == "spcomplaint" + assert search_paths(index, "debt eviction")["total"] == 0 + + +def test_stems_typos_ranking_and_pagination(index): + assert search_paths(index, "motions to dismiss", initial=False)["total"] == 2 + assert search_paths(index, "complants")["corrected_terms"] + assert search_paths(index, "complants")["total"] == 4 + assert search_paths(index, "eviciton")["total"] == 2 + assert search_paths(index, "eviciton")["corrected_terms"] == ["eviction"] + assert search_paths(index, "unlawfull detainer")["total"] == 2 + assert search_paths(index, "notice", initial=False)["total"] == 0 + assert search_paths(index, "eviction", initial=False)["total"] == 4 + first = search_paths(index, "complaint", limit=1) + second = search_paths(index, "complaint", offset=1, limit=1) + assert first["total"] == 4 + assert first["results"][0]["id"] != second["results"][0]["id"] + assert search_paths(index, "the and")["total"] == 0 + assert search_paths(index, "")["total"] == 0 + + +def test_phrase_concepts_do_not_expand_individual_words(): + assert "concepteviction" not in search_tokens("summary judgment", "massachusetts") + assert "conceptprotection" not in search_tokens("order", "vermont") + assert "conceptanswer" not in search_tokens("complaint", "illinois") + + +def test_synonym_results_explain_concepts_without_changing_matches(index): + groups = search_grouped_paths(index, "unlawful detainer")["groups"] + assert len(groups) == 1 + group = groups[0] + assert group["match_reason"]["kind"] == "related_concept" + assert group["match_reason"]["concepts"] == ["eviction"] + meaning = group["concepts"][0] + assert meaning["scope"] == "state" and meaning["scope_name"] == "Massachusetts" + assert "summary process" in meaning["text"] + assert search_grouped_paths(index, "summary process")["groups"][0]["match_reason"] is None + path = search_paths(index, "unlawful detainer")["results"][0] + assert path["match_reason"]["kind"] == "related_concept" + assert path["concepts"][0]["text"] == meaning["text"] + typo = search_grouped_paths(index, "unlawfull detainer")["groups"][0] + assert typo["match_reason"]["concepts"] == ["eviction"] + assert typo["match_reason"]["corrected_terms"] == ["unlawful"] + + +def test_case_topic_is_not_a_definition_of_the_document(): + context = search_context( + "massachusetts", "unlawful detainer", [("filing_type", "Affidavit"), ("case_type", "Summary Process")] + ) + assert context["match_reason"]["kind"] == "related_concept" + assert context["concepts"][0]["key"] == "eviction" + assert "affidavit" not in context["concepts"][0]["text"].lower() + assert search_context("massachusetts", "summary judgment", [("filing_type", "Motion")])["concepts"] == [] + alternate = search_context( + "massachusetts", + "summary process", + [("filing_type", "Eviction complaint"), ("filing_type", "Summary process complaint")], + ) + assert alternate["match_reason"]["kind"] == "alternate_name" + + +def test_concept_definition_precedence_and_local_scope(monkeypatch): + config = deepcopy(concept_definitions()) + state = config["concepts"]["eviction"]["jurisdictions"]["illinois"] + state["counties"]["Cook"] = {"text": "County-specific explanation.", "source": "https://example.org/county"} + court = "Cook County - Municipal Civil - District 1 - Chicago" + state["courts"][court] = {"text": "Court-specific explanation."} + monkeypatch.setattr("efile.services.filing_code_search.concept_definitions", lambda: config) + assert concept_meaning("eviction", "illinois")["scope"] == "state" + assert concept_meaning("eviction", "illinois", "Cook County - Chancery")["text"] == "County-specific explanation." + assert concept_meaning("eviction", "illinois", "Cook County")["scope"] == "county" + specific = concept_meaning("eviction", "illinois", court) + assert specific["text"] == "Court-specific explanation." + assert specific["scope"] == "court" and specific["source"] == "" + assert concept_meaning("eviction", "illinois", "Cooksville County")["scope"] == "state" + assert concept_meaning("eviction", "vermont", court)["scope_name"] == "Vermont" + assert concept_meaning("answer", "illinois")["scope"] == "general" + + +def test_local_definition_is_applied_after_court_selection_and_without_reindex(index, monkeypatch): + before = search_grouped_paths(index, "unlawful detainer") + config = deepcopy(concept_definitions()) + config["concepts"]["eviction"]["jurisdictions"]["massachusetts"]["courts"] = { + "Boston Housing Court": {"text": "Local explanation.", "source": "https://example.org/local"} + } + monkeypatch.setattr("efile.services.filing_code_search.concept_definitions", lambda: config) + after = search_grouped_paths(index, "unlawful detainer") + assert after == before # No court has been chosen: keep the statewide meaning. + paths = search_grouped_paths(index, "unlawful detainer", group_key=after["groups"][0]["key"], court="housing") + assert paths["results"][0]["concepts"][0]["text"] == "Local explanation." + config["concepts"]["eviction"]["jurisdictions"]["massachusetts"]["text"] = "Updated statewide explanation." + assert ( + search_grouped_paths(index, "unlawful detainer")["groups"][0]["concepts"][0]["text"] + == "Updated statewide explanation." + ) + assert current_index("massachusetts").pk == index.pk + + +def test_every_search_concept_has_a_short_sourced_definition(): + from efile.services.filing_code_search import rules + + for concept in rules()[0]["concepts"]: + for state in rules()[0]["jurisdictions"]: + meaning = concept_meaning(concept, state) + assert meaning["text"] and meaning["source"].startswith("https://") + + +@pytest.mark.parametrize( + ("name", "purpose", "document"), + [ + ("Eviction Complaint", "starting", "main"), + ("Answer to Complaint", "responding", "main"), + ("Motion to Dismiss Complaint", "either", "main"), + ("Exhibit - Complaint", "unknown", "attachment"), + ("Exhibits in support of motion", "unknown", "attachment"), + ("Affidavit supporting complaint", "either", "unknown"), + ("Order on complaint", "unknown", "unknown"), + ("Unfamiliar local code", "unknown", "unknown"), + ], +) +def test_facets_preserve_document_role_and_leave_ambiguous_types_unknown(name, purpose, document): + assert filing_facets(name, "illinois") == {"purpose": purpose, "document": document} + + +def test_facets_filter_groups_before_pagination_and_preserve_exact_paths(catalog): + catalog.filings.return_value = [ + option("complaint", "Complaint"), + option("answer", "Answer to Complaint"), + option("motion", "Motion to Dismiss Complaint"), + option("exhibit", "Exhibit"), + option("unknown", "Unfamiliar local code"), + ] + catalog.filings.side_effect = None + refresh_index("massachusetts") + index = current_index("massachusetts") + all_groups = search_grouped_paths(index, "eviction") + assert all_groups["total"] == 5 + attachments = search_grouped_paths(index, "eviction", document="attachment", limit=1) + assert attachments["total"] == 1 + assert attachments["groups"][0]["name"] == "Exhibit" + assert search_grouped_paths(index, "eviction", document="attachment", offset=1)["groups"] == [] + responses = search_grouped_paths(index, "eviction", initial=False, purpose="responding", document="main") + assert responses["total"] == 1 + group = responses["groups"][0] + assert group["facets"] == {"purpose": "responding", "document": "main"} + paths = search_grouped_paths( + index, "eviction", initial=False, purpose="responding", document="main", group_key=group["key"], court="housing" + ) + assert paths["total"] == 1 and paths["results"][0]["filing_type"]["code"] == "answer" + assert paths["results"][0]["initial"] is False + assert search_grouped_paths(index, "eviction", document="unknown")["groups"][0]["name"] == "Unfamiliar local code" + + +def test_endpoint_validates_and_applies_facets(signed_in, index): + params = { + "jurisdiction": "massachusetts", + "q": "eviction", + "grouped": "true", + "existing_case": "yes", + "purpose": "responding", + "document": "main", + } + result = signed_in.get(URL, params).json() + assert result["total"] == 1 + assert result["groups"][0]["facets"]["purpose"] == "responding" + assert signed_in.get(URL, {**params, "purpose": "unexpected"}).status_code == 400 + assert signed_in.get(URL, {**params, "document": "unexpected"}).status_code == 400 + + +def test_generic_filing_names_include_the_matching_case_context(catalog): + catalog.categories.return_value = [option("small", "Small claims"), option("civil", "Civil")] + catalog.types.side_effect = lambda court, category, initial: ( + [option("small", "Small claims")] + if category == "small" + else [option("eviction", "Eviction"), option("debt", "Debt collection")] + ) + catalog.filings.side_effect = None + catalog.filings.return_value = [option("petition", "Petition")] + refresh_index("massachusetts") + index = current_index("massachusetts") + group = search_grouped_paths(index, "petition")["groups"][0] + assert group["case_context_count"] == 3 + assert set(group["case_contexts"]) == {"Small claims", "Civil › Eviction", "Civil › Debt collection"} + assert group["path_count"] == 6 + matched = search_grouped_paths(index, "eviction")["groups"][0] + assert matched["case_contexts"] == ["Civil › Eviction"] + assert matched["path_count"] == 2 + detail = search_grouped_paths(index, "petition", group_key=group["key"], court="housing") + assert {path["case_context"] for path in detail["results"]} == set(group["case_contexts"]) + small = next(item for item in group["case_context_options"] if item["label"] == "Small claims") + narrowed = search_grouped_paths(index, "petition", group_key=group["key"], court="housing", context=small["key"]) + assert narrowed["total"] == 1 + assert narrowed["results"][0]["case_category"]["code"] == "small" + assert narrowed["results"][0]["case_type"]["code"] == "small" + courts = search_grouped_paths(index, "petition", group_key=group["key"], context=small["key"]) + assert {item["code"] for item in courts["courts"]} == {"housing", "district"} + with pytest.raises(ValueError, match="case type"): + search_grouped_paths(index, "eviction", group_key=group["key"], context=small["key"]) + + +def test_context_preview_is_bounded_without_hiding_the_number_of_case_types(catalog): + catalog.types.return_value = [option(str(i), f"Case type {i}") for i in range(12)] + catalog.filings.side_effect = None + catalog.filings.return_value = [option("petition", "Petition")] + refresh_index("massachusetts") + group = search_grouped_paths(current_index("massachusetts"), "petition")["groups"][0] + assert len(group["case_contexts"]) == 8 + assert group["case_context_count"] == 12 + assert group["path_count"] == 24 + + +def test_sqlite_search_uses_full_text_index(index): + if connection.vendor != "sqlite": + pytest.skip("SQLite query plan") + paths, _ = matching_paths(index, "eviction", True) + plan = paths.explain() + assert "filing_code_search_fts VIRTUAL TABLE INDEX" in plan + assert "rowid=?" in plan + assert 'search_text" LIKE' not in str(paths.query) + # Neither SQL nor FTS query operators in user input can bypass scoping. + assert search_paths(index, 'eviction OR "debt"')["total"] == 0 + + +def test_sqlite_full_text_index_tracks_edits_deletes_and_rollback(index): + if connection.vendor != "sqlite": + pytest.skip("SQLite index triggers") + path = index.paths.filter(initial=True, case_type__code="sp").first() + with pytest.raises(RuntimeError), transaction.atomic(): + index.paths.filter(pk=path.pk).update(search_text=" debt ") + assert search_paths(index, "eviction")["total"] == 1 + assert search_paths(index, "debt")["total"] == 3 + raise RuntimeError("Abort catalog publication") + assert search_paths(index, "eviction")["total"] == 2 + index.paths.filter(pk=path.pk).update(initial=False) + assert search_paths(index, "eviction")["total"] == 1 + assert search_paths(index, "eviction", initial=False)["total"] == 5 + path.delete() + assert search_paths(index, "eviction")["total"] == 1 + assert search_paths(index, "eviction", initial=False)["total"] == 4 + with connection.cursor() as cursor: + cursor.execute("INSERT INTO filing_code_search_fts(filing_code_search_fts, rank) VALUES ('integrity-check', 1)") + + +def test_sqlite_migration_indexes_existing_catalog(index): + if connection.vendor != "sqlite": + pytest.skip("SQLite migration backfill") + migration = import_module("efile.migrations.0034_sqlite_filing_search_index") + editor = connection.schema_editor() + vars(migration)["remove_sqlite_search_index"](None, editor) + vars(migration)["add_sqlite_search_index"](None, editor) + assert search_paths(index, "eviction")["total"] == 2 + assert search_paths(index, "district civil eviction complaint")["total"] == 1 + + +def test_subsequent_only_hierarchy_is_indexed_and_validated(catalog): + catalog.categories.side_effect = lambda court, initial: [option("civil", "Civil")] if not initial else [] + catalog.types.side_effect = ( + lambda court, category, initial: [option("sp", "Summary Process")] if not initial else [] + ) + refresh_index("massachusetts") + index = current_index("massachusetts") + assert search_paths(index, "eviction", initial=True)["total"] == 0 + assert search_paths(index, "eviction", initial=False)["total"] == 4 + path = index.paths.first() + assert validate_path(path)["initial"] is False + catalog.categories.assert_called_with(path.court["code"], False) + catalog.types.assert_called_with(path.court["code"], "civil", False) + + +def test_explanations_are_exact_optional_and_regenerated(index, catalog): + path = index.paths.filter(initial=True, case_type__code="sp").first() + assert path.explanation + raw = {facet: getattr(path, facet) for facet in ("court", "case_category", "case_type", "filing_type")} + raw["filing_type"] = option("spcomplaint", "Motion to dismiss Summary Process Complaint") + assert explanation_for("massachusetts", raw, True) == ("", "") + assert explanation_for("vermont", raw, True) == ("", "") + catalog.filings.side_effect = None + catalog.filings.return_value = [option("spcomplaint", "Renamed code")] + validated = validate_path(path) + assert validated["explanation"] == "" + assert validated["filing_type"]["name"] == "Renamed code" + assert validated["options"]["court"] + + +@pytest.mark.parametrize("facet", ["court", "case_category", "case_type"]) +def test_cached_explanations_preserve_context_and_filing_timing(facet): + from efile.services.filing_code_search import _explanation_for_names, explanation_rules + + config = { + "explanations": [ + { + "jurisdictions": ["illinois"], + "initial": True, + "filing_type": ["Motion"], + facet: ["Specific context"], + "text": "Context-specific explanation", + "source": "https://example.com/court", + } + ] + } + explanation_rules.cache_clear() + _explanation_for_names.cache_clear() + try: + with patch("efile.services.filing_code_search.rules", return_value=(config, "test")): + path = {"filing_type": option("motion", "Motion"), facet: option("context", "Specific context")} + assert explanation_for("illinois", path, True)[0] == "Context-specific explanation" + assert explanation_for("illinois", path, False) == ("", "") + assert explanation_for("vermont", path, True) == ("", "") + path[facet] = option("other", "Other context") + assert explanation_for("illinois", path, True) == ("", "") + finally: + explanation_rules.cache_clear() + _explanation_for_names.cache_clear() + + +def test_failed_refresh_keeps_snapshot_then_success_removes_retired_paths(index, catalog): + ids = list(index.paths.values_list("pk", flat=True)) + stamp = index.refreshed_at + catalog.types.side_effect = requests.Timeout("unavailable") + with pytest.raises(requests.Timeout): + refresh_index("massachusetts") + index.refresh_from_db() + assert index.refreshed_at == stamp + assert list(index.paths.values_list("pk", flat=True)) == ids + catalog.types.side_effect = None + catalog.courts.return_value = [option("district", "Boston District Court")] + refresh_index("massachusetts") + assert not index.paths.filter(court__code="housing").exists() + assert not index.paths.filter(pk__in=ids).exists() + + +def test_empty_refresh_preserves_previous_index(index, catalog): + catalog.courts.return_value = [] + with pytest.raises(ValueError, match="No filing paths"): + refresh_index("massachusetts") + assert index.paths.exists() + + +def test_parallel_filing_request_failure_does_not_publish_partial_index(index, catalog): + ids = list(index.paths.values_list("pk", flat=True)) + catalog.filings.side_effect = requests.Timeout("filing branch unavailable") + with pytest.raises(requests.Timeout): + refresh_index("massachusetts") + assert list(index.paths.values_list("pk", flat=True)) == ids + + +def test_wrong_efsp_or_rules_revision_not_served(index, settings): + settings.EFSP_URL = "https://other.example/api" + assert current_index("massachusetts") is None + settings.EFSP_URL = index.source_url + index.rules_digest = "old" + index.save() + assert current_index("massachusetts") is None + + +def test_live_validation_rejects_removed_or_restricted_path(index, catalog): + path = index.paths.first() + catalog.courts.return_value = [] + with pytest.raises(ValueError, match="no longer offered"): + validate_path(path) + catalog.courts.return_value = [path.court] + with patch("efile.services.filing_code_search.filing_unavailable_message", return_value="Not available here"): + with pytest.raises(ValueError, match="Not available here"): + validate_path(path) + + +def test_search_exposes_availability_without_hiding_other_courts(index): + with patch("efile.services.filing_code_search.filing_unavailable_message", return_value="Temporarily unavailable"): + data = search_paths(index, "eviction") + assert data["total"] == 2 + assert all(row["unavailable"] == "Temporarily unavailable" for row in data["results"]) + + +def test_a_hundred_courts_and_synonyms_become_one_result(catalog): + catalog.courts.return_value = [option(f"court-{i}", f"County {i}") for i in range(100)] + catalog.types.return_value = [option("sp", "Summary Process")] + catalog.filings.side_effect = lambda court, category, case_type, initial: [ + option(f"filing-{court}", "Eviction Complaint" if int(court.split("-")[1]) % 2 else "Summary Process Complaint") + ] + refresh_index("massachusetts") + index = current_index("massachusetts") + grouped = search_grouped_paths(index, "eviction") + assert grouped["total"] == 1 + group = grouped["groups"][0] + assert group["path_count"] == 100 + assert set(group["variants"]) == {"Eviction Complaint", "Summary Process Complaint"} + courts = search_grouped_paths(index, "eviction", group_key=group["key"])["courts"] + assert len(courts) == 100 + detail = search_grouped_paths(index, "eviction", group_key=group["key"], court="court-7") + assert detail["total"] == 1 + assert detail["results"][0]["court"]["code"] == "court-7" + assert detail["results"][0]["filing_type"]["code"] == "filing-court-7" + assert search_grouped_paths(index, "County 7 eviction")["groups"][0]["path_count"] == 1 + assert search_grouped_paths(index, "eviction", group_key=group["key"], court="unknown")["total"] == 0 + + +def test_grouping_keeps_different_documents_separate(): + def key(name): + return filing_group_key(name, "massachusetts") + + assert key("Answer") == key("Answer to Complaint") + assert key("Affidavit") == key("Affidavit filed.") + assert key("Eviction Complaint") == key("E-filed - Eviction Complaint filed.") + assert key("Eviction Complaint") == key("Complaint for Summary Process") + assert key("Eviction") == key("Summary Process") + assert key("Fee waiver") == key("In forma pauperis") + assert key("Divorce") == key("Dissolution of marriage") + assert key("Complaint") != key("Answer to Complaint") + assert key("Motion to Dismiss") != key("Complaint") + assert key("Motion to Dismiss") != key("Dismissal of Motion") + assert key("Verified Complaint") != key("Complaint") + assert key("Proposed Order") != key("Order") + assert key("Answer and Counterclaim") != key("Answer") + assert key("Motion for Summary Judgment") != key("Summary Process Complaint") + + +def test_grouped_endpoint_keeps_court_selection_separate(signed_in, index): + params = {"jurisdiction": "massachusetts", "q": "eviction", "grouped": "true"} + result = signed_in.get(URL, params).json() + assert result["total"] == 1 + assert "results" not in result + params["group"] = result["groups"][0]["key"] + assert len(signed_in.get(URL, params).json()["courts"]) == 2 + params["court"] = "housing" + paths = signed_in.get(URL, params).json()["results"] + assert len(paths) == 1 and paths[0]["court"]["code"] == "housing" + params["group"] = "retired-group" + assert signed_in.get(URL, params).status_code == 409 + + +def test_refresh_resumes_completed_courts_after_late_failure(catalog, tmp_path): + def fail_district(court, category, initial): + if court == "district": + raise requests.Timeout("late failure") + return [option("sp", "Summary Process")] + + catalog.types.side_effect = fail_district + with pytest.raises(requests.Timeout): + refresh_index("massachusetts", cache_dir=tmp_path) + assert current_index("massachusetts") is None + assert len(list(tmp_path.rglob("*.json.gz"))) == 1 + catalog.types.reset_mock() + catalog.types.side_effect = None + catalog.types.return_value = [option("sp", "Summary Process")] + refresh_index("massachusetts", cache_dir=tmp_path) + assert current_index("massachusetts").paths.count() == 6 + assert {call.args[0] for call in catalog.types.call_args_list} == {"district"} + + +def test_catalog_retries_failures_while_reading_response_body(): + with ( + patch("efile.services.filing_code_search.requests.Session") as session, + patch("efile.services.filing_code_search.time.sleep") as sleep, + ): + response = session.return_value.get.return_value + response.json.side_effect = [requests.ConnectionError("body timed out"), [{"code": "civil", "name": "Civil"}]] + api = CodeCatalog("illinois") + assert api.categories("court") == [option("civil", "Civil")] + assert session.return_value.get.call_count == 2 + sleep.assert_called_once_with(1) + api.close() + + +def test_catalog_encodes_codes_and_rejects_malformed_data(): + with patch("efile.services.filing_code_search.requests.Session") as session: + response = session.return_value.get.return_value + response.json.return_value = [{"code": 123, "name": "Civil"}] + api = CodeCatalog("illinois") + assert api.categories("cook:cvd1") == [option("123", "Civil")] + assert "cook%3Acvd1/categories" in session.return_value.get.call_args.args[0] + response.json.return_value = {"error": "oops"} + with pytest.raises(ValueError): + api.courts() + response.json.return_value = [{"name": "No code"}] + with pytest.raises(ValueError): + api.courts() + response.raise_for_status.side_effect = requests.HTTPError() + with pytest.raises(requests.HTTPError): + api.courts() + + +@pytest.fixture +def signed_in(client, django_user_model): + user = django_user_model.objects.create_user(username="searcher", tyler_jurisdiction="massachusetts") + client.force_login(user) + session = client.session + session["auth_tokens"] = {"TYLER-TOKEN-MASSACHUSETTS": "test-token"} + session.save() + return client + + +URL = reverse("api:filing_code_search") + + +def test_endpoint_auth_input_missing_index_and_jurisdiction_isolation(signed_in, index, client): + for query in ("", "x" * 161): + assert signed_in.get(URL, {"jurisdiction": "massachusetts", "q": query}).status_code == 400 + assert signed_in.get(URL, {"jurisdiction": "massachusetts", "q": "m"}).status_code == 200 + assert signed_in.get(URL, {"jurisdiction": "massachusetts", "q": "eviction", "offset": -1}).status_code == 400 + assert signed_in.post(URL).status_code == 405 + index.delete() + assert signed_in.get(URL, {"jurisdiction": "massachusetts", "q": "eviction"}).status_code == 503 + assert signed_in.get(URL, {"jurisdiction": "vermont", "q": "eviction"}).status_code == 403 + assert signed_in.get(URL, {"jurisdiction": "unsupported", "q": "eviction"}).status_code == 400 + client.logout() + assert client.get(URL, {"jurisdiction": "massachusetts", "q": "eviction"}).status_code == 403 + + +def test_endpoint_search_ignores_current_facets_and_validates_without_saving(signed_in, index, catalog): + params = {"jurisdiction": "massachusetts", "q": "eviction", "court": "wrong-court", "case_type": "wrong"} + response = signed_in.get(URL, params) + assert response.status_code == 200 + assert response.json()["total"] == 2 + assert response["Cache-Control"] == "no-store" + path_id = response.json()["results"][0]["id"] + response = signed_in.get(URL, {"jurisdiction": "massachusetts", "path_id": path_id}) + assert response.status_code == 200 + assert response.json()["path"]["options"] + assert not FilingDraft.objects.exists() + assert ( + signed_in.get(URL, {"jurisdiction": "massachusetts", "path_id": path_id, "existing_case": "yes"}).status_code + == 409 + ) + catalog.courts.side_effect = requests.Timeout() + assert signed_in.get(URL, {"jurisdiction": "massachusetts", "path_id": path_id}).status_code == 503 + assert signed_in.get(URL, {"jurisdiction": "massachusetts", "path_id": "bad"}).status_code == 400 + + +def test_using_a_path_remembers_its_zip_newest_first(signed_in, index, catalog, django_user_model): + path_id = signed_in.get(URL, {"jurisdiction": "massachusetts", "q": "eviction"}).json()["results"][0]["id"] + user = django_user_model.objects.get(username="searcher") + with patch("efile.api.filing_code_search.validate_path", return_value={}): + for postal_code in ("02108", "02139", "", "not-a-zip", "02108", "01060-1234", "02445"): + signed_in.get(URL, {"jurisdiction": "massachusetts", "path_id": path_id, "zip": postal_code}) + user.refresh_from_db() + assert user.recent_case_zips == ["02445", "01060", "02108"] + with patch("efile.api.filing_code_search.validate_path", side_effect=ValueError("gone")): + signed_in.get(URL, {"jurisdiction": "massachusetts", "path_id": path_id, "zip": "02210"}) + user.refresh_from_db() + assert user.recent_case_zips[0] == "02445" + + +def test_zip_shortcuts_put_the_filers_own_zip_before_recent_ones(rf, django_user_model): + from efile.views.extraction_review import _zip_shortcuts + + user = django_user_model.objects.create_user( + username="filer", tyler_jurisdiction="illinois", recent_case_zips=["60187", "60601"] + ) + draft = FilingDraft.objects.create(user=user, jurisdiction="illinois") + FilingParty.objects.create(draft=draft, role="filer", zip_code="60601-1234") + request = rf.post("/") + request.user = user + request.session = {} + assert _zip_shortcuts(request, draft, "illinois") == [ + {"zip": "60601", "kind": "address"}, + {"zip": "60187", "kind": "recent"}, + ] + + +def test_management_command_attempts_each_jurisdiction_and_reports_failure(): + with patch( + "efile.management.commands.refresh_filing_code_index.refresh_index", side_effect=[ValueError("bad"), 2, 3] + ) as refresh: + with pytest.raises(CommandError, match="illinois"): + call_command("refresh_filing_code_index", stdout=Mock(), stderr=Mock()) + assert [call.args[0] for call in refresh.call_args_list] == ["illinois", "massachusetts", "vermont"] + + +@pytest.mark.django_db(transaction=True) +def test_worker_retries_failed_state_soon_and_skips_fresh_states(index): + from efile.models import FilingCodeIndex + + command = "efile.management.commands.refresh_filing_code_index" + + # Massachusetts is already ready. Vermont succeeds on the first pass; + # Illinois fails once and must be retried without rebuilding either. + attempts = 0 + + def refresh(jurisdiction, **kwargs): + nonlocal attempts + if jurisdiction == "illinois" and attempts == 0: + attempts += 1 + raise requests.Timeout("temporary failure") + FilingCodeIndex.objects.create( + jurisdiction=jurisdiction, + source_url=index.source_url, + rules_digest=index.rules_digest, + refreshed_at=index.refreshed_at, + ) + return 2 + + with ( + patch(f"{command}.refresh_index", side_effect=refresh) as build, + patch(f"{command}.time.sleep", side_effect=[None, KeyboardInterrupt]) as sleep, + ): + with pytest.raises(KeyboardInterrupt): + call_command("refresh_filing_code_index", interval=86400, stdout=Mock(), stderr=Mock()) + assert [call.args[0] for call in build.call_args_list] == ["illinois", "vermont", "illinois"] + assert [call.args[0] for call in sleep.call_args_list] == [60, 86400] + + +def test_catalog_retries_temporary_http_failure(): + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + from threading import Thread + + class Handler(BaseHTTPRequestHandler): + attempts = 0 + + def do_GET(self): + Handler.attempts += 1 + self.send_response(503 if Handler.attempts == 1 else 200) + self.end_headers() + self.wfile.write(b'[{"code": "civil", "name": "Civil"}]') + + def log_message(self, *args): + pass + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = Thread(target=server.serve_forever) + thread.start() + catalog = CodeCatalog("illinois") + catalog.base = f"http://127.0.0.1:{server.server_port}/" + try: + assert catalog.categories("court") == [option("civil", "Civil")] + assert Handler.attempts == 2 + finally: + catalog.close() + server.shutdown() + server.server_close() + thread.join() + + +def test_classification_change_clears_dependent_codes_but_keeps_documents(django_user_model): + user = django_user_model.objects.create_user(username="draft-owner") + draft = FilingDraft.objects.create( + user=user, + jurisdiction="illinois", + court_code="old", + case_category_code="civil", + case_type_code="eviction", + quoted_fee_total="10", + quoted_fee_fingerprint="old-fees", + previous_case_id="old-case", + ) + doc = FilingDocument.objects.create( + draft=draft, + role="lead", + name="keep.pdf", + s3_key="keep.pdf", + filing_type_code="old", + document_type_code="private", + document_type_confirmed=True, + filing_component_code="old-component", + requested_optional_services=["old-service"], + ) + party = FilingParty.objects.create(draft=draft, party_type="old", first_name="Keep") + assert not clear_changed_classification(draft, "old", "civil", "eviction") + assert clear_changed_classification(draft, "new", "civil", "eviction") + doc.refresh_from_db() + party.refresh_from_db() + draft.refresh_from_db() + assert doc.s3_key == "keep.pdf" + assert not doc.filing_type_code and not doc.document_type_code and not doc.document_type_confirmed + assert not doc.filing_component_code and not doc.requested_optional_services + assert not party.party_type and party.first_name == "Keep" + assert not draft.quoted_fee_total and not draft.previous_case_id and not draft.filing_type_code + + +def test_zip_filter_scopes_groups_contexts_courts_and_paths(catalog): + catalog.courts.return_value = [option("cook:cvd1", "Cook County"), option("adams", "Adams County")] + refresh_index("illinois") + index = current_index("illinois") + all_groups = search_grouped_paths(index, "eviction") + scoped = search_grouped_paths(index, "eviction", postal_code="60601") + assert scoped["location_counties"] == ["Cook"] + group = scoped["groups"][0] + assert group["path_count"] < all_groups["groups"][0]["path_count"] + args = {"group_key": group["key"], "postal_code": "60601", "context": group["case_context_options"][0]["key"]} + courts = search_grouped_paths(index, "eviction", **args) + assert [court["code"] for court in courts["courts"]] == ["cook:cvd1"] + assert search_grouped_paths(index, "eviction", court="adams", **args)["total"] == 0 + assert search_grouped_paths(index, "eviction", court="cook:cvd1", **args)["total"] == 1 + assert search_grouped_paths(index, "eviction")["groups"] == all_groups["groups"] + + +@pytest.mark.parametrize("postal_code", ["606", "00000", "abcde"]) +def test_zip_filter_rejects_invalid_or_unknown_zip(catalog, postal_code): + from efile.services.case_location import locate + + refresh_index("illinois") + with pytest.raises(ValueError): + locate(current_index("illinois"), postal_code) + + +def test_zip_filter_preserves_multiple_counties(): + from efile.services.case_location import zip_counties + + with patch("efile.utils.zip_to_county_il.COUNTY_TO_ZIPS_IL", {"Cook": ["60601"], "Lake": ["60601"]}): + assert zip_counties("illinois", "60601-1234") == ["Cook", "Lake"] + + +def test_census_table_places_zips_in_every_state(): + from efile.services.case_location import zip_counties + + assert zip_counties("massachusetts", "02139") == ["Middlesex"] + assert zip_counties("vermont", "05602") == ["Washington"] + assert zip_counties("vermont", "02139") == [] + + +@pytest.mark.parametrize( + ("jurisdiction", "code", "name", "counties"), + [ + ("illinois", "cook:cvd1", "Cook County - Civil", {"Cook"}), + ("illinois", "jodaviess", "Jo Daviess County", {"Jo Daviess"}), + ("vermont", "sc:grandisle", "Grand Isle Unit", {"Grand Isle"}), + ( + "massachusetts", + "0965:BE", + "Juvenile Court -- Franklin-Hampshire County -- Belchertown", + {"Franklin", "Hampshire"}, + ), + ("massachusetts", "490", "District Court - Cambridge", {"Middlesex"}), + ("illinois", "ilsc", "Illinois Supreme Court", set()), + ], +) +def test_courts_are_placed_by_county_in_their_code_or_name_or_by_town(jurisdiction, code, name, counties): + from efile.services.case_location import court_counties + + assert set(court_counties(jurisdiction, code, name)) == counties + + +def test_county_states_keep_courts_that_name_no_county(catalog): + from efile.services.case_location import locate + + refresh_index("illinois") + index = current_index("illinois") + index.vocabulary["courts"] = {"cook:cvd1": "Cook County", "adams": "Adams County", "ilsc": "Supreme Court"} + assert locate(index, "60601") == {"counties": ["Cook"], "courts": ["cook:cvd1", "ilsc"]} + + +def test_matcher_states_narrow_only_courts_the_matcher_knows(index): + from efile.services import court_location + from efile.services.case_location import locate + + record = lambda code: Mock(tyler_code=code) # noqa: E731 + finder = Mock() + finder.find_by_postal_code.return_value = [Mock(records=[record("0490")])] + finder.catalog.records = [record("490"), record("500")] + index.vocabulary["courts"] = { + "490": "Cambridge", + "500": "Somewhere else", + "sjc": "Supreme Judicial Court", + "new-1": "District Court - Winchendon", + "new-2": "District Court - Somerville", + } + with patch.object(court_location, "_finder", return_value=finder): + # Courts MACourts doesn't know fall back to their town's county. + assert locate(index, "02139")["courts"] == ["490", "new-2", "sjc"] + finder.find_by_postal_code.return_value = [] + with pytest.raises(ValueError): + locate(index, "02139") + finder.find_by_postal_code.assert_called_with("02139") + + +def test_vermont_narrows_units_by_name_and_keeps_statewide_courts(): + from efile.services import court_location + + finder = Mock() + finder.find_units.return_value = [Mock(unit="Washington")] + courts = {"sc:washington": "Washington Unit", "sc:orange": "Orange Unit", "vermont:supreme": "Supreme Court"} + with patch.object(court_location, "_vermont_finder", return_value=finder): + assert court_location.courts_for_postal_code("vtcourts", "05602", courts) == { + "matched": {"sc:washington"}, + "unknown": {"vermont:supreme"}, + } + + +def test_refresh_records_each_courts_name(catalog): + refresh_index("illinois") + assert current_index("illinois").vocabulary["courts"] + + +def test_overlapping_words_do_not_need_a_synonym_explanation(): + data = search_context( + "illinois", "response to complaint", [("filing_type", "Answer/Response to Complaint/Petition")] + ) + assert data["match_reason"] is None + data = search_context("illinois", "complaint response", [("filing_type", "Answer/Response to Complaint/Petition")]) + assert data["match_reason"] is None + data = search_context("illinois", "unlawful detainer", [("filing_type", "Eviction")]) + assert data["match_reason"]["kind"] == "related_concept" + + +def test_path_description_uses_case_meaning_not_document_definition(index): + path = search_paths(index, "eviction")["results"][0] + assert path["case_description"] == concept_meaning("eviction", "massachusetts", path["court"]["name"])["text"] + debt = search_paths(index, "debt")["results"][0] + assert "creditor" in debt["case_description"] + + +@pytest.mark.parametrize( + ("name", "property_kind", "relief", "text"), + [ + ("Commercial - Possession Only", "commercial", "possession", "business"), + ("Residential - Eviction", "residential", "unknown", "does not say whether money"), + ("Residential - Possession and Rent", "residential", "money", "money claim"), + ("Ejectment", "unknown", "unknown", "legal right"), + ], +) +def test_eviction_case_type_guidance(name, property_kind, relief, text): + from efile.services.case_type_guidance import case_guidance + + guidance = case_guidance("illinois", "Eviction", name, "Answer/Response to Complaint/Petition") + assert guidance["property"] == property_kind + assert guidance["relief"] == relief + assert guidance["role"] == "tenant" + assert text in guidance["text"] + assert case_guidance("illinois", "Contracts", "Commercial")["topic"] != "eviction" + assert case_guidance("vermont", "Eviction", name)["topic"] == "eviction" + + +def test_eviction_facets_filter_before_pagination_and_preserve_unknowns(catalog): + catalog.categories.return_value = [option("eviction", "Eviction")] + catalog.types.return_value = [ + option("business", "Commercial"), + option("home", "Residential - Possession Only"), + option("other", "Ejectment"), + ] + catalog.filings.side_effect = None + catalog.filings.return_value = [ + option("answer", "Answer"), + option("complaint", "Complaint"), + option("motion", "Motion"), + ] + refresh_index("illinois") + index = current_index("illinois") + data = search_grouped_paths( + index, "eviction", role="tenant", property_kind="residential", relief="possession", limit=1 + ) + assert data["eviction_facets"] + assert data["total"] == 2 # Answer and Motion; landlord Complaint excluded. + group = data["groups"][0] + assert "Eviction › Commercial" not in group["case_contexts"] + assert "Eviction › Ejectment" in group["case_contexts"] # Unspecified property/relief stays visible. + paths = search_grouped_paths( + index, + "eviction", + group_key=group["key"], + court="housing", + role="tenant", + property_kind="residential", + relief="possession", + ) + assert paths["total"] == 2 + assert {p["case_type"]["code"] for p in paths["results"]} == {"home", "other"} + assert any("home or apartment" in p["case_description"] for p in paths["results"]) + assert search_grouped_paths(index, "eviction")["total"] == 3 + + +def test_small_claim_amount_filters_groups_courts_and_exact_paths(catalog): + catalog.categories.return_value = [option("sc", "Small Claims")] + catalog.types.return_value = [ + option("low", "Contract (Up to $2,500)"), + option("high", "Contract ($2,500.01 to $10K)"), + option("unknown", "Other"), + ] + catalog.filings.side_effect = None + catalog.filings.return_value = [option("complaint", "Complaint")] + refresh_index("illinois") + index = current_index("illinois") + args = {"case_topic": "small_claims", "case_filters": {"amount": "250001:1000000"}} + data = search_grouped_paths(index, "small claims", **args) + assert data["case_topic"] == "small_claims" + assert data["total"] == 1 + group = data["groups"][0] + assert group["path_count"] == 4 + assert not any("Up to $2,500" in label for label in group["case_contexts"]) + detail = search_grouped_paths(index, "small claims", group_key=group["key"], court="housing", **args) + assert {path["case_type"]["code"] for path in detail["results"]} == {"high", "unknown"} + assert detail["total"] == 2 + assert len(search_grouped_paths(index, "small claims", group_key=group["key"], **args)["courts"]) == 2 + assert search_grouped_paths(index, "small claims")["groups"][0]["path_count"] == 6 + + +@pytest.mark.parametrize( + ("name", "action"), + [ + ("Small Claims Complaint", "start"), + ("Answer/Response to Complaint/Petition", "respond"), + ("Response to Motion to Dismiss", "respond"), + ("Motion to Dismiss", "ask"), + ("Stipulation to Dismiss", "settle"), + ("Agreed Order", "settle"), + ("Petition for Dissolution of Marriage", "start"), + ("Petition to Modify Custody", "ask"), + ("Certificate of Service", "proof"), + ("Proposed Order", "order"), + ("Notice of Hearing", "notice"), + ("Additional Defendants", ""), + ], +) +def test_filing_names_map_to_a_plain_language_action(name, action): + from efile.services.filing_code_search import filing_action + + assert filing_action(name) == action + + +def test_jury_government_and_fee_variants_share_one_group(): + from efile.services.filing_code_search import filing_group_key, unqualified_label + + names = [ + "Complaint / Petition - Fraud - Fee", + "Complaint / Petition - Fraud (Jury - 12) - Fee", + "Complaint / Petition - Fraud (Govn't) - Fee", + "Complaint / Petition - Fraud (Jury - 6) (Govn't)", + ] + assert {unqualified_label(name) for name in names} == {"Complaint / Petition - Fraud"} + assert len({filing_group_key(name, "illinois") for name in names}) == 1 + assert filing_group_key("Complaint / Petition - Fraud", "illinois") != filing_group_key( + "Complaint / Petition - Breach Of Contract", "illinois" + ) + + +def test_name_matches_lead_and_case_only_matches_are_counted_apart(): + from efile.services.filing_code_search import action_options, strong_matches + + groups = [ + {"name_rank": 24, "_labels": [{"name": "Small Claims Complaint"}]}, + {"name_rank": 16, "_labels": [{"name": "Notice of Small Claims"}]}, + {"name_rank": 8, "_labels": [{"name": "Claim Affidavit"}]}, + {"name_rank": 0, "_labels": [{"name": "Affidavit"}, {"name": "Exhibit"}]}, + ] + assert strong_matches(groups) == 2 + assert strong_matches([{"name_rank": 0}]) == 0 + assert action_options(groups) == [ + {"value": "start", "label": "Start a case", "count": 1}, + {"value": "proof", "label": "Supporting papers", "count": 2}, + {"value": "notice", "label": "Notice", "count": 1}, + ] + + +def test_endpoint_filters_by_action_and_rejects_unknown_ones(signed_in, index): + params = {"jurisdiction": "massachusetts", "q": "eviction", "grouped": "true"} + data = signed_in.get(URL, params).json() + assert {"actions", "strong_total"} <= set(data) + assert signed_in.get(URL, {**params, "action": "bogus"}).status_code == 400 + for option in data["actions"]: + narrowed = signed_in.get(URL, {**params, "action": option["value"]}).json() + assert narrowed["total"] == option["count"] + + +@pytest.mark.parametrize( + ("category", "case_type", "filing", "expected"), + [ + ( + "Civil", + "Other Personal Injury Complaint - Jury - Self-Represented Litigant", + "Complaint / Petition - Personal Injury (Jury - 6) (Govn't) - Fee", + {"jury": "6", "representation": "self", "government": "yes", "amount": None}, + ), + ( + "Civil - Amount Claimed Greater Than $10,000", + "Personal Injury Complaint - Non-Jury", + "Complaint / Petition - Personal Injury - Fee", + { + "jury": "none", + "representation": "lawyer", + "government": "no", + "amount": {"value": "1000001:", "label": "More than $10,000"}, + }, + ), + ( + "Civil", + "Personal Injury Complaint - Non-Jury- Small Claims $0 to $10,000- SRL", + "Complaint / Petition - Personal Injury - Fee", + { + "jury": "none", + "representation": "self", + "government": "no", + "amount": {"value": "0:1000000", "label": "$0 to $10,000"}, + }, + ), + ( + "Personal Injury/Wrongful Death", + "Other Personal Injury/Wrongful Death - Jury Demand", + "Complaint / Petition - Personal Injury (Jury - 12) - Fee", + {"jury": "12", "representation": "lawyer", "government": "no", "amount": None}, + ), + ], +) +def test_paths_are_tagged_with_the_choices_courts_code_separately(category, case_type, filing, expected): + from efile.services.filing_code_search import path_qualifiers + + assert path_qualifiers(category, case_type, filing) == expected + + +def test_glossary_finds_terms_once_with_state_wording(): + from efile.services.glossary import glossary_for + + terms = glossary_for( + "illinois", ["Complaint / Petition - Intentional Tort", "Personal Injury - Motor Vehicle Subrogation - SRL"] + ) + assert [term["label"] for term in terms] == ["Intentional tort", "Subrogation"] + assert glossary_for("illinois", ["Tortious Interference? No: Torte Bakery"]) == [] + assert "Illinois' older name" in glossary_for("illinois", ["Forcible Entry and Detainer"])[0]["text"] + assert glossary_for("massachusetts", ["Summary Process Complaint"])[0]["label"] == "Summary process" + assert glossary_for("illinois", ["Summary Process Complaint"]) == [] + + +def test_case_type_variants_are_shown_once_in_previews(): + from efile.services.filing_code_search import case_contexts, unqualified_case_type + + assert unqualified_case_type("Personal Injury Complaint - Non-Jury- Small Claims $0 to $10,000- SRL") == ( + "Personal Injury Complaint" + ) + assert unqualified_case_type("Other Personal Injury/Wrongful Death - Jury Demand") == ( + "Other Personal Injury/Wrongful Death" + ) + labels = [ + {"category": "Civil", "case_type": name, "path_count": 1, "rank": 1} + for name in [ + "Personal Injury Complaint - Jury", + "Personal Injury Complaint - Non-Jury - Self-Represented Litigant", + ] + ] + assert case_contexts(labels) == ["Civil › Personal Injury Complaint"] + + +def test_fee_waivers_ask_the_court_rather_than_start_a_case(): + from efile.services.filing_code_search import filing_action + + assert filing_action("Fee Waiver Petition Filed - Petitioner/Plaintiff") == "ask" + + +def test_endpoint_offers_and_applies_case_categories(signed_in, index): + params = {"jurisdiction": "massachusetts", "q": "eviction", "grouped": "true"} + data = signed_in.get(URL, params).json() + assert data["categories"] and all({"value", "label", "count"} <= set(item) for item in data["categories"]) + first = data["categories"][0] + narrowed = signed_in.get(URL, {**params, "category": first["value"]}).json() + assert narrowed["total"] == first["count"] + + +def test_data_files_have_no_duplicate_keys(): + # YAML lets a repeated key silently replace the first, as a second + # "sealing:" concept once did. + from pathlib import Path + + import yaml + + class Strict(yaml.SafeLoader): + pass + + def mapping(loader, node, deep=False): + keys = [loader.construct_object(key, deep=deep) for key, _ in node.value] + assert len(keys) == len(set(keys)), f"duplicate keys in {node.start_mark}" + return yaml.SafeLoader.construct_mapping(loader, node, deep) + + Strict.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, mapping) + for path in (Path(__file__).resolve().parents[1] / "data").glob("*.yaml"): + yaml.load(path.read_text(), Loader=Strict) + + +def test_seal_searches_find_impound_and_redaction_filings(): + from efile.services.filing_code_search import search_tokens + + for name in ["Impound/Seal", "Filed Under Seal", "Motion for Redaction & Confidential Filing"]: + assert "conceptsealing" in search_tokens(name, "illinois") + assert "conceptsealing" in search_tokens("seal eviction", "illinois", query=True) + assert "conceptsealing" not in search_tokens("Other Document Not Listed (Confidential)", "illinois") + + +def test_other_stage_count_points_new_case_searches_at_existing_filings(catalog): + from efile.services.filing_code_search import search_grouped_paths + + refresh_index("massachusetts") + index = current_index("massachusetts") + new = search_grouped_paths(index, "eviction", initial=True) + existing = search_grouped_paths(index, "eviction", initial=False) + assert new["other_stage_total"] == existing["total"] + assert existing["other_stage_total"] == new["total"] + assert search_grouped_paths(index, "eviction", initial=True, offset=20)["other_stage_total"] is None diff --git a/efile_app/efile/tests/test_filing_code_sync.py b/efile_app/efile/tests/test_filing_code_sync.py new file mode 100644 index 00000000..3934ea0e --- /dev/null +++ b/efile_app/efile/tests/test_filing_code_sync.py @@ -0,0 +1,196 @@ +"""Incremental imports preserve unchanged courts and roll back failed updates.""" + +from copy import deepcopy +from unittest.mock import patch + +import pytest +import requests + +from efile.services.filing_code_search import current_index, refresh_index, search_paths +from efile.services.filing_code_sync import export_entries + +pytestmark = pytest.mark.django_db + + +def export(code="housing", name="Boston Housing Court", label="first"): + return { + "version": 1, + "jurisdiction": "massachusetts", + "court": {"code": code, "name": name, "revision": code + label}, + "categories": [{"code": "civil", "name": "Civil"}], + "case_types": [ + {"code": "sp", "name": "Summary Process", "case_category": "civil", "initial": True}, + {"code": "debt", "name": "Debt collection", "case_category": "civil", "initial": False}, + ], + "filing_types": [ + {"code": "complaint", "name": "Complaint", "case_category": "", "case_type": "", "timing": "Initial"}, + {"code": "answer", "name": "Answer", "case_category": "", "case_type": "", "timing": "Subsequent"}, + { + "code": "motion", + "name": "Motion to Dismiss", + "case_category": None, + "case_type": "sp", + "timing": "Subsequent", + }, + ], + } + + +@pytest.fixture +def bulk(settings): + settings.FILING_CODE_SYNC_MODE = "bulk" + data = { + "housing": export(), + "district": export("district", "Boston District Court"), + } + with patch("efile.services.filing_code_sync.BulkCodeCatalog") as factory: + catalog = factory.return_value + catalog.jurisdiction = "massachusetts" + catalog.manifest.side_effect = lambda: {code: deepcopy(item["court"]) for code, item in data.items()} + catalog.court.side_effect = lambda court: deepcopy(data[court["code"]]) + yield data, catalog + + +def test_initial_import_and_unchanged_refresh_do_not_rewrite_paths(bulk): + _, catalog = bulk + assert refresh_index("massachusetts") == 6 + index = current_index("massachusetts") + ids = list(index.paths.order_by("pk").values_list("pk", flat=True)) + assert search_paths(index, "eviction")["total"] == 2 + stamp = index.refreshed_at + catalog.reset_mock() + assert refresh_index("massachusetts") == 0 + assert catalog.manifest.call_count == 1 + catalog.court.assert_not_called() + index.refresh_from_db() + assert index.refreshed_at > stamp + assert list(index.paths.order_by("pk").values_list("pk", flat=True)) == ids + + +def test_changed_court_only_and_retired_court_cleanup(bulk): + data, catalog = bulk + refresh_index("massachusetts") + index = current_index("massachusetts") + ids = set(index.paths.filter(court__code="district").values_list("pk", flat=True)) + data["housing"]["court"]["revision"] = "housing changed" + data["housing"]["filing_types"][0]["name"] = "New complaint" + catalog.reset_mock() + assert refresh_index("massachusetts") == 3 + assert [call.args[0]["code"] for call in catalog.court.call_args_list] == ["housing"] + assert set(index.paths.filter(court__code="district").values_list("pk", flat=True)) == ids + assert index.paths.filter(filing_type__name="New complaint").exists() + del data["housing"] + assert refresh_index("massachusetts") == 0 + assert not index.paths.filter(court__code="housing").exists() + index.refresh_from_db() + assert set(index.court_snapshots) == {"district"} + assert set(index.paths.values_list("pk", flat=True)) == ids + + +def test_failed_download_or_changed_manifest_preserves_entire_index(bulk): + data, catalog = bulk + refresh_index("massachusetts") + index = current_index("massachusetts") + ids = set(index.paths.values_list("pk", flat=True)) + data["housing"]["court"]["revision"] = "update" + catalog.court.side_effect = requests.Timeout("unavailable") + with pytest.raises(requests.Timeout): + refresh_index("massachusetts") + assert set(index.paths.values_list("pk", flat=True)) == ids + catalog.court.side_effect = lambda court: deepcopy(data[court["code"]]) + manifest = {code: item["court"] for code, item in data.items()} + catalog.manifest.side_effect = [manifest, {"housing": data["housing"]["court"]}] + with pytest.raises(ValueError, match="changed during synchronization"): + refresh_index("massachusetts") + assert set(index.paths.values_list("pk", flat=True)) == ids + # A removal-only refresh must also reject a manifest that changes mid-import. + catalog.reset_mock() + catalog.manifest.side_effect = [{"district": manifest["district"]}, manifest] + with pytest.raises(ValueError, match="changed during synchronization"): + refresh_index("massachusetts") + catalog.court.assert_not_called() + assert set(index.paths.values_list("pk", flat=True)) == ids + + +def test_database_failure_rolls_back_changes_and_deletions(bulk): + data, _ = bulk + refresh_index("massachusetts") + index = current_index("massachusetts") + ids = set(index.paths.values_list("pk", flat=True)) + del data["district"] + data["housing"]["court"]["revision"] = "changed" + data["housing"]["filing_types"][0]["name"] = "Changed" + with patch("efile.services.filing_code_sync._insert_paths", side_effect=ValueError("insert failed")): + with pytest.raises(ValueError, match="insert failed"): + refresh_index("massachusetts") + assert set(index.paths.values_list("pk", flat=True)) == ids + index.refresh_from_db() + assert set(index.court_snapshots) == {"housing", "district"} + + +def test_specific_filing_lists_override_generic_lists_and_timing_is_preserved(): + data = export() + entries = list(export_entries(data, "massachusetts")) + assert {(entry["case_type"]["code"], entry["filing_type"]["code"], entry["initial"]) for entry in entries} == { + ("sp", "complaint", True), + ("sp", "motion", False), + ("debt", "answer", False), + } + data["filing_types"].append( + { + "code": "category", + "name": "Category filing", + "case_category": "civil", + "case_type": "", + "timing": "Both", + } + ) + entries = list(export_entries(data, "massachusetts")) + assert not any(entry["filing_type"]["code"] in ("complaint", "answer") for entry in entries) + assert len(entries) == 4 + + +def test_subsequent_category_gate_matches_proxy_sql(): + data = export() + data["case_types"] = [data["case_types"][0]] + entries = list(export_entries(data, "massachusetts")) + assert len(entries) == 1 + assert entries[0]["initial"] is True + + +def test_forced_court_redownloads_without_a_revision_change(bulk): + _, catalog = bulk + refresh_index("massachusetts") + catalog.reset_mock() + assert refresh_index("massachusetts", force=["housing"]) == 3 + assert [call.args[0]["code"] for call in catalog.court.call_args_list] == ["housing"] + with pytest.raises(ValueError, match="nowhere"): + refresh_index("massachusetts", force=["nowhere"]) + + +def test_dry_run_reports_changes_without_saving(bulk): + data, _ = bulk + refresh_index("massachusetts") + index = current_index("massachusetts") + ids, snapshots = set(index.paths.values_list("pk", flat=True)), index.court_snapshots + data["housing"]["court"]["revision"] = "changed" + reports = [] + assert refresh_index("massachusetts", dry_run=True, progress=reports.append) == 3 + assert "Boston Housing Court (housing): 3 -> 3 paths" in reports + index.refresh_from_db() + assert index.court_snapshots == snapshots + assert set(index.paths.values_list("pk", flat=True)) == ids + + +def test_court_emptied_by_a_partial_proxy_load_is_rejected_unless_forced(bulk): + data, _ = bulk + refresh_index("massachusetts") + index = current_index("massachusetts") + ids = set(index.paths.values_list("pk", flat=True)) + data["housing"]["court"]["revision"] = "half loaded" + data["housing"]["filing_types"] = [] + with pytest.raises(ValueError, match="no filing paths"): + refresh_index("massachusetts") + assert set(index.paths.values_list("pk", flat=True)) == ids + assert refresh_index("massachusetts", force=["housing"]) == 0 + assert not index.paths.filter(court__code="housing").exists() diff --git a/efile_app/efile/tests/test_sqlite_transactions.py b/efile_app/efile/tests/test_sqlite_transactions.py new file mode 100644 index 00000000..350814ca --- /dev/null +++ b/efile_app/efile/tests/test_sqlite_transactions.py @@ -0,0 +1,82 @@ +"""File-backed coverage for overlapping local upload/worker transactions.""" + +import sqlite3 +from copy import deepcopy +from unittest.mock import Mock, patch + +import pytest +from django.conf import settings +from django.core.management import call_command +from django.db import OperationalError +from django.db.utils import ConnectionHandler + + +@pytest.mark.django_db(transaction=True) +def test_upload_can_write_after_a_worker_attempts_to_write(tmp_path): + config = deepcopy(settings.DATABASES["default"]) + if config["ENGINE"] != "django.db.backends.sqlite3": + pytest.skip("SQLite-specific local configuration") + config["NAME"] = tmp_path / "concurrent-upload.sqlite3" + config.setdefault("OPTIONS", {})["timeout"] = 0.05 + upload = ConnectionHandler({"default": config})["default"] + worker = sqlite3.connect(config["NAME"], timeout=0.05, isolation_level=None) + try: + with upload.cursor() as cursor: + cursor.execute("CREATE TABLE activity (kind TEXT)") + # Start as Django's outer atomic block does, then read before the + # document is sent to storage. Another process attempts a write. + upload.set_autocommit(False, force_begin_transaction_with_broken_autocommit=True) + with upload.cursor() as cursor: + cursor.execute("SELECT * FROM activity") + cursor.fetchall() + worker.execute("BEGIN") + worker_waiting = False + try: + worker.execute("INSERT INTO activity VALUES ('worker')") + except sqlite3.OperationalError as error: + assert "locked" in str(error) + worker_waiting = True + worker.rollback() + # DEFERRED lets the worker reserve the writer above, then raises + # "database is locked" here instead of waiting, regardless of timeout. + with upload.cursor() as cursor: + cursor.execute("INSERT INTO activity VALUES ('upload')") + upload.commit() + if worker_waiting: + worker.execute("INSERT INTO activity VALUES ('worker')") + worker.commit() + assert worker.execute("SELECT kind FROM activity ORDER BY kind").fetchall() == [("upload",), ("worker",)] + finally: + worker.rollback() + worker.close() + upload.close() + + +def test_extraction_supervisor_retries_sqlite_contention(): + command = "efile.management.commands.process_document_extractions" + with ( + patch(f"{command}.connection", vendor="sqlite"), + patch(f"{command}.close_old_connections"), + patch(f"{command}.call_command"), + patch(f"{command}.time.sleep", side_effect=[None, KeyboardInterrupt]), + patch( + "efile.services.document_extractions.claim_next_extraction", + side_effect=[OperationalError("database is locked"), None], + ) as claim, + ): + with pytest.raises(KeyboardInterrupt): + call_command("process_document_extractions", stdout=Mock()) + assert claim.call_count == 2 + + +@pytest.mark.parametrize("once,message", [(True, "database is locked"), (False, "no such table: missing")]) +def test_extraction_supervisor_does_not_hide_other_database_failures(once, message): + command = "efile.management.commands.process_document_extractions" + with ( + patch(f"{command}.connection", vendor="sqlite"), + patch(f"{command}.close_old_connections"), + patch(f"{command}.call_command"), + patch("efile.services.document_extractions.claim_next_extraction", side_effect=OperationalError(message)), + ): + with pytest.raises(OperationalError, match=message): + call_command("process_document_extractions", once=once, stdout=Mock()) diff --git a/efile_app/efile/views/extraction_review.py b/efile_app/efile/views/extraction_review.py index 28533997..7be71296 100644 --- a/efile_app/efile/views/extraction_review.py +++ b/efile_app/efile/views/extraction_review.py @@ -1,3 +1,5 @@ +import re + from django.contrib import messages from django.shortcuts import redirect, render from django.views.decorators.http import require_http_methods @@ -5,6 +7,7 @@ from efile.api.suffolk_api_views import get_tyler_token from efile.models import DocumentExtraction, FilingDocument from efile.party_sides import PARTY_SIDE_HELP, PARTY_SIDE_LABELS, PartySide +from efile.services.account_profile import cached_account_profile from efile.services.current_drafts import ensure_current_draft from efile.services.document_checklists import resolve_filer_roles from efile.services.document_extractions import extraction_for_document @@ -13,7 +16,12 @@ from efile.services.extracted_parties import review_rows, save_reviewed_parties from efile.services.extraction_fields import display_extracted_fields, document_summary_details from efile.services.filing_availability import filing_unavailable_message -from efile.services.filing_path import change_filing_path, describe_path_change, filing_path_conflict +from efile.services.filing_path import ( + change_filing_path, + clear_changed_classification, + describe_path_change, + filing_path_conflict, +) from efile.workflow import ( ExistingCase, WorkflowStepKey, @@ -82,9 +90,47 @@ def _set_lead_filing_type(draft, filing_type_code, filing_type_name): return if lead.filing_type_code != filing_type_code: lead.document_type_confirmed = False + lead.document_type_code = "" + lead.document_type_name = "" + lead.filing_component_code = "" + lead.filing_component_name = "" + lead.requested_optional_services = [] + lead.filing_requires_amount_in_controversy = False lead.filing_type_code = filing_type_code lead.filing_type_name = filing_type_name - lead.save(update_fields=["filing_type_code", "filing_type_name", "document_type_confirmed", "updated_at"]) + lead.save( + update_fields=[ + "filing_type_code", + "filing_type_name", + "document_type_confirmed", + "document_type_code", + "document_type_name", + "filing_component_code", + "filing_component_name", + "requested_optional_services", + "filing_requires_amount_in_controversy", + "updated_at", + ] + ) + + +def _zip_shortcuts(request, draft, jurisdiction): + """ZIP codes code search offers one click away: the filer's own, then recent ones. + + The account profile is the same cached fetch "Your information" makes, so + asking for it here moves the call earlier rather than adding one. + """ + filer = draft.parties.filter(role="filer").first() + own = filer.zip_code if filer else "" + if not own and request.method == "GET": + own = (cached_account_profile(request, jurisdiction) or {}).get("zip") or "" + shortcuts = [] + if re.fullmatch(r"[0-9]{5}(?:-[0-9]{4})?", own): + shortcuts.append({"zip": own[:5], "kind": "address"}) + for postal_code in request.user.recent_case_zips or []: + if all(item["zip"] != postal_code for item in shortcuts): + shortcuts.append({"zip": postal_code, "kind": "recent"}) + return shortcuts def _case_identity(existing_case, docket_number): @@ -190,6 +236,8 @@ def extraction_review(request, jurisdiction): path_change = change_filing_path(draft, existing_case) if path_change.changed and path_change.cleared: messages.info(request, describe_path_change(path_change)) + if clear_changed_classification(draft, court_code, case_category_code, case_type_code): + messages.info(request, "Filing path changed. Check each document's filing options and fees again.") write_case_data( draft, { @@ -261,6 +309,7 @@ def classified(level, key): # Only some case types have sides. The screen asks for one as soon as # the chosen case type turns out to be one of them. "filer_role": draft.filer_role, + "zip_shortcuts": _zip_shortcuts(request, draft, jurisdiction), } suggested_existing_case = ( "new" diff --git a/efile_app/pyproject.toml b/efile_app/pyproject.toml index 54213fa1..b935a359 100644 --- a/efile_app/pyproject.toml +++ b/efile_app/pyproject.toml @@ -30,6 +30,7 @@ dependencies = [ # Vermont Superior Court unit lookup by town, county, or ZIP. "suffolklitlab-vtcourts @ git+https://github.com/SuffolkLITLab/VTCourts@main", "django-otp>=1.7.0,<2", + "snowballstemmer>=3,<4", ] [build-system] @@ -42,7 +43,7 @@ exclude = [ ] [tool.setuptools.package-data] -efile = ["prompts/*.yaml", "data/*.json"] +efile = ["prompts/*.yaml", "data/*.json", "data/*.yaml"] [tool.uv] @@ -107,7 +108,9 @@ testpaths = [ # new: #addopts = "--reuse-db --nomigrations" # merged: -addopts = "-ra --strict-markers --strict-config --reuse-db --nomigrations" +# Run schema migrations: SQLite's full-text index and synchronization triggers +# are migration-managed database objects, not ordinary Django model tables. +addopts = "-ra --strict-markers --strict-config --reuse-db" markers = [ "slow: marks tests as slow (deselect with '-m \"not slow\"')", "integration: marks tests as integration tests", diff --git a/efile_app/pytest.ini b/efile_app/pytest.ini index 7986eb97..ad010d77 100644 --- a/efile_app/pytest.ini +++ b/efile_app/pytest.ini @@ -1,7 +1,7 @@ [pytest] DJANGO_SETTINGS_MODULE = efile.settings python_files = tests.py test_*.py *_tests.py -addopts = --reuse-db --nomigrations +addopts = --reuse-db testpaths = efile markers = diff --git a/efile_app/scripts/build_zip_counties.py b/efile_app/scripts/build_zip_counties.py new file mode 100644 index 00000000..2cfbb3f6 --- /dev/null +++ b/efile_app/scripts/build_zip_counties.py @@ -0,0 +1,68 @@ +"""Build efile/data/zip_counties.json.gz from Census 2020 relationship files. + +Code search narrows courts by the county a case ZIP is in. Most states name +courts after counties; some (Massachusetts district courts) name them after +towns, so the table also maps each state's towns to their counties. + + curl -O https://www2.census.gov/geo/docs/maps-data/data/rel2020/zcta520/tab20_zcta520_county20_natl.txt + curl -O https://www2.census.gov/geo/docs/maps-data/data/gazetteer/2020_Gazetteer/2020_Gaz_cousubs_national.zip + unzip 2020_Gaz_cousubs_national.zip + uv run python scripts/build_zip_counties.py tab20_zcta520_county20_natl.txt 2020_Gaz_cousubs_national.txt + +ZCTAs approximate ZIP codes; PO-box-only ZIPs have no ZCTA and stay unmatched. +""" + +import csv +import gzip +import json +import re +import sys +from collections import defaultdict +from pathlib import Path + +OUTPUT = Path(__file__).resolve().parent.parent / "efile" / "data" / "zip_counties.json.gz" +COUNTY_SUFFIX = re.compile( + r"\s+(County|Parish|Borough|City and Borough|Census Area|Municipality|Planning Region|city)$" +) +TOWN_SUFFIX = re.compile( + r"\s+(city|town|township|village|borough|plantation|CCD|gore|grant|location|purchase|" + r"unorganized territory|UT|charter township|municipality|district|precinct)$", + re.IGNORECASE, +) + + +def main(zcta_path, cousub_path): + states = {} + towns = defaultdict(lambda: defaultdict(set)) + with open(cousub_path, encoding="utf-8-sig") as handle: + for row in csv.DictReader(handle, delimiter="\t"): + row = {key.strip(): value.strip() for key, value in row.items()} + states[row["GEOID"][:2]] = row["USPS"] + towns[row["USPS"]][TOWN_SUFFIX.sub("", row["NAME"]).casefold()].add(row["GEOID"][:5]) + county_names = {} + zips = defaultdict(lambda: defaultdict(list)) + with open(zcta_path, encoding="utf-8-sig") as handle: + for row in csv.DictReader(handle, delimiter="|"): + county = row["GEOID_COUNTY_20"] + county_names[county] = COUNTY_SUFFIX.sub("", row["NAMELSAD_COUNTY_20"]) + # Territories without county subdivisions in the gazetteer are skipped. + if row["GEOID_ZCTA5_20"] and county[:2] in states: + # Largest land overlap first, so a split ZIP leads with its main county. + zips[states[county[:2]]][row["GEOID_ZCTA5_20"]].append((-int(row["AREALAND_PART"]), county)) + table = {} + for state in sorted(set(zips) | set(towns)): + table[state] = { + "zips": {code: [county_names[c] for _, c in sorted(parts)] for code, parts in sorted(zips[state].items())}, + "towns": { + name: sorted(county_names[c] for c in counties if c in county_names) + for name, counties in sorted(towns[state].items()) + if name and any(c in county_names for c in counties) + }, + } + with gzip.open(OUTPUT, "wt", encoding="utf-8") as handle: + json.dump(table, handle, separators=(",", ":"), sort_keys=True) + print(f"Wrote {OUTPUT} ({OUTPUT.stat().st_size:,} bytes, {sum(len(s['zips']) for s in table.values()):,} ZIPs)") + + +if __name__ == "__main__": + main(*sys.argv[1:3]) diff --git a/efile_app/tests/confirm-case-editing.spec.js b/efile_app/tests/confirm-case-editing.spec.js index dab1bd7f..0dc3efae 100644 --- a/efile_app/tests/confirm-case-editing.spec.js +++ b/efile_app/tests/confirm-case-editing.spec.js @@ -15,6 +15,24 @@ const { expect } = require('@playwright/test'); +const { + default: AxeBuilder +} = require('@axe-core/playwright'); +const fs = require('node:fs'); + +async function auditCodeSearch(page, state) { + const accessibility = await new AxeBuilder({ + page + }).include('#code-search-dialog').analyze(); + const reportPath = test.info().outputPath(`filing-search-${state}.json`); + fs.writeFileSync(reportPath, JSON.stringify(accessibility, null, 2)); + await test.info().attach(`filing-search-${state}`, { + path: reportPath, + contentType: 'application/json' + }); + expect(accessibility.violations).toEqual([]); +} + const PAGE = '/jurisdiction/illinois/extraction-review/'; // The seeded draft is filed in "cook:cvd1", Small Claims (6198), Contract @@ -248,6 +266,1176 @@ test.use({ }); test.skip(!process.env.A11Y_STORAGE_STATE, 'Needs the seeded session: run seed_accessibility_session and set A11Y_STORAGE_STATE.'); +test.describe('filing code search', () => { + const result = { + id: 101, + initial: true, + court: { + code: 'vt:orange', + name: 'Orange Unit' + }, + case_category: { + code: '7000', + name: 'Civil' + }, + case_type: { + code: '7100', + name: 'Landlord tenant' + }, + filing_type: { + code: '143132', + name: 'Complaint' + }, + explanation: '', + explanation_source: '', + unavailable: '', + }; + const options = { + court: Object.entries(UNITS).map(([code, name]) => ({ + code, + name + })), + case_category: [{ + code: '7000', + name: 'Civil' + }], + case_type: [{ + code: '7100', + name: 'Landlord tenant' + }], + filing_type: [{ + code: '143132', + name: 'Complaint' + }], + }; + + async function mockSearch(page, { + reject = false, + unavailable = false + } = {}) { + const queries = []; + await page.route('**/api/filing-code-search/**', async route => { + const params = new URL(route.request().url()).searchParams; + queries.push(Object.fromEntries(params)); + if (params.has('path_id')) { + return route.fulfill(reject ? { + status: 409, + json: { + error: 'The code list has changed. Search again.' + }, + } : { + json: { + path: { + ...result, + options + } + } + }); + } + return route.fulfill({ + json: { + results: [{ + ...result, + unavailable: unavailable ? 'Temporarily unavailable' : '' + }], + total: 1, + corrected_terms: [], + stale: false, + } + }); + }); + return queries; + } + + const searchbox = page => page.getByRole('searchbox', { + name: "What you're filing" + }); + const button = (page, name) => page.getByRole('dialog').getByRole('button', { + name, + exact: true + }); + const toCourts = page => button(page, 'Next: court and case type').click(); + const toCheck = page => button(page, 'Next: check and use').click(); + const useCode = page => button(page, 'Use this filing code').click(); + + async function openSearch(page) { + await page.getByRole('button', { + name: 'Find the right filing code' + }).click(); + return page.getByRole('dialog'); + } + + function courtRoutes(page, { + groups, + courts = [{ + code: 'vt:orange', + name: 'Orange Unit' + }], + paths = params => [{ + ...result, + court: { + code: params.get('court'), + name: UNITS[params.get('court')] + } + }], + extra = {}, + queries = [], + }) { + return page.route('**/api/filing-code-search/**', route => { + const params = new URL(route.request().url()).searchParams; + queries.push({ + ...Object.fromEntries(params), + choices: JSON.parse(params.get('case_filters') || '{}') + }); + if (params.has('path_id')) return route.fulfill({ + json: { + path: { + ...result, + initial: params.get('existing_case') !== 'yes', + options + } + } + }); + if (params.has('court')) { + const results = paths(params); + return route.fulfill({ + json: { + results, + total: results.length, + corrected_terms: [] + } + }); + } + if (params.has('group')) return route.fulfill({ + json: { + courts, + corrected_terms: [] + } + }); + const shown = typeof groups === 'function' ? groups(params) : groups; + return route.fulfill({ + json: { + groups: shown, + total: shown.length, + corrected_terms: [], + stale: false, + ...(typeof extra === 'function' ? extra(params) : extra), + } + }); + }); + } + + test('empty search offers common types, clears back to them, and accepts one character', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const queries = await mockSearch(page); + await openSavedDraft(page); + const dialog = await openSearch(page); + await expect(dialog.getByRole('heading', { + name: 'Common filing types' + })).toBeVisible(); + await expect(dialog).toContainText('Step 1 of 3'); + expect(queries).toEqual([]); + await button(page, 'Complaint').click(); + await expect(dialog.getByRole('radio', { + name: 'Complaint' + })).toBeVisible(); + expect(queries[0].q).toBe('complaint'); + await dialog.getByRole('radio').check(); + await expect(button(page, 'Next: check and use')).toBeEnabled(); + await searchbox(page).fill(''); + await expect(dialog.getByRole('heading', { + name: 'Common filing types' + })).toBeVisible(); + await expect(dialog.getByRole('radio')).toHaveCount(0); + await expect(button(page, 'Next: court and case type')).toBeDisabled(); + await searchbox(page).fill('m'); + await expect(dialog.getByRole('radio')).toBeVisible(); + expect(queries.at(-1).q).toBe('m'); + }); + + test('an HTML server error gives a readable message and search can be retried', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await mockSearch(page); + await page.route('**/api/filing-code-search/**', route => route.fulfill({ + status: 503, + contentType: 'text/html', + body: 'Unavailable', + }), { + times: 1 + }); + await openSavedDraft(page); + await openSearch(page); + await button(page, 'Complaint').click(); + await expect(page.locator('#code-search-status')).toContainText('Code search could not be loaded'); + await searchbox(page).press('Enter'); + await expect(page.getByRole('dialog').getByRole('radio')).toBeVisible(); + }); + + test('search becomes usable when the first index finishes without reopening the dialog', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await mockSearch(page); + await page.clock.install(); + await page.route('**/api/filing-code-search/**', route => route.fulfill({ + status: 503, + json: { + error: 'Code search is not available yet.' + }, + }), { + times: 1 + }); + await openSavedDraft(page); + await openSearch(page); + await button(page, 'Complaint').click(); + await expect(page.locator('#code-search-status')).toContainText('We will try again shortly.'); + await page.clock.fastForward(10000); + await expect(page.getByRole('dialog').getByRole('radio')).toBeVisible(); + }); + + for (const flat of [true, false]) { + test(`a full path result skips to the check step and fills all four fields (${flat ? 'flat' : 'guided'})`, async ({ + page + }) => { + await mockCourtLists(page, { + flat + }); + const queries = await mockSearch(page); + await openSavedDraft(page); + const open = page.getByRole('button', { + name: 'Find the right filing code' + }); + const dialog = await openSearch(page); + await expect(searchbox(page)).toBeFocused(); + await searchbox(page).fill('eviction'); + await dialog.getByRole('radio', { + name: 'Complaint' + }).check(); + await toCheck(page); + await expect(dialog.getByRole('heading', { + name: 'Check your choices' + })).toBeFocused(); + await expect(dialog.locator('#code-search-confirm')).toContainText('Landlord tenant'); + await expect(dialog.locator('#code-search-confirm')).toContainText('Orange Unit'); + await auditCodeSearch(page, 'check-step'); + await useCode(page); + await expect(dialog).not.toBeVisible(); + await expect(open).toBeFocused(); + await expect(page.locator('#court_code')).toHaveValue('vt:orange'); + await expect(field(page, 'case_category').value).toHaveText('Civil'); + await expect(field(page, 'case_type').value).toHaveText('Landlord tenant'); + await expect(field(page, 'filing_type').value).toHaveText('Complaint'); + if (!flat) await expect(page.locator('.court-selector__result')).toContainText('Orange Unit'); + expect(queries[0]).toMatchObject({ + jurisdiction: 'illinois', + existing_case: 'no', + q: 'eviction', + offset: '0' + }); + expect(queries[0]).not.toHaveProperty('court'); + expect(queries[0]).not.toHaveProperty('purpose'); + expect(queries[1].path_id).toBe('101'); + }); + } + + for (const hasCourt of [false, true]) { + test(`the court step offers court cards with ${hasCourt ? 'the current court first' : 'no court chosen'}`, async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await courtRoutes(page, { + groups: [{ + key: 'eviction', + name: 'Eviction Complaint', + variants: ['Eviction Complaint', 'Summary Process Complaint'], + path_count: 100 + }], + courts: [{ + code: 'vt:orange', + name: 'Orange Unit' + }, { + code: 'cook:cvd1', + name: 'Chittenden Unit' + }], + }); + await openSavedDraft(page); + if (!hasCourt) await page.locator('#court_code').evaluate(input => input.value = ''); + const dialog = await openSearch(page); + await searchbox(page).fill('eviction'); + await expect(dialog.locator('.code-search__group')).toHaveCount(1); + await expect(dialog).toContainText('Also listed as: Summary Process Complaint'); + await expect(button(page, 'Next: court and case type')).toBeDisabled(); + await dialog.getByRole('radio', { + name: 'Eviction Complaint' + }).check(); + await toCourts(page); + await expect(dialog).toContainText('Step 2 of 3'); + await expect(dialog.locator('#code-search-summary')).toContainText('Eviction Complaint'); + const courts = dialog.getByRole('group', { + name: 'Which court will hear the case?' + }).getByRole('radio'); + await expect(courts).toHaveCount(2); + await expect(courts.first()).toHaveAccessibleName(hasCourt ? 'Chittenden Unit' : 'Orange Unit'); + const caseTypes = dialog.locator('input[name="code-search-result"]'); + if (hasCourt) { + await expect(courts.first()).toBeChecked(); + await expect(caseTypes).toHaveCount(1); + await caseTypes.check(); + await expect(button(page, 'Next: check and use')).toBeEnabled(); + } else { + await expect(courts.first()).not.toBeChecked(); + await expect(caseTypes).toHaveCount(0); + } + await dialog.getByRole('radio', { + name: 'Orange Unit' + }).check(); + await expect(button(page, 'Next: check and use')).toBeDisabled(); + await expect(caseTypes).toHaveCount(1); + await caseTypes.check(); + await auditCodeSearch(page, `court-cards-${hasCourt ? 'current' : 'none'}`); + await toCheck(page); + await button(page, 'Back').click(); + await expect(dialog.getByRole('radio', { + name: 'Orange Unit' + })).toBeChecked(); + await toCheck(page); + await useCode(page); + await expect(dialog).not.toBeVisible(); + await expect(page.locator('#court_code')).toHaveValue('vt:orange'); + await expect(page.locator('#case_type_code')).toHaveValue('7100'); + }); + } + + test('many courts fall back to a dropdown and the summary returns to step one', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await courtRoutes(page, { + groups: [{ + key: 'petition', + name: 'Petition', + variants: ['Petition'] + }], + courts: Object.entries(UNITS).map(([code, name]) => ({ + code, + name + })), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('petition'); + await dialog.getByRole('radio', { + name: 'Petition' + }).check(); + await toCourts(page); + const court = dialog.getByLabel('Which court will hear the case?'); + await expect(court).toHaveValue('cook:cvd1'); + await court.selectOption('vt:addison'); + await expect(dialog.locator('input[name="code-search-result"]')).toHaveCount(1); + await button(page, 'Change what you\'re filing').click(); + await expect(dialog).toContainText('Step 1 of 3'); + await expect(dialog.getByRole('radio', { + name: 'Petition' + })).toBeChecked(); + await toCourts(page); + await expect(court).toHaveValue('vt:addison'); + }); + + test('synonym results explain their concepts and show local meanings after court selection', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const meaning = { + key: 'eviction', + label: 'Eviction', + text: 'Statewide eviction meaning.', + scope: 'state', + scope_name: 'Vermont', + source: 'https://www.vtcourts.gov/civil/eviction-process', + }; + const reasonFor = params => params.get('q') === 'eviction' ? null : { + kind: 'related_concept', + query: params.get('q'), + concepts: ['eviction'], + corrected_terms: [], + }; + await courtRoutes(page, { + groups: params => [{ + key: 'eviction', + name: 'Eviction complaint', + variants: ['Eviction complaint'], + concepts: [meaning], + match_reason: reasonFor(params) + }], + courts: [{ + code: 'vt:orange', + name: 'Orange Unit' + }, { + code: 'cook:cvd1', + name: 'Chittenden Unit' + }], + paths: params => { + const county = params.get('court') === 'vt:orange' ? 'Orange' : 'Chittenden'; + return [{ + ...result, + case_description: `${county} local meaning.` + }]; + }, + }); + await openSavedDraft(page); + await page.locator('#court_code').evaluate(input => input.value = ''); + const dialog = await openSearch(page); + await searchbox(page).fill('unlawful detainer'); + const group = page.locator('.code-search__group'); + await expect(group.locator('.code-search__meta')).toHaveText('Related to eviction'); + const about = group.locator('details'); + await expect(about.locator('p').first()).not.toBeVisible(); + await about.locator('summary').focus(); + await page.keyboard.press('Enter'); + await expect(about).toContainText('Meaning in Vermont: Statewide eviction meaning.'); + await expect(group.getByRole('radio')).not.toBeChecked(); + await expect(group.getByRole('link', { + name: 'About this concept' + })).toHaveAttribute('href', meaning.source); + await auditCodeSearch(page, 'synonym-explanation'); + await group.getByRole('radio').check(); + await toCourts(page); + await dialog.getByRole('radio', { + name: 'Orange Unit' + }).check(); + await expect(dialog).toContainText('Orange local meaning.'); + await dialog.getByRole('radio', { + name: 'Chittenden Unit' + }).check(); + await expect(dialog).toContainText('Chittenden local meaning.'); + await expect(dialog).not.toContainText('Orange local meaning.'); + await button(page, 'Back').click(); + await searchbox(page).fill('eviction'); + await expect(group.locator('.code-search__meta')).toHaveCount(0); + await expect(group).toContainText('Statewide eviction meaning.'); + }); + + test('case questions are chips that reset when the case area changes', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const queries = []; + await courtRoutes(page, { + queries, + groups: [{ + key: 'answer', + name: 'Answer', + variants: ['Answer'] + }], + extra: params => { + const topic = params.get('case_topic') || (params.get('q') === 'eviction' ? 'eviction' : 'small_claims'); + return { + case_topic: topic, + case_topics: [{ + value: 'eviction', + label: 'Eviction' + }, { + value: 'small_claims', + label: 'Small claims' + }], + case_facets: topic === 'eviction' ? [{ + key: 'role', + label: 'Your role', + options: [{ + value: 'tenant', + label: 'Tenant / occupant' + }, { + value: 'landlord', + label: 'Landlord / owner' + }] + }, { + key: 'property', + label: 'Property', + options: [{ + value: 'residential', + label: 'Home' + }, { + value: 'commercial', + label: 'Business' + }] + }] : [{ + key: 'amount', + label: 'Claim amount', + options: [{ + value: '0:250000', + label: 'Up to $2,500' + }, { + value: '250001:1000000', + label: '$2,500.01–$10,000' + }], + help: 'State-specific limits and exceptions.', + source: 'https://www.illinoiscourts.gov/' + }], + }; + }, + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('eviction'); + const facets = page.locator('#code-search-case-filters'); + await expect(facets.getByRole('group', { + name: 'Your role' + })).toBeVisible(); + await expect(facets.getByRole('radio')).toHaveCount(0); + await expect(facets.getByRole('heading')).toHaveCount(0); + const chip = name => facets.getByRole('button', { + name, + exact: true + }); + await chip('Tenant / occupant').click(); + await expect.poll(() => queries.at(-1)?.choices.role).toBe('tenant'); + await expect(chip('Tenant / occupant')).toHaveAttribute('aria-pressed', 'true'); + await chip('Home').click(); + await expect.poll(() => queries.at(-1)?.choices.property).toBe('residential'); + await chip('Business').focus(); + await page.keyboard.press('Enter'); + await expect.poll(() => queries.at(-1)?.choices.property).toBe('commercial'); + await expect(chip('Business')).toBeFocused(); + await expect(chip('Business')).toHaveAttribute('aria-pressed', 'true'); + await expect(chip('Home')).toHaveAttribute('aria-pressed', 'false'); + await auditCodeSearch(page, 'case-chips-desktop'); + await page.setViewportSize({ + width: 390, + height: 844 + }); + expect(await dialog.evaluate(el => el.scrollWidth <= el.clientWidth)).toBe(true); + await auditCodeSearch(page, 'case-chips-mobile'); + await expect(chip('Eviction')).toHaveAttribute('aria-pressed', 'true'); + await facets.getByRole('group', { + name: 'Kind of case' + }).getByRole('button', { + name: 'Small claims' + }).click(); + await expect(chip('Up to $2,500')).toBeVisible(); + expect(queries.at(-1).choices).toEqual({}); + await chip('$2,500.01–$10,000').click(); + await expect.poll(() => queries.at(-1)?.choices.amount).toBe('250001:1000000'); + await expect(chip('$2,500.01–$10,000')).toHaveAttribute('aria-pressed', 'true'); + const amountHelp = facets.getByText('About claim amounts', { + exact: true + }); + await amountHelp.focus(); + await page.keyboard.press('Enter'); + await expect(amountHelp).toBeFocused(); + await expect(facets.getByText('State-specific limits and exceptions.')).toBeVisible(); + await auditCodeSearch(page, 'claim-amount-help-mobile'); + await page.setViewportSize({ + width: 1366, + height: 768 + }); + await dialog.getByRole('radio', { + name: 'Answer' + }).check(); + await toCourts(page); + await expect(dialog.locator('#code-search-summary')).toContainText('Small claims · $2,500.01–$10,000'); + await button(page, 'Back').click(); + await searchbox(page).fill('small claims'); + await expect.poll(() => queries.at(-1)?.q).toBe('small claims'); + expect(queries.at(-1).choices).toEqual({}); + }); + + test('case types are the choice in step two, each with its own description', async ({ + page + }) => { + await page.setViewportSize({ + width: 1366, + height: 768 + }); + await mockCourtLists(page, { + flat: true + }); + const filingName = 'Answer/Response to Complaint/Petition'; + const caseTypes = ['Commercial', 'Commercial - Possession Only', 'Ejectment', 'Residential - Eviction', 'Residential - Eviction Possession Only']; + await courtRoutes(page, { + groups: [{ + key: 'answer', + name: filingName, + variants: [filingName] + }], + paths: () => caseTypes.map((name, i) => ({ + ...result, + id: i + 100, + filing_type: { + ...result.filing_type, + name: i === 2 ? 'Answer - Ejectment' : filingName + }, + case_type: { + code: `case-${i}`, + name + }, + case_context: `Eviction › ${name}`, + case_description: 'An existing plain-language case description.', + })), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('answer'); + await dialog.getByRole('radio', { + name: filingName + }).check(); + await toCourts(page); + await expect(dialog.getByRole('radio', { + name: 'Orange Unit' + })).toBeChecked(); + const choices = page.locator('.code-search__path-choice'); + await expect(choices).toHaveCount(5); + await expect(page.locator('#code-search-path-status')).toHaveText('5 case types fit'); + for (const name of caseTypes) await expect(page.getByRole('radio', { + name: `Eviction › ${name}`, + exact: true + })).toBeVisible(); + await expect(choices.nth(2)).toContainText('This court calls it “Answer - Ejectment”.'); + await expect(choices.nth(0)).not.toContainText('This court calls it'); + const residential = page.getByRole('radio', { + name: 'Eviction › Residential - Eviction', + exact: true + }); + await residential.focus(); + await page.keyboard.press('Space'); + await expect(residential).toBeChecked(); + await expect(residential).toHaveAccessibleDescription('An existing plain-language case description.'); + await expect(button(page, 'Next: check and use')).toBeEnabled(); + await auditCodeSearch(page, 'case-types-desktop'); + await page.setViewportSize({ + width: 390, + height: 844 + }); + expect(await page.getByRole('dialog').evaluate(el => el.scrollWidth <= el.clientWidth)).toBe(true); + await auditCodeSearch(page, 'case-types-mobile'); + }); + + test('case stage and document filters sit behind a short line and apply case stage only with a path', async ({ + page + }) => { + await page.setViewportSize({ + width: 1366, + height: 768 + }); + await mockCourtLists(page, { + flat: true + }); + const queries = []; + await courtRoutes(page, { + queries, + groups: Array.from({ + length: 6 + }, (_, i) => ({ + key: `type-${i}`, + name: `Complaint / Petition - Eviction - Residential - Possession Only - Fee ${i}`, + variants: ['Complaint', 'Petition'], + case_contexts: ['Small claims', 'Civil › Eviction'], + case_context_count: 2, + concepts: [{ + key: 'eviction', + label: 'Eviction', + text: 'A short explanation of this concept.', + scope: 'general' + }], + })), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('eviction'); + await expect(page.locator('.code-search__group')).toHaveCount(6); + await expect(page.locator('.code-search__group').first().locator('.code-search__meta')).toContainText('2 case types, including Small claims'); + await expect(dialog.getByLabel('Your task')).toHaveCount(0); + await expect(dialog.getByRole('button', { + name: 'Main document' + })).not.toBeVisible(); + const before = await page.locator('input[name="existing_case"]:checked').inputValue(); + await expect(page.locator('#code-search-stage-label')).toHaveText('New case'); + await button(page, 'Change case stage').click(); + await expect(page.locator('#code-search-stage-label')).toHaveText('Existing case'); + await expect.poll(() => queries.at(-1)?.existing_case).toBe('yes'); + const toggle = dialog.locator('#code-search-more-filters'); + await expect(toggle).toHaveText('More filters'); + await expect(toggle).toHaveAttribute('aria-expanded', 'false'); + await toggle.click(); + await expect(toggle).toHaveText('Fewer filters'); + await expect(toggle).toHaveAttribute('aria-expanded', 'true'); + await button(page, 'Main document').click(); + await expect.poll(() => queries.at(-1)?.document).toBe('main'); + await auditCodeSearch(page, 'more-filters-open'); + await toggle.click(); + await expect(toggle).toHaveText('More filters (1)'); + await expect(button(page, 'Main document')).toBeHidden(); + await expect(page.locator('input[name="existing_case"]:checked')).toHaveValue(before); + await dialog.getByRole('radio').first().check(); + await toCourts(page); + await expect(dialog.locator('#code-search-summary')).toContainText('Existing case'); + await dialog.locator('input[name="code-search-result"]').check(); + await toCheck(page); + await useCode(page); + await expect(dialog).not.toBeVisible(); + await expect(page.locator('input[name="existing_case"]:checked')).toHaveValue('existing'); + await expect(page.locator('#case_type_code')).toHaveValue('7100'); + }); + + test('ZIP narrows the search, offers the filer\'s ZIP as a shortcut, and clears back to every county', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const queries = []; + await courtRoutes(page, { + queries, + groups: [{ + key: 'petition', + name: 'Petition', + variants: ['Petition'] + }], + extra: params => ({ + location_counties: params.get('zip') ? ['Cook'] : [] + }), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + const zip = dialog.getByLabel('Case ZIP code', { + exact: true + }); + await expect(zip).toHaveValue(''); + await searchbox(page).fill('eviction'); + await expect(page.locator('.code-search__group')).toHaveCount(1); + const shortcut = button(page, 'Use your ZIP (60601)'); + await expect(shortcut).toBeVisible(); + await zip.fill('606'); + await expect(page.locator('#code-search-zip-status')).toHaveText('Enter all 5 digits.'); + expect(queries.at(-1).zip).toBe(''); + await shortcut.click(); + await expect(zip).toHaveValue('60601'); + await expect(zip).toBeFocused(); + await expect(page.locator('#code-search-zip-status')).toHaveText('60601 is in Cook County.'); + await expect(shortcut).toBeHidden(); + expect(queries.at(-1).zip).toBe('60601'); + await auditCodeSearch(page, 'zip-shortcut'); + await dialog.getByRole('radio', { + name: 'Petition' + }).check(); + await toCourts(page); + await expect(dialog.locator('#code-search-summary')).toContainText('60601 · Cook'); + await button(page, 'Back').click(); + await zip.fill(''); + await expect(page.locator('#code-search-zip-status')).toHaveText(''); + await expect.poll(() => queries.at(-1)?.zip).toBe(''); + await expect(shortcut).toBeVisible(); + }); + + test('many results ask what the filer wants to do, with counts, and show name matches first', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const queries = []; + const all = Array.from({ + length: 12 + }, (_, i) => ({ + key: `type-${i}`, + name: i < 3 ? `Small Claims Complaint ${i}` : `Affidavit ${i}`, + variants: [`Filing ${i}`], + })); + await courtRoutes(page, { + queries, + groups: params => { + const offset = Number(params.get('offset')); + const shown = params.get('strong') === 'true' ? all.slice(0, 3) : all; + return shown.slice(offset, offset + 20); + }, + extra: params => ({ + total: params.get('action') === 'start' ? 3 : 12, + strong_total: 3, + actions: [{ + value: 'start', + label: 'Start a case', + count: 3 + }, { + value: 'proof', + label: 'Supporting papers', + count: 9 + }], + case_topics: [], + case_facets: [], + }), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('small claims'); + await expect(dialog.locator('.code-search__group')).toHaveCount(3); + await expect(page.locator('#code-search-status')).toHaveText('3 filing types match “small claims”'); + expect(queries.at(-1).strong).toBe('true'); + const question = dialog.getByRole('group', { + name: 'What do you want to do?' + }); + await expect(question.getByRole('button', { + name: 'Supporting papers 9' + })).toBeVisible(); + await auditCodeSearch(page, 'action-question'); + await dialog.getByRole('button', { + name: 'Show 9 more filing types used in these case types' + }).click(); + await expect(dialog.locator('.code-search__group')).toHaveCount(12); + expect(queries.at(-1)).toMatchObject({ + strong: 'false', + offset: '3' + }); + await question.getByRole('button', { + name: 'Start a case 3' + }).click(); + await expect.poll(() => queries.at(-1)?.action).toBe('start'); + await expect(question).toBeVisible(); + // The card's padding, outside its label, still chooses it. + await dialog.locator('.code-search__group').first().click({ + position: { + x: 4, + y: 4 + } + }); + await expect(dialog.locator('.code-search__group').first().getByRole('radio')).toBeChecked(); + await toCourts(page); + await expect(dialog.locator('#code-search-summary')).toContainText('Start a case'); + }); + + test('the court step asks only about choices its case types differ on', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const over = { + value: '1000001:', + label: 'More than $10,000' + }; + const rows = [ + ['Civil - Amount Claimed Greater Than $10,000', 'Other Personal Injury Complaint - Jury', '(Jury - 12) (Govn\'t)', '12', 'lawyer', 'yes', over], + ['Civil - Amount Claimed Greater Than $10,000', 'Other Personal Injury Complaint - Jury', '(Jury - 12)', '12', 'lawyer', 'no', over], + ['Civil - Amount Claimed Greater Than $10,000', 'Other Personal Injury Complaint - Jury', '(Jury - 6)', '6', 'lawyer', 'no', over], + ['Civil', 'Other Personal Injury Complaint - Jury - Self-Represented Litigant', '(Jury - 12)', '12', 'self', 'no', null], + ['Civil', 'Other Personal Injury Complaint - Jury - Self-Represented Litigant', '(Jury - 6)', '6', 'self', 'no', null], + ['Civil - Amount Claimed Greater Than $10,000', 'Other Personal Injury Complaint - Non-Jury', '', 'none', 'lawyer', 'no', over], + ['Civil', 'Other Personal Injury Complaint - Non-Jury - Self-Represented Litigant', '', 'none', 'self', 'no', null], + ['Civil', 'Personal Injury Complaint - Jury', '(Jury - 12) (Govn\'t)', '12', 'lawyer', 'yes', null], + ]; + const filingName = 'Complaint / Petition - Personal Injury'; + await courtRoutes(page, { + groups: [{ + key: 'pi', + name: filingName, + variants: [filingName] + }], + courts: [{ + code: 'vt:orange', + name: 'Orange Unit' + }], + paths: () => rows.map(([category, caseType, qualifier, jury, representation, government, amount], i) => ({ + ...result, + id: 300 + i, + case_context: `${category} › ${caseType}`, + filing_type: { + code: `pi-${i}`, + name: `${filingName} ${qualifier} - Fee` + }, + filing_label: filingName, + qualifiers: { + jury, + representation, + government, + amount + }, + })), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('personal injury'); + await dialog.getByRole('radio', { + name: filingName + }).check(); + await toCourts(page); + const questions = page.locator('#code-search-path-questions'); + const status = page.locator('#code-search-path-status'); + const government = questions.getByRole('group', { + name: 'Filing for a government body?' + }); + await expect(government.getByRole('button', { + name: 'No', + exact: true + })).toHaveAttribute('aria-pressed', 'true'); + await expect(questions.getByRole('group', { + name: 'How much are you asking for?' + })).toHaveCount(0); + await expect(status).toHaveText('6 case types fit'); + await questions.getByRole('button', { + name: "No, I'm representing myself" + }).click(); + await expect(status).toHaveText('3 case types fit'); + await expect(government).toHaveCount(0); + await questions.getByRole('button', { + name: 'Jury of 12' + }).click(); + await expect(status).toHaveText('1 case type fits'); + await expect(dialog.locator('.code-search__path-choice')).not.toContainText('This court calls it'); + await auditCodeSearch(page, 'court-step-questions'); + await dialog.locator('input[name="code-search-result"]').check(); + await toCheck(page); + await expect(dialog.locator('#code-search-confirm')).toContainText('(Jury - 12)'); + }); + + test('long choice rows collapse, categories filter, and court terms explain themselves', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const queries = []; + const categories = ['Personal Injury/Wrongful Death', 'Civil', 'Civil - Amount Claimed Greater Than $10,000', 'Probate', 'Chancery', 'Law', 'Other Actions', 'Commercial Litigation']; + const subrogation = { + key: 'subrogation', + label: 'Subrogation', + text: 'When an insurance company that paid for someone\'s loss sues the person who caused it.', + source: '' + }; + await courtRoutes(page, { + queries, + groups: ['Personal Injury', 'Personal Injury - Subrogation', 'Dram Shop'].map((name, i) => ({ + key: `g-${i}`, + name: `Complaint / Petition - ${name}`, + variants: [`Complaint / Petition - ${name}`], + glossary: i === 1 ? [subrogation] : [], + })), + extra: { + categories: categories.map((label, i) => ({ + value: label, + label, + count: 10 - i + })), + }, + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('complaint'); + const row = dialog.getByRole('group', { + name: 'Case category' + }); + await expect(row.getByRole('button')).toHaveCount(7); + await row.getByRole('button', { + name: 'Show 3 more choices for Case category' + }).click(); + await expect(row.getByRole('button')).toHaveCount(9); + await expect(row.getByRole('button', { + name: 'Law 5' + })).toBeFocused(); + await row.getByRole('button', { + name: 'Civil 9', + exact: true + }).click(); + await expect.poll(() => queries.at(-1)?.category).toBe('Civil'); + const prefix = dialog.locator('.code-search__name-prefix').first(); + await expect(prefix).toHaveClass(/is-shared/); + await expect(dialog.getByRole('radio', { + name: 'Complaint / Petition - Personal Injury', + exact: true + })).toBeVisible(); + const term = dialog.locator('.code-search__glossary summary'); + await expect(term).toHaveText('What does “subrogation” mean?'); + await term.click(); + await expect(dialog.locator('.code-search__glossary')).toContainText('insurance company'); + await expect(dialog.getByRole('radio', { + name: 'Complaint / Petition - Personal Injury - Subrogation' + })).not.toBeChecked(); + await auditCodeSearch(page, 'categories-and-glossary'); + }); + + test('a search with matches only in the other case stage links to them', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + const queries = []; + await courtRoutes(page, { + queries, + groups: params => params.get('existing_case') === 'yes' ? [{ + key: 'seal', + name: 'Impound/Seal', + variants: ['Impound/Seal'] + }] : [], + extra: params => ({ + other_stage_total: params.get('existing_case') === 'yes' ? 0 : 10 + }), + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await searchbox(page).fill('seal eviction'); + await expect(page.locator('#code-search-status')).toContainText('No matching filing types'); + const link = button(page, '10 results in “existing case” filings'); + await expect(link).toBeVisible(); + await auditCodeSearch(page, 'other-stage-link'); + await link.click(); + await expect(page.locator('#code-search-stage-label')).toHaveText('Existing case'); + await expect(dialog.getByRole('radio', { + name: 'Impound/Seal' + })).toBeVisible(); + expect(queries.at(-1)).toMatchObject({ + existing_case: 'yes', + q: 'seal eviction' + }); + await expect(dialog.locator('#code-search-other-stage')).toBeHidden(); + }); + + test('earlier groups remain usable after loading another page of groups', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await courtRoutes(page, { + groups: params => Array.from({ + length: Number(params.get('offset')) ? 1 : 20 + }, (_, i) => ({ + key: `type-${Number(params.get('offset')) + i}`, + name: `Filing ${Number(params.get('offset')) + i}`, + variants: [`Filing ${Number(params.get('offset')) + i}`], + })), + extra: { + total: 21 + }, + }); + await openSavedDraft(page); + const dialog = await openSearch(page); + await button(page, 'Complaint').click(); + await expect(page.locator('.code-search__group')).toHaveCount(20); + await dialog.getByRole('button', { + name: /^Show more/ + }).click(); + await expect(page.locator('.code-search__group')).toHaveCount(21); + await dialog.getByRole('radio', { + name: 'Filing 0', + exact: true + }).check(); + await toCourts(page); + await dialog.locator('input[name="code-search-result"]').check(); + await expect(button(page, 'Next: check and use')).toBeEnabled(); + }); + + test('rejects a retired path without changing form choices and restores focus on Escape', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await mockSearch(page, { + reject: true + }); + await openSavedDraft(page); + await openSearch(page); + await searchbox(page).fill('eviction'); + await page.getByRole('dialog').getByRole('radio').check(); + await toCheck(page); + await useCode(page); + await expect(page.getByRole('dialog')).toContainText('The code list has changed'); + await expect(page.locator('#court_code')).toHaveValue('cook:cvd1'); + await page.keyboard.press('Escape'); + await expect(page.getByRole('dialog')).not.toBeVisible(); + await expect(page.getByRole('button', { + name: 'Find the right filing code' + })).toBeFocused(); + }); + + test('a late search cannot replace a newer query or leave an old selection active', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await mockSearch(page); + let delivered = false; + await page.route('**/api/filing-code-search/**', async route => { + const params = new URL(route.request().url()).searchParams; + if (params.get('q') !== 'delayed') return route.fallback(); + await new Promise(resolve => setTimeout(resolve, 800)); + await route.fulfill({ + json: { + results: [{ + ...result, + filing_type: { + code: 'old', + name: 'Outdated result' + } + }], + total: 1, + corrected_terms: [], + stale: false, + } + }).catch(() => {}); + delivered = true; + }); + await openSavedDraft(page); + await openSearch(page); + await searchbox(page).fill('eviction'); + await page.getByRole('dialog').getByRole('radio').check(); + const waiting = page.waitForRequest(request => new URL(request.url()).searchParams.get('q') === 'delayed'); + await searchbox(page).fill('delayed'); + await expect(button(page, 'Next: court and case type')).toBeDisabled(); + await waiting; + await searchbox(page).fill('eviction'); + await expect(page.getByRole('dialog').getByRole('radio', { + name: 'Complaint' + })).toBeVisible(); + await expect.poll(() => delivered).toBe(true); + await expect(page.getByRole('dialog')).not.toContainText('Outdated result'); + await expect(button(page, 'Next: court and case type')).toBeDisabled(); + }); + + test('shows unavailable paths without allowing selection, and keeps focus in the modal', async ({ + page + }) => { + await mockCourtLists(page, { + flat: true + }); + await mockSearch(page, { + unavailable: true + }); + await openSavedDraft(page); + await openSearch(page); + await searchbox(page).fill('eviction'); + await expect(page.getByRole('dialog').getByRole('radio')).toBeDisabled(); + await expect(button(page, 'Next: court and case type')).toBeDisabled(); + for (let i = 0; i < 12; i++) { + await page.keyboard.press('Tab'); + expect(await page.evaluate(() => document.querySelector('#code-search-dialog').contains(document.activeElement))).toBe(true); + } + await page.setViewportSize({ + width: 375, + height: 740 + }); + expect(await page.getByRole('dialog').evaluate(el => el.scrollWidth <= el.clientWidth)).toBe(true); + await auditCodeSearch(page, 'unavailable-path-mobile'); + }); +}); + test.describe('guided court questions (Vermont-shaped)', () => { test('rapid unit changes stay open until Update, and a late answer cannot win', async ({ page diff --git a/efile_app/uv.lock b/efile_app/uv.lock index 5af1b657..faf4b4af 100644 --- a/efile_app/uv.lock +++ b/efile_app/uv.lock @@ -997,6 +997,7 @@ dependencies = [ { name = "python-dotenv" }, { name = "pyyaml" }, { name = "requests" }, + { name = "snowballstemmer" }, { name = "suffolklitlab-macourts" }, { name = "suffolklitlab-vtcourts" }, { name = "tiktoken" }, @@ -1039,6 +1040,7 @@ requires-dist = [ { name = "python-dotenv", specifier = ">=1.0" }, { name = "pyyaml", specifier = ">=6.0,<7" }, { name = "requests", specifier = ">=2.31,<3" }, + { name = "snowballstemmer", specifier = ">=3,<4" }, { name = "suffolklitlab-macourts", git = "https://github.com/SuffolkLITLab/MACourts?rev=main" }, { name = "suffolklitlab-vtcourts", git = "https://github.com/SuffolkLITLab/VTCourts?rev=main" }, { name = "tiktoken", specifier = ">=0.12.0" }, @@ -2186,6 +2188,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, ] +[[package]] +name = "snowballstemmer" +version = "3.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/43/f8/0a71edf031f03c40db17503cb8ca78a69a171254e568e7db241b0ab57ea1/snowballstemmer-3.1.1.tar.gz", hash = "sha256:e07bbc54a0d798fe6010a12398422e62a8bfbba95c394fd0956ef58cb4d3e260", size = 123314, upload-time = "2026-06-03T00:56:40.194Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4c/07/2ebca9b11fb9be7340a818d8d6f63feaebb146be2c4afbd6061701d6df6e/snowballstemmer-3.1.1-py3-none-any.whl", hash = "sha256:7e207fa178741da09cdee59d3ecec3827ad5f92b1fc5c9ff3755b639f71f5752", size = 104164, upload-time = "2026-06-03T00:56:38.614Z" }, +] + [[package]] name = "sortedcontainers" version = "2.4.0" diff --git a/fly.toml b/fly.toml index e67d00f7..024802e3 100644 --- a/fly.toml +++ b/fly.toml @@ -10,14 +10,26 @@ primary_region = 'lax' [env] DJANGO_SETTINGS_MODULE = "efile.settings_staging" + # Long-running workers' progress reaches fly logs as it happens. + PYTHONUNBUFFERED = "1" + # Copy filing codes straight from the EFSP proxy's codes database, read + # only. Set the connection with: fly secrets set EFSP_CODES_DATABASE_URL=... + FILING_CODE_SYNC_MODE = "database" + # Staging uses the test EFSP, which updates its codes from Tyler daily at + # 19:35 Eastern; copy them two hours later. (Production EFSP: 02:13, so 04:30.) + FILING_CODE_SYNC_TIME = "21:30" + FILING_CODE_SYNC_TIMEZONE = "America/New_York" [deploy] # Run database migrations before each release release_command = "uv run python manage.py migrate --noinput --fake-initial" + release_command_timeout = "30m" [processes] app = "uv run gunicorn efile.asgi:application -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 --workers 2 --timeout 60 --access-logfile - --error-logfile - --log-level debug" extraction_worker = "uv run python manage.py process_document_extractions" + # Daily copy + index at FILING_CODE_SYNC_TIME, plus staff-requested resyncs and rebuilds. + code_index_worker = "uv run python manage.py refresh_filing_code_index --daily --retry-interval 900" [http_service] internal_port = 8000 diff --git a/run_all.sh b/run_all.sh new file mode 100755 index 00000000..93effc8f --- /dev/null +++ b/run_all.sh @@ -0,0 +1,315 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Script location and project paths +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$SCRIPT_DIR" +APP_DIR="$REPO_ROOT/efile_app" + +# Terminal colors (if running in interactive terminal) +if [ -t 1 ]; then + BOLD="\033[1m" + GREEN="\033[0;32m" + BLUE="\033[0;34m" + YELLOW="\033[0;33m" + RED="\033[0;31m" + NC="\033[0m" # No Color +else + BOLD="" + GREEN="" + BLUE="" + YELLOW="" + RED="" + NC="" +fi + +log_info() { + echo -e "${BLUE}[INFO]${NC} $*" +} + +log_success() { + echo -e "${GREEN}[SUCCESS]${NC} $*" +} + +log_warn() { + echo -e "${YELLOW}[WARN]${NC} $*" +} + +log_error() { + echo -e "${RED}[ERROR]${NC} $*" >&2 +} + +# Default configuration +HOST="127.0.0.1" +PORT="8000" +START_LOCALSTACK=true +STOP_DOCKER_ON_EXIT=false +START_EXTRACTION_WORKER=true +START_CODE_INDEX_WORKER=true +LEGACY_CODE_CRAWL=false +SKIP_SYNC=false +SKIP_MIGRATE=false + +show_help() { + echo -e "${BOLD}Usage:${NC} ./run_all.sh [options]" + echo "" + echo "Start the LITEFile application for local development without building Docker containers." + echo "Automatically sets up a local .venv (using Astral uv), runs database migrations," + echo "manages LocalStack for S3 document storage, and starts extraction and filing code search workers." + echo "" + echo -e "${BOLD}Options:${NC}" + echo " --host Bind address for Django development server (default: 127.0.0.1)" + echo " --port , -p Port for Django development server (default: 8000)" + echo " --no-docker Skip starting LocalStack via Docker" + echo " --stop-docker-on-exit Stop LocalStack container when exiting (default: keep running)" + echo " --no-worker Do not start the document extraction worker" + echo " --no-code-index Do not start the filing code index refresh worker (enabled by default)" + echo " --with-code-index Enable the filing code index worker (default; kept for compatibility)" + echo " --legacy-code-crawl Use the expensive full crawler until the proxy bulk API is deployed" + echo " --skip-sync Skip 'uv sync' check on startup" + echo " --skip-migrate Skip running database migrations on startup" + echo " -h, --help Show this help message and exit" + echo "" + echo -e "${BOLD}Examples:${NC}" + echo " ./run_all.sh" + echo " ./run_all.sh --port 8080" + echo " ./run_all.sh --no-docker --no-worker" + echo " ./run_all.sh --no-code-index" +} + +# Parse command line arguments +while [[ $# -gt 0 ]]; do + case "$1" in + --host) + HOST="$2" + shift 2 + ;; + --port|-p) + PORT="$2" + shift 2 + ;; + --no-docker|--skip-localstack|--no-localstack) + START_LOCALSTACK=false + shift + ;; + --stop-docker-on-exit) + STOP_DOCKER_ON_EXIT=true + shift + ;; + --no-worker|--skip-worker) + START_EXTRACTION_WORKER=false + shift + ;; + --with-code-index) + START_CODE_INDEX_WORKER=true + shift + ;; + --no-code-index) + START_CODE_INDEX_WORKER=false + shift + ;; + --legacy-code-crawl) + LEGACY_CODE_CRAWL=true + shift + ;; + --skip-sync) + SKIP_SYNC=true + shift + ;; + --skip-migrate) + SKIP_MIGRATE=true + shift + ;; + -h|--help) + show_help + exit 0 + ;; + *) + log_error "Unknown option: $1" + echo "Run './run_all.sh --help' for available options." + exit 1 + ;; + esac +done + +# 1. Ensure Astral uv is available +ensure_uv() { + if command -v uv >/dev/null 2>&1; then + return 0 + fi + for candidate in "$HOME/.local/bin/uv" "$HOME/.cargo/bin/uv"; do + if [ -x "$candidate" ]; then + export PATH="$(dirname "$candidate"):$PATH" + return 0 + fi + done + + log_warn "'uv' not found in PATH. Attempting to install via official script..." + if command -v curl >/dev/null 2>&1; then + curl -LsSf https://astral.sh/uv/install.sh | sh + export PATH="$HOME/.local/bin:$PATH" + if command -v uv >/dev/null 2>&1; then + log_success "'uv' installed successfully." + return 0 + fi + fi + + log_error "Astral 'uv' is required to manage dependencies and virtualenv." + log_error "Install uv with: curl -LsSf https://astral.sh/uv/install.sh | sh" + exit 1 +} + +ensure_uv + +# 2. Check / initialize environment file (.env) +if [ ! -f "$APP_DIR/.env" ]; then + if [ -f "$APP_DIR/.env.example" ]; then + log_info "No efile_app/.env found. Creating one from efile_app/.env.example..." + cp "$APP_DIR/.env.example" "$APP_DIR/.env" + log_success "Created efile_app/.env" + else + log_warn "Neither efile_app/.env nor efile_app/.env.example found." + fi +fi + +# 3. Create / sync local .venv +if [ "$SKIP_SYNC" = false ]; then + log_info "Synchronizing dependencies and virtual environment (uv sync --group dev)..." + uv --directory "$APP_DIR" sync --group dev + log_success "Virtual environment ready at efile_app/.venv" +fi + +# 4. Optional LocalStack (S3 mock) startup via Docker +LOCALSTACK_STARTED=false +if [ "$START_LOCALSTACK" = true ]; then + if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then + log_info "Starting LocalStack (S3 mock) container..." + docker compose -f "$REPO_ROOT/compose.yml" up -d localstack + LOCALSTACK_STARTED=true + log_success "LocalStack is running on port 4566." + + # Ensure target S3 bucket exists in LocalStack + uv --directory "$APP_DIR" run python -c " +import os, boto3 +from dotenv import load_dotenv +load_dotenv('.env') +endpoint = os.getenv('AWS_S3_ENDPOINT_URL', '') +bucket = os.getenv('AWS_S3_BUCKET_NAME', '') +if endpoint and ('localhost' in endpoint or '127.0.0.1' in endpoint) and bucket: + try: + s3 = boto3.client( + 's3', + endpoint_url=endpoint, + aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID', 'test'), + aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY', 'test'), + region_name=os.getenv('AWS_S3_REGION_NAME', 'us-east-1'), + ) + try: + s3.head_bucket(Bucket=bucket) + except Exception: + s3.create_bucket(Bucket=bucket) + print(f'Created LocalStack S3 bucket: {bucket}') + except Exception as err: + print(f'LocalStack bucket check warning: {err}') +" 2>/dev/null || true + else + log_warn "Docker is not installed or daemon is not running. Skipping LocalStack." + log_warn "Document uploads require AWS S3 or LocalStack running on port 4566." + fi +fi + +# 5. Run database migrations +if [ "$SKIP_MIGRATE" = false ]; then + log_info "Applying database migrations..." + uv --directory "$APP_DIR" run python manage.py migrate --noinput --fake-initial + log_success "Database migrations up to date." +fi + +# 6. Process supervision and cleanup handling +CLEANED_UP=0 +cleanup() { + if [ "$CLEANED_UP" -eq 1 ]; then + return + fi + CLEANED_UP=1 + trap - INT TERM EXIT + + echo "" + log_info "Stopping application services..." + + # Terminate all background child jobs + local pids + pids=$(jobs -p) + if [ -n "$pids" ]; then + for pid in $pids; do + kill -TERM "$pid" 2>/dev/null || true + done + # Allow processes up to 1 second to exit gracefully + sleep 1 + for pid in $pids; do + if kill -0 "$pid" 2>/dev/null; then + kill -KILL "$pid" 2>/dev/null || true + fi + done + wait 2>/dev/null || true + fi + + if [ "$LOCALSTACK_STARTED" = true ]; then + if [ "$STOP_DOCKER_ON_EXIT" = true ]; then + log_info "Stopping LocalStack container..." + docker compose -f "$REPO_ROOT/compose.yml" stop localstack 2>/dev/null || true + else + log_info "LocalStack container left running in background." + log_info "To stop it manually: docker compose stop localstack" + fi + fi + + log_success "All local dev processes stopped." + exit 0 +} + +trap cleanup INT TERM + +echo "" +echo -e "${BOLD}======================================================${NC}" +echo -e "${BOLD}${GREEN} LITEFile local development environment is ready!${NC}" +echo -e "${BOLD}======================================================${NC}" +echo -e " ${BOLD}Web application:${NC} http://${HOST}:${PORT}/" +echo -e " ${BOLD}Login page:${NC} http://${HOST}:${PORT}/login" +echo -e " ${BOLD}Staff portal:${NC} http://${HOST}:${PORT}/staff-7c83f0a2/" +if [ "$START_EXTRACTION_WORKER" = true ]; then + echo -e " ${BOLD}Extraction worker:${NC} ACTIVE (processing PDF lead documents)" +fi +if [ "$START_CODE_INDEX_WORKER" = true ]; then + echo -e " ${BOLD}Code index worker:${NC} ENABLED (building or refreshing court filing codes)" + echo " Filing code search becomes available after the first successful index build." +fi +if [ "$LOCALSTACK_STARTED" = true ]; then + echo -e " ${BOLD}S3 Storage:${NC} LocalStack (http://127.0.0.1:4566)" +fi +echo -e "${BOLD}======================================================${NC}" +echo -e "${YELLOW}Press Ctrl+C to shut down all processes.${NC}" +echo "" + +# 7. Start workers and web server +if [ "$START_EXTRACTION_WORKER" = true ]; then + log_info "Starting document extraction worker in background..." + PYTHONUNBUFFERED=1 uv --directory "$APP_DIR" run python manage.py process_document_extractions & +fi + +if [ "$START_CODE_INDEX_WORKER" = true ]; then + log_info "Starting filing code index worker in background..." + CODE_INDEX_ARGS=(--daily --retry-interval 900 --verbosity 2) + if [ "$LEGACY_CODE_CRAWL" = true ]; then + log_warn "Legacy code crawling is enabled: every refresh downloads and rebuilds the full catalog." + CODE_INDEX_ARGS+=(--legacy-crawl) + fi + PYTHONUNBUFFERED=1 uv --directory "$APP_DIR" run python manage.py refresh_filing_code_index "${CODE_INDEX_ARGS[@]}" & +fi + +log_info "Starting Django development server at http://${HOST}:${PORT}..." +PYTHONUNBUFFERED=1 uv --directory "$APP_DIR" run python manage.py runserver "${HOST}:${PORT}" & + +# Wait for any child job to exit; if any crashes or exits, trigger cleanup +wait -n || true +cleanup diff --git a/testing/README.md b/testing/README.md index df3d48d9..20534fef 100644 --- a/testing/README.md +++ b/testing/README.md @@ -20,6 +20,43 @@ awslocal s3 ls s3://litefile-staging/efile-documents/ --recursive Also make sure that `AWS_S3_ENDPOINT_URL = "http://host.docker.internal:4566"` and `AWS_ACCOUNT_ID_ENDPOINT_MODE = "disabled"` in your env. +## Filing code search + +`docker compose up` also starts `code_index_worker`, which builds the filing-code +search indexes and refreshes them daily. The web container's migrations must +finish before the worker starts. Search is unavailable until a state's first +complete catalog has been indexed; opening the search dialog does not start a +build. The court lists remain usable while indexing runs. + +If Compose is already running, start the worker in a terminal you control: + +```bash +docker compose up code_index_worker +``` + +To build only Illinois once in the existing web container instead: + +```bash +docker compose exec web uv run python manage.py refresh_filing_code_index --jurisdiction illinois --verbosity 2 --cache-dir /data/filing-code-cache +``` + +Progress lists each court as it is fetched. Wait for `indexed ... filing paths`, +then search becomes available (an open search retries automatically). Indexes +persist in the shared database volume. If a refresh +fails, the command reports the error and keeps any previous complete index. +Temporary court-service failures are retried within each request. The recurring +worker retries failed states after one minute and skips indexes refreshed within +the daily interval, including after a restart. A one-shot command still exits +with an error if its retries fail. Completed courts are checkpointed in compressed +files in the shared volume and reused for up to six hours after a failed build. + +Local keyword searches use SQLite FTS5, including jurisdiction and new/existing-case +filters. The schema migration builds postings from any existing catalog; later +catalog changes maintain them automatically. Run `docker compose exec web uv run +python manage.py migrate --noinput` when updating a running stack. Compose runs +migrations automatically when starting the web container. Python tests run migrations +too, because `--nomigrations` would omit the FTS5 table and its synchronization triggers. + ## Calculating fees and filing end-to-end The EFSP proxy **downloads each document itself** from the `data_url` in the