Preserve existing-case parties through fees and submission

docs/developer-notes/existing-case-import.md

@@ -0,0 +1,93 @@
+# Importing existing court parties
+
+Implementation and validation for [issue #291](https://github.com/SuffolkLITLab/LITEFile/issues/291#issuecomment-6069909727).
+
+## Case-detail contract
+
+`services/existing_cases.py` separates authenticated retrieval, normalization, preview, atomic application, and envelope reconciliation. It uses the raw detail endpoint:
+
+```text
+GET /jurisdictions/{jurisdiction}/cases/courts/{court}/cases/{tracking_id}
+```
+
+The seven authorized Illinois staging examples were queried using both the raw response and `X-API-VERSION: JSON-V1`. Raw responses remain the import source: JSON-V1 does not preserve available contacts, and its individual ID extraction does not explicitly select the `CASEPARTYID` category.
+
+Two differences found during live validation are covered by fixtures:
+
+- Search returns the EFM UUID needed by detail retrieval, fees, and submission. Detail responses place a local court-system ID in their top-level `caseTrackingID`. The original EFM UUID appears in `caseLineageCase`. Import verifies that identity, the selected docket number (allowing punctuation differences), and the returned court. It preserves the EFM UUID as `previous_case_id` and records the local ID separately in the snapshot.
+- Marion, Kane, and Winnebago return additional parties with roles absent from the selected case type's current party-type list. Import verifies those roles against the court's full `/party_types` catalog. It does not reinterpret those parties as a plaintiff or defendant. Required-role checks still use the selected case type's requirements.
+
+## Field mapping
+
+| Raw detail field | Durable destination | Behavior |
+| --- | --- | --- |
+| Search UUID, verified against top-level ID or `caseLineageCase` | Draft `previous_case_id`; snapshot `tracking_id` | Used for both fees and submission; never replaced with the local ID |
+| Top-level detail `caseTrackingID` | Snapshot `court_local_case_id` | Retained separately |
+| `caseCourt.organizationIdentification.identificationID` | Snapshot `court` | Must match the selected court |
+| `caseDocketID`, `caseTitleText` | Draft docket and title; snapshot | Court values replace search/browser suggestions on confirmation |
+| `caseCategoryText`, Tyler augmentation `caseTypeText` | Draft category/type codes; snapshot | Court classifications drive the outgoing envelope |
+| Individual `personOtherIdentification` or organization's nested `identification` with category `CASEPARTYID` | Party `external_party_id` | Category-specific selection; unrelated IDs are ignored |
+| `EntityPerson` / `EntityOrganization` | Person fields / `organization_name` | Organizations serialize with `person_type: business` |
+| `personGivenName`, `personMiddleName`, `personSurName`, `personNameSuffixText` | First, middle, last name, suffix | Preserved without name-based deduplication |
+| `organizationName` | Organization name | Preserved as one name |
+| `caseParticipantRoleCode` | Party type code and verified catalog name | Missing/unrecognized roles make the import partial |
+| `ContactEmailID`, `ContactTelephoneNumber.telephoneNumberFullID` | Email and phone | First available supported value |
+| `ContactMailingAddress` / `ContactAddress`, structured delivery points, city, state, postal code, country | Mailing address fields | Optional information; missing country remains unknown |
+| `caseOtherEntityAttorney`, `CASEPARTYATTORNEYID` or `ATTORNEYID`, bar identification, represented-party references | Party `representation.attorneys` | Normalized attorney identities with their ID category, supported names/contacts, and associations retained |
+| No returned attorney association | `representation.status: unknown` | Does not imply self-representation |
+
+Unsupported personal attributes, historical documents, prior filing codes, payments, and service selections are not imported. The current proxy person input exposes no representation-change fields. The UI preserves and describes returned representation; it does not send guessed pro-se or attorney changes. Existing-party IDs reference the court's existing associations. `is_form_filler` retains its separate firm-filer meaning.
+
+The bounded snapshot includes schema version, court and case identity, jurisdiction, source endpoint, retrieval time, status, problems, and normalized parties. An absent snapshot is `not_loaded`; a verified empty roster is `loaded`; missing essential fields are `partial`; retrieval/authentication/format failures are `failed`. Credentials and raw responses are not saved in the draft or logged by the import service.
+
+## Workflow and identity rules
+
+Confirmation previews the server-fetched roster and applies it when the filer confirms. Plan-linked cases use the same confirmation step. Active older drafts cannot proceed to People, Fees, or Review until the court import is confirmed; envelope preparation also enforces this requirement. Submitted drafts cannot be imported into or rewritten.
+
+Imported parties have `source: court`, independently of whether their ID is present. Their ordinary edit/remove actions are unavailable and direct requests are rejected. “This is me” links the account to the existing row without renaming or deleting it. “Who are you filing for?” selects durable court rows independently of that account link. Account contact information remains separate. A separate “Add a new party” action creates `source: added`; the new party's role must be chosen from the court's current published choices. If a required role is absent from the verified roster, the UI explains the missing role and asks for an explicit addition or support; it does not fabricate a placeholder. Server required-role validation still runs.
+
+Atomic imports serialize on the draft and reuse court rows by ID. A conditional database constraint prevents duplicate nonblank IDs within a draft and source case. Equal names with different IDs remain distinct. Case changes/rejection clear court and explicitly added case-specific parties, links, and quotes while retaining uploads and account contact details. A stale detail response cannot overwrite another selection.
+
+Extraction suggestions cannot edit a confirmed court roster. The People screen asks the filer to compare document caption evidence with the court roster. Legacy draft writes cannot replace imported parties or verified case classifications.
+
+The shared fee/submission preparation receives the authoritative draft. It verifies case identity, party names and roles against the snapshot, filing-party selection, unique IDs, and document references. Existing rows serialize with `tyler_id` and `is_new: false` in either party collection. Explicitly added rows are reconciled against their saved draft row and serialize with `is_new: true`, without a court ID. Internal draft-row identifiers are removed before forwarding. Each party appears once. Quote fingerprints include imported IDs, names, representation, and filing-party selections.
+
+## Illinois staging results
+
+Validated on October 8, 2026 (America/New_York), against `https://efile-test.suffolklitlab.org`, using authorized local test credentials. Each test used its captured court roster, a selected existing filing party, a current Motion filing code, a non-confidential document type, the required filing component, the repository's public test PDF, and an existing staging waiver payment account.
+
+The browser payload module generated the envelopes; the shared server preparation reconciled them against persisted imported drafts before the live fee and filing requests. Every outgoing court participant had the original `CASEPARTYID` and `is_new: false`, and every document referenced the selected entry in `users`. Returned `caseId` values matched the original search UUIDs. Raw responses, payloads, and credentials stayed under `/tmp`; committed fixtures replace names, contacts, party/attorney IDs, and bar numbers with synthetic values.
+
+| Court | Supplied case number | Imported parties | Motion code | Fee response | Submission response | Staging envelope |
+| --- | --- | ---: | --- | --- | --- | --- |
+| Marion | `2025SC5` | 3 | `143177` | 200 | 200 | `324947` |
+| Kane | `2024EV001752` | 4 | `77747` | 200 | 200 | `324948` |
+| Kane | `2005SC000985` | 4 | `6344` | 200 | 200 | `324949` |
+| Winnebago | `2019-D-0000655` | 5 | `6344` | 200 | 200 | `324950` |
+| Winnebago | `2019-SC-0001642` | 2 | `196589` | 200 | 200 | `324951` |
+| Lake | `2024SC00003388` | 2 | `53156` | 200 after timeout/retry | 200 | `324952` |
+| Lake | `2024DC00000572` | 2 | `53159` | 200 | 200 | `324953` |
+
+All seven submissions returned an envelope ID and one filing ID. This verifies staging envelope acceptance, not a later clerk disposition. No production filing was made.
+
+## Regression checks
+
+`efile/tests/test_existing_cases.py` covers the seven sanitized detail responses and full role catalogs, category-specific ID selection, organizations and equal names, import idempotency, uniqueness, required roles, immutable details, account linking, explicit new parties, lifecycle changes, stale results, resume, failures, and substituted payload identities.
+
+Its browser test runs the actual Django views with sanitized court responses: lookup, confirmation, read-only roster, native keyboard filing-party selection, account linking, and Review. The JavaScript suite checks both party collections and document references. Existing routing tests now provide a trusted detail response rather than assuming confirmation can bypass import.
+
+From `efile_app/`:
+
+```bash
+uv run pytest -q efile/tests/test_existing_cases.py
+uv run pytest -q efile/tests/test_existing_cases.py efile/tests/test_fee_quotes.py efile/tests/test_handoff.py efile/tests/test_people_flow.py efile/tests/test_review_submit_flow.py
+uv run pytest -q efile/tests --ignore=efile/tests/test_integration.py --ignore=efile/tests/tests.py -m 'not integration'
+npm run test:unit
+uv run ruff check .
+uv run ty check
+uv run python manage.py makemigrations --check --dry-run
+```
+
+The repeatable backend suite passed with **1,675 passed and three integration tests deselected**. The final affected regression suite passed with **167 tests**, including the real-view browser flow and attorney-association mapping. JavaScript unit tests passed with **88 tests**. Ruff, Ty, changed-file ESLint, template formatting, migration consistency, and `git diff --check` passed. The repository-wide JavaScript lint command also scans an unrelated local broken virtual environment; ESLint was therefore run against every changed JavaScript file.
+
+The browser test requires the repository's npm dependencies and installed Playwright Chromium. The live seven-case matrix was performed separately from the repeatable fixture/browser suite. Migration `0038_existing_case_identity` must be applied before deploying the application changes.

efile_app/efile/api/filing_views.py

@@ -166,7 +166,7 @@ def payment_fees(request):
             # Must match what submit_final_filing sends, or fees are quoted
             # against a payload that differs from the one actually filed.
             try:
-                prepare_efile_payload(efile_data, jurisdiction_id, court_id)
+                prepare_efile_payload(efile_data, jurisdiction_id, court_id, draft=draft)
             except PayloadValidationError as error:
                 # Known-bad payload: answer with the specific reason rather than
                 # letting the EFSP reply with a code-list error no filer can act on.

efile_app/efile/api/suffolk_api_views.py

@@ -160,9 +160,8 @@ def lookup_case(request):
 
         success = True
         if not case_info.get("caseTrackingID") and not case_info.get("caseCategoryText"):
-            logger.warning(f"Could not extract case information from API response for case {case_number}: {api_data}")
-            # Return what we have from the API for debugging
-            case_info = api_data[0] if api_data and len(api_data) > 0 else {}
+            logger.warning("Could not extract case information from API response")
+            case_info = {}
             success = False
 
         response = JsonResponse(

efile_app/efile/middleware.py

@@ -1,12 +1,19 @@
+from django.contrib import messages
 from django.contrib.auth import logout
 from django.http import JsonResponse
-from django.shortcuts import render
+from django.shortcuts import redirect, render
 from django.utils.deprecation import MiddlewareMixin
 
 from efile.models import FilingDraft
-from efile.services.current_drafts import DraftIdentityError, resolve_explicit_draft
+from efile.services.current_drafts import DraftIdentityError, get_current_draft, resolve_explicit_draft
 from efile.services.draft_urls import draft_url
+from efile.services.existing_cases import import_ready
 from efile.utils.jurisdiction_stuff import get_jurisdiction_from_request
+from efile.workflow import ExistingCase
+
+# Steps that read or file the court roster. An existing case reaches them only
+# once its import is loaded and confirmed; this is the one place that says so.
+NEEDS_CONFIRMED_CASE = {"parties", "party_details", "case_questions", "payment", "case_review"}
 
 
 class DraftIdentityMiddleware(MiddlewareMixin):
@@ -21,7 +28,13 @@ def process_view(self, request, view_func, view_args, view_kwargs):
         if request.resolver_match.url_name == "filing_confirmation":
             statuses = (FilingDraft.Status.SUBMITTED,)
         try:
-            resolve_explicit_draft(request, jurisdiction=view_kwargs.get("jurisdiction"), statuses=statuses)
+            jurisdiction = view_kwargs.get("jurisdiction")
+            draft = resolve_explicit_draft(request, jurisdiction=jurisdiction, statuses=statuses)
+            if request.resolver_match.url_name in NEEDS_CONFIRMED_CASE:
+                draft = draft or get_current_draft(request, jurisdiction=jurisdiction)
+                if draft and draft.existing_case == ExistingCase.EXISTING and not import_ready(draft):
+                    messages.error(request, "Confirm the court case and load its parties before continuing.")
+                    return redirect("case_confirmation", jurisdiction=draft.jurisdiction)
         except DraftIdentityError as error:
             return self.process_exception(request, error)
 

efile_app/efile/migrations/0038_existing_case_identity.py

@@ -0,0 +1,40 @@
+# Generated by Django 5.2.17 on 2026-10-08 22:43
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("efile", "0037_efsp_code_copies_and_jobs"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="filingdraft",
+            name="existing_case_snapshot",
+            field=models.JSONField(blank=True, default=dict),
+        ),
+        migrations.AddField(
+            model_name="filingparty",
+            name="representation",
+            field=models.JSONField(blank=True, default=dict),
+        ),
+        migrations.AddField(
+            model_name="filingparty",
+            name="source",
+            field=models.CharField(default="manual", max_length=20),
+        ),
+        migrations.AddField(
+            model_name="filingparty",
+            name="source_case_id",
+            field=models.CharField(blank=True, max_length=255),
+        ),
+        migrations.AddConstraint(
+            model_name="filingparty",
+            constraint=models.UniqueConstraint(
+                condition=models.Q(("source", "court"), models.Q(("external_party_id", ""), _negated=True)),
+                fields=("draft", "source_case_id", "external_party_id"),
+                name="unique_imported_case_party",
+            ),
+        ),
+    ]

efile_app/efile/models.py

@@ -311,6 +311,7 @@ class Status(models.TextChoices):
     document_type_code = models.CharField(max_length=100, blank=True)
     document_type_name = models.CharField(max_length=255, blank=True)
 
+    existing_case_snapshot = models.JSONField(default=dict, blank=True)
     previous_case_id = models.CharField(max_length=255, blank=True)
     docket_number = models.CharField(max_length=255, blank=True)
     case_title = models.CharField(max_length=500, blank=True)
@@ -573,13 +574,20 @@ def __str__(self):
 class FilingParty(models.Model):
     """Person or organization associated with a filing draft."""
 
+    # Rows on an existing court case's roster: "court" rows came from the
+    # court and cannot be edited; "added" rows join the case in this filing.
+    CASE_ROSTER_SOURCES = ("court", "added")
+
     draft = models.ForeignKey(FilingDraft, on_delete=models.CASCADE, related_name="parties")
     role = models.CharField(max_length=50)
     sort_order = models.PositiveIntegerField(default=0)
 
     party_type = models.CharField(max_length=100, blank=True)
     party_type_name = models.CharField(max_length=255, blank=True)
     external_party_id = models.CharField(max_length=255, blank=True)
+    source = models.CharField(max_length=20, default="manual")
+    source_case_id = models.CharField(max_length=255, blank=True)
+    representation = models.JSONField(default=dict, blank=True)
 
     # Set on the review screen, where the people read off the document are
     # first shown: "this one is me". It is recorded on the party rather than
@@ -632,9 +640,18 @@ class Meta:
         ordering = ["role", "sort_order", "created_at"]
         constraints = [
             models.UniqueConstraint(fields=["draft", "role", "sort_order"], name="unique_party_order_per_draft_role"),
+            models.UniqueConstraint(
+                fields=["draft", "source_case_id", "external_party_id"],
+                condition=models.Q(source="court") & ~models.Q(external_party_id=""),
+                name="unique_imported_case_party",
+            ),
         ]
         verbose_name_plural = "Filing parties"
 
+    @property
+    def on_case_roster(self) -> bool:
+        return self.source in self.CASE_ROSTER_SOURCES
+
     def __str__(self):
         display_name = " ".join(part for part in [self.first_name, self.middle_name, self.last_name] if part)
         return display_name or self.organization_name or f"{self.role} for draft #{self.draft_id}"

efile_app/efile/services/drafts.py

@@ -208,6 +208,8 @@ def write_case_data(
     """
 
     data = dict(case_data or {})
+    draft.refresh_from_db(from_queryset=FilingDraft.objects.select_for_update())
+    old_identity = (draft.court_code, draft.previous_case_id, draft.existing_case)
     update_fields: list[str] = []
 
     for field, sources in _DRAFT_FIELD_SOURCES.items():
@@ -221,6 +223,19 @@ def write_case_data(
             setattr(draft, field, value)
             update_fields.append(field)
 
+    # existing_cases imports this module's draft statuses.
+    from efile.services.existing_cases import SNAPSHOT_CASE_FIELDS, clear_case_import
+
+    if old_identity != (draft.court_code, draft.previous_case_id, draft.existing_case):
+        clear_case_import(draft)
+
+    elif draft.existing_case_snapshot.get("status") == "loaded":
+        snapshot = draft.existing_case_snapshot
+        for field in SNAPSHOT_CASE_FIELDS:
+            if getattr(draft, field) != snapshot[field]:
+                setattr(draft, field, snapshot[field])
+                update_fields.append(field)
+
     if "optional_services" in data:
         services = data.get("optional_services") or []
         if draft.optional_services != services:
@@ -270,7 +285,7 @@ def _write_parties(draft: FilingDraft, data: dict[str, Any]) -> None:
             party_type = _first_present(data, _PETITIONER_PARTY_TYPE_KEYS)
             if party_type is not _MISSING:
                 values["party_type"] = _as_str(party_type)
-        if values:
+        if values and not draft.existing_case_snapshot:
             FilingParty.objects.update_or_create(draft=draft, role=role, sort_order=0, defaults=values)
 
 
@@ -334,6 +349,11 @@ def read_case_data(draft: FilingDraft | None) -> dict[str, Any]:
     filing_parties = [
         {
             "id": party.pk,
+            "external_party_id": party.external_party_id,
+            "source": party.source,
+            "source_case_id": party.source_case_id,
+            "representation": party.representation,
+            "is_self": party.is_self,
             "role": party.role,
             # Whether the filing is made on behalf of this party -- the filer
             # themselves when they are one, someone they are filing for when