feat: add verified industry reference examples - #224
Open
svonava wants to merge 4 commits into
Open
Conversation
Contributor
Author
|
@coderabbitai review |
Contributor
Author
|
Review update:\n\n- Removed the endpoint-specific Docling identifier translation.\n- All new runs now use the canonical SIE model ID, docling, on local and Cloud endpoints.\n- Preserved the acquisition-time configured identifier in each verified manifest and documented why it differs from the canonical model actually served.\n- Reran 42 locked offline tests plus Ruff lint and format across all three packages.\n\nThis keeps the examples compatible with PR #222 and avoids client-side model naming rules. |
📝 WalkthroughWalkthroughThree runnable SIE examples were added: financial filing review, maintenance triage, and prior authorization review. Each includes an end-to-end pipeline, deterministic validation, artifact manifests, CLI commands, configuration, documentation, and tests. ChangesFinancial Filing Review
Maintenance Triage Review
Prior Authorization Review
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Reviewer
participant AgentCLI
participant SIE
participant Validator
participant RunDirectory
Reviewer->>AgentCLI: Start review with run ID
AgentCLI->>SIE: Parse, retrieve, rerank, and extract evidence
SIE-->>AgentCLI: Model outputs and ranked source chunks
AgentCLI->>Validator: Gate evidence and construct review
Validator-->>AgentCLI: Validated review or failure
AgentCLI->>RunDirectory: Write review, evaluation, and manifest artifacts
RunDirectory-->>Reviewer: Return evidence bundle
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Author
|
@coderabbitai full review |
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 49 minutes. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
examples/maintenance-triage-agent/tests/test_review.py (1)
214-221: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a failure test for the chunk-identity gate.
_require_ranked_evidencerejects evidence rows missingchunk_idortext(review.py lines 181-182), and_row_for_locationrejects ambiguous location matches (review.py lines 227-228). Neither branch is exercised, yet both are provenance guarantees this example advertises.🧪 Suggested addition
def test_review_fails_closed_on_evidence_without_chunk_identity() -> None: evidence = ranked_evidence() del evidence[0]["chunk_id"] try: build_review(structured_data(), evidence) except RuntimeError as exc: assert "chunk identity" in str(exc) else: raise AssertionError("build_review accepted evidence without a chunk id")As per path instructions, "Check that commands match the implementation and that tests cover failure paths".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/maintenance-triage-agent/tests/test_review.py` around lines 214 - 221, Add failure-path coverage in test_review.py for the chunk-identity gate used by build_review: create valid ranked evidence, remove a row’s chunk_id (or text), and assert RuntimeError with the expected “chunk identity” message; fail explicitly if build_review accepts it. Also add coverage for _row_for_location rejecting ambiguous location matches, preserving the advertised provenance guarantees.Source: Path instructions
examples/prior-authorization-review-agent/tests/test_review.py (1)
1-360: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd failure-path tests for the remaining fail-closed gates.
Coverage exists for
build_review's field-mismatch gates, but not for_source_scope(no matching fragment),_require_ranked_evidence(missing/malformed reranked rows),_chunks(no usable chunks), or the "not overdue" branch inbuild_review. These are part of the same deterministic fail-closed contract the README highlights ("Missing, conflicting, or untraceable model output stops the run").As per path instructions, "Check that commands match the implementation and that tests cover failure paths."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/tests/test_review.py` around lines 1 - 360, Add failure-path tests covering the remaining fail-closed gates: make _source_scope reject evidence with no matching fragment, make _require_ranked_evidence reject missing or malformed reranked rows, and make _chunks reject input with no usable chunks. Add a build_review test for the not-overdue encounter case, asserting the documented failure behavior. Reuse existing fixtures/helpers and assert the expected RuntimeError messages.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/financial-filing-agent/financial_filing/review.py`:
- Around line 136-143: Update _require_entity_evidence so required_spans also
includes the restated diluted EPS source span "$1.34", preserving the existing
normalization, missing-span detection, and error behavior.
In `@examples/maintenance-triage-agent/maintenance_triage/__init__.py`:
- Line 1: Update the module docstring in maintenance_triage to describe that the
example performs read-only maintenance triage using the published NTSB SPC-24-06
page spread, removing the inaccurate reference to synthetic plant records.
In `@examples/maintenance-triage-agent/maintenance_triage/review.py`:
- Around line 478-480: Update the SIEClient construction in the maintenance flow
to pass the already-read timeout variable from
config["cluster"]["provision_timeout_s"] instead of the hardcoded 900, while
preserving the existing timeout forwarding for subsequent calls.
- Around line 452-462: Update the mapped-field and triage calculation flow to
derive both asserted values: add and validate east_palestine_time via
_source_match in _map_exact_source_fields, then compute
salem_to_east_palestine_minutes from the mapped times like
sebring_to_salem_minutes; also derive derailment.total_cars from the validated
derailment_statement as 1 hopper car plus 37 others, replacing the literal 38.
In `@examples/maintenance-triage-agent/README.md`:
- Around line 62-77: Update the evidence-bundle artifact list in the README to
include the per-stage GLiNER outputs written by run():
raw/entities-detector-{index}.json and raw/entities-outcome.json. Keep the
existing artifact entries unchanged and ensure the documented paths match the
implementation.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 389-415: Update review.py lines 389-415 to populate
missing_documentation, review_conclusion, and payment_action from the validated
outcome fields rather than literals; update README.md lines 26-32 to accurately
describe verbatim versus canonical fields; update evaluate.py lines 50-60 to use
token/substring assertions matching _require_tokens instead of fixed-string
equality.
---
Nitpick comments:
In `@examples/maintenance-triage-agent/tests/test_review.py`:
- Around line 214-221: Add failure-path coverage in test_review.py for the
chunk-identity gate used by build_review: create valid ranked evidence, remove a
row’s chunk_id (or text), and assert RuntimeError with the expected “chunk
identity” message; fail explicitly if build_review accepts it. Also add coverage
for _row_for_location rejecting ambiguous location matches, preserving the
advertised provenance guarantees.
In `@examples/prior-authorization-review-agent/tests/test_review.py`:
- Around line 1-360: Add failure-path tests covering the remaining fail-closed
gates: make _source_scope reject evidence with no matching fragment, make
_require_ranked_evidence reject missing or malformed reranked rows, and make
_chunks reject input with no usable chunks. Add a build_review test for the
not-overdue encounter case, asserting the documented failure behavior. Reuse
existing fixtures/helpers and assert the expected RuntimeError messages.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 45314b75-564f-486d-8637-593b0866fab3
⛔ Files ignored due to path filters (66)
.coderabbit.yamlis excluded by none and included by noneexamples/financial-filing-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/financial-filing-agent/fixtures/pathward-filing-packet.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/fixtures/pathward-q3-fy2023.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/financial-filing-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/maintenance-triage-agent/fixtures/STRUCTURED_OUTPUT_PROBE.mdis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/east-palestine-bearing-spread.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-request.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-response.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/maintenance-triage-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-cause.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-derailment.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-east-palestine.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-engineer.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-salem.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-sebring.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/prior-authorization-review-agent/fixtures/cms-l1851-insufficient-documentation.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/prior-authorization-review-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/prior-authorization-review-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-3.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-4.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-requirement-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-requirements.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-submission.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (28)
examples/README.mdexamples/financial-filing-agent/.env.exampleexamples/financial-filing-agent/.gitignoreexamples/financial-filing-agent/README.mdexamples/financial-filing-agent/config.yamlexamples/financial-filing-agent/financial_filing/__init__.pyexamples/financial-filing-agent/financial_filing/evaluate.pyexamples/financial-filing-agent/financial_filing/review.pyexamples/financial-filing-agent/pyproject.tomlexamples/financial-filing-agent/tests/test_review.pyexamples/maintenance-triage-agent/.env.exampleexamples/maintenance-triage-agent/.gitignoreexamples/maintenance-triage-agent/README.mdexamples/maintenance-triage-agent/config.yamlexamples/maintenance-triage-agent/maintenance_triage/__init__.pyexamples/maintenance-triage-agent/maintenance_triage/evaluate.pyexamples/maintenance-triage-agent/maintenance_triage/review.pyexamples/maintenance-triage-agent/pyproject.tomlexamples/maintenance-triage-agent/tests/test_review.pyexamples/prior-authorization-review-agent/.env.exampleexamples/prior-authorization-review-agent/.gitignoreexamples/prior-authorization-review-agent/README.mdexamples/prior-authorization-review-agent/config.yamlexamples/prior-authorization-review-agent/prior_authorization/__init__.pyexamples/prior-authorization-review-agent/prior_authorization/evaluate.pyexamples/prior-authorization-review-agent/prior_authorization/review.pyexamples/prior-authorization-review-agent/pyproject.tomlexamples/prior-authorization-review-agent/tests/test_review.py
Comment on lines
+136
to
+143
| def _require_entity_evidence(result: dict[str, Any]) -> None: | ||
| observed = _normalize_source_text(" ".join(str(entity.get("text", "")) for entity in result.get("entities", []))) | ||
| required_spans = ("45,096", "36,080", "$1.68") | ||
| missing = sorted(span for span in required_spans if span.casefold() not in observed) | ||
| if missing: | ||
| raise RuntimeError(f"GLiNER omitted required source spans: {missing}") | ||
| if "pathward financial" not in observed or "june 30, 2023" not in observed: | ||
| raise RuntimeError("GLiNER omitted the company or reporting-period source span") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
GLiNER evidence gate is missing the restated diluted EPS span.
required_spans checks original income, restated income, and original diluted EPS, but never checks for the restated diluted EPS ("$1.34"). The downstream GLiNER2 gate (_require_gliner2_figure_spans) does check it, so the overall pipeline still fails closed today, but this specific gate doesn't fully match the "fails closed if either entity model omits a required source span" guarantee documented in the README.
🔧 Proposed fix
- required_spans = ("45,096", "36,080", "$1.68")
+ required_spans = ("45,096", "36,080", "$1.68", "$1.34")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _require_entity_evidence(result: dict[str, Any]) -> None: | |
| observed = _normalize_source_text(" ".join(str(entity.get("text", "")) for entity in result.get("entities", []))) | |
| required_spans = ("45,096", "36,080", "$1.68") | |
| missing = sorted(span for span in required_spans if span.casefold() not in observed) | |
| if missing: | |
| raise RuntimeError(f"GLiNER omitted required source spans: {missing}") | |
| if "pathward financial" not in observed or "june 30, 2023" not in observed: | |
| raise RuntimeError("GLiNER omitted the company or reporting-period source span") | |
| def _require_entity_evidence(result: dict[str, Any]) -> None: | |
| observed = _normalize_source_text(" ".join(str(entity.get("text", "")) for entity in result.get("entities", []))) | |
| required_spans = ("45,096", "36,080", "$1.68", "$1.34") | |
| missing = sorted(span for span in required_spans if span.casefold() not in observed) | |
| if missing: | |
| raise RuntimeError(f"GLiNER omitted required source spans: {missing}") | |
| if "pathward financial" not in observed or "june 30, 2023" not in observed: | |
| raise RuntimeError("GLiNER omitted the company or reporting-period source span") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/financial-filing-agent/financial_filing/review.py` around lines 136
- 143, Update _require_entity_evidence so required_spans also includes the
restated diluted EPS source span "$1.34", preserving the existing normalization,
missing-span detection, and error behavior.
| @@ -0,0 +1 @@ | |||
| """Read-only maintenance triage over synthetic plant records.""" | |||
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Docstring contradicts the example's actual input.
This example reads the published NTSB SPC-24-06 page spread, not synthetic plant records. Given the example's provenance claims, the wording is misleading.
📝 Proposed fix
-"""Read-only maintenance triage over synthetic plant records."""
+"""Read-only maintenance triage over the NTSB East Palestine bearing-detector spread."""As per path instructions, "Flag client-side model-name translation and undocumented synthetic results".
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| """Read-only maintenance triage over synthetic plant records.""" | |
| """Read-only maintenance triage over the NTSB East Palestine bearing-detector spread.""" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/maintenance-triage-agent/maintenance_triage/__init__.py` at line 1,
Update the module docstring in maintenance_triage to describe that the example
performs read-only maintenance triage using the published NTSB SPC-24-06 page
spread, removing the inaccurate reference to synthetic plant records.
Source: Path instructions
Comment on lines
+452
to
+462
| "trend": { | ||
| "successive_increases_degrees_f": detector_deltas, | ||
| "total_increase_degrees_f": temperatures[-1] - temperatures[0], | ||
| "sebring_to_salem_minutes": salem_minute - sebring_minute, | ||
| "salem_to_east_palestine_minutes": 39, | ||
| }, | ||
| "engineer_action": "began slowing before 8:54 p.m.", | ||
| "derailment": { | ||
| "total_cars": 38, | ||
| "statement": "the hopper car and 37 others derailed as emergency braking activated", | ||
| }, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
salem_to_east_palestine_minutes: 39 and total_cars: 38 are asserted, not derived.
Every other trend number here is computed from validated mapped fields, but 39 is a literal and there is no east_palestine_time in MAPPED_FIELDS to validate it against (the 8:52 p.m. span is only checked indirectly via GLINER2_REQUIRED_SPANS["east_palestine"]). For an example whose claim is that all arithmetic is transparent and source-derived, map an east_palestine_time field and compute the delta the same way as sebring_to_salem_minutes.
total_cars: 38 has the same shape, though README lines 42-43 documents the 1 + 37 conversion; deriving it from derailment_statement would close the loop.
♻️ Sketch
+ east_palestine_minute = _minute_of_day(str(data["east_palestine_time"]))
...
- "salem_to_east_palestine_minutes": 39,
+ "salem_to_east_palestine_minutes": east_palestine_minute - salem_minute,Add "east_palestine_time" to MAPPED_FIELDS and map it in _map_exact_source_fields with _source_match(east_palestine_text, r"\b8:52\s+p\.m", "East Palestine time").
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/maintenance-triage-agent/maintenance_triage/review.py` around lines
452 - 462, Update the mapped-field and triage calculation flow to derive both
asserted values: add and validate east_palestine_time via _source_match in
_map_exact_source_fields, then compute salem_to_east_palestine_minutes from the
mapped times like sebring_to_salem_minutes; also derive derailment.total_cars
from the validated derailment_statement as 1 hopper car plus 37 others,
replacing the literal 38.
Comment on lines
+478
to
+480
| timeout = config["cluster"]["provision_timeout_s"] | ||
|
|
||
| with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Client timeout hardcodes 900 instead of using the configured value.
config["cluster"]["provision_timeout_s"] is read into timeout and forwarded to each call, but SIEClient(..., timeout_s=900) ignores it, so editing config.yaml only half-applies.
🔧 Proposed fix
- with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client:
+ with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=timeout) as client:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| timeout = config["cluster"]["provision_timeout_s"] | |
| with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: | |
| timeout = config["cluster"]["provision_timeout_s"] | |
| with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=timeout) as client: |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/maintenance-triage-agent/maintenance_triage/review.py` around lines
478 - 480, Update the SIEClient construction in the maintenance flow to pass the
already-read timeout variable from config["cluster"]["provision_timeout_s"]
instead of the hardcoded 900, while preserving the existing timeout forwarding
for subsequent calls.
Comment on lines
+62
to
+77
| runs/<run-id>/manifest.json endpoint, model IDs, source hash, latency | ||
| runs/<run-id>/raw/parse.json complete Docling response | ||
| runs/<run-id>/raw/retrieve.json embeddings and cosine ranking | ||
| runs/<run-id>/raw/rerank.json complete reranker response | ||
| runs/<run-id>/raw/entities.json combined GLiNER entity spans | ||
| runs/<run-id>/raw/gliner2-sebring.json GLiNER2 Sebring spans | ||
| runs/<run-id>/raw/gliner2-salem.json GLiNER2 Salem spans | ||
| runs/<run-id>/raw/gliner2-east-palestine.json GLiNER2 East Palestine spans | ||
| runs/<run-id>/raw/gliner2-cause.json GLiNER2 bearing-failure span | ||
| runs/<run-id>/raw/gliner2-engineer.json GLiNER2 engineer-action spans | ||
| runs/<run-id>/raw/gliner2-derailment.json GLiNER2 derailed-railcar spans | ||
| runs/<run-id>/raw/mapped.json validated exact-fragment mapping | ||
| runs/<run-id>/parsed.md parsed NTSB spread used downstream | ||
| runs/<run-id>/review.json detector trend and explicit boundary | ||
| runs/<run-id>/evaluation.json deterministic checks | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Evidence bundle omits the per-stage GLiNER entity artifacts.
run() also writes raw/entities-detector-{index}.json (review.py line 597) and raw/entities-outcome.json (review.py line 615). Since the example's core claim is a complete artifact bundle, list them.
📄 Proposed doc addition
runs/<run-id>/raw/rerank.json complete reranker response
+runs/<run-id>/raw/entities-detector-0.json GLiNER spans per detector paragraph
+runs/<run-id>/raw/entities-outcome.json GLiNER spans for the outcome paragraphs
runs/<run-id>/raw/entities.json combined GLiNER entity spansAs per path instructions, "Check that commands match the implementation".
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| runs/<run-id>/manifest.json endpoint, model IDs, source hash, latency | |
| runs/<run-id>/raw/parse.json complete Docling response | |
| runs/<run-id>/raw/retrieve.json embeddings and cosine ranking | |
| runs/<run-id>/raw/rerank.json complete reranker response | |
| runs/<run-id>/raw/entities.json combined GLiNER entity spans | |
| runs/<run-id>/raw/gliner2-sebring.json GLiNER2 Sebring spans | |
| runs/<run-id>/raw/gliner2-salem.json GLiNER2 Salem spans | |
| runs/<run-id>/raw/gliner2-east-palestine.json GLiNER2 East Palestine spans | |
| runs/<run-id>/raw/gliner2-cause.json GLiNER2 bearing-failure span | |
| runs/<run-id>/raw/gliner2-engineer.json GLiNER2 engineer-action spans | |
| runs/<run-id>/raw/gliner2-derailment.json GLiNER2 derailed-railcar spans | |
| runs/<run-id>/raw/mapped.json validated exact-fragment mapping | |
| runs/<run-id>/parsed.md parsed NTSB spread used downstream | |
| runs/<run-id>/review.json detector trend and explicit boundary | |
| runs/<run-id>/evaluation.json deterministic checks | |
| ``` | |
| runs/<run-id>/manifest.json endpoint, model IDs, source hash, latency | |
| runs/<run-id>/raw/parse.json complete Docling response | |
| runs/<run-id>/raw/retrieve.json embeddings and cosine ranking | |
| runs/<run-id>/raw/rerank.json complete reranker response | |
| runs/<run-id>/raw/entities-detector-0.json GLiNER spans per detector paragraph | |
| runs/<run-id>/raw/entities-outcome.json GLiNER spans for the outcome paragraphs | |
| runs/<run-id>/raw/entities.json combined GLiNER entity spans | |
| runs/<run-id>/raw/gliner2-sebring.json GLiNER2 Sebring spans | |
| runs/<run-id>/raw/gliner2-salem.json GLiNER2 Salem spans | |
| runs/<run-id>/raw/gliner2-east-palestine.json GLiNER2 East Palestine spans | |
| runs/<run-id>/raw/gliner2-cause.json GLiNER2 bearing-failure span | |
| runs/<run-id>/raw/gliner2-engineer.json GLiNER2 engineer-action spans | |
| runs/<run-id>/raw/gliner2-derailment.json GLiNER2 derailed-railcar spans | |
| runs/<run-id>/raw/mapped.json validated exact-fragment mapping | |
| runs/<run-id>/parsed.md parsed NTSB spread used downstream | |
| runs/<run-id>/review.json detector trend and explicit boundary | |
| runs/<run-id>/evaluation.json deterministic checks |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/maintenance-triage-agent/README.md` around lines 62 - 77, Update the
evidence-bundle artifact list in the README to include the per-stage GLiNER
outputs written by run(): raw/entities-detector-{index}.json and
raw/entities-outcome.json. Keep the existing artifact entries unchanged and
ensure the documented paths match the implementation.
Source: Path instructions
Comment on lines
+389
to
+415
| return { | ||
| "scope": "Reproduction of CMS's published L1851 insufficient-documentation example", | ||
| "route": "insufficient_documentation", | ||
| "headline": "The face-to-face encounter fell one month outside CMS's six-month window", | ||
| "hcpcs_code": code, | ||
| "required_face_to_face_within_months": required_months, | ||
| "documented_face_to_face_age_months": observed_months, | ||
| "overdue_by_months": overdue_by_months, | ||
| "submitted_documentation": { | ||
| "order": str(submission["submitted_order"]), | ||
| "medical_record": str(submission["submitted_medical_record"]), | ||
| "proof_of_delivery": str(submission["submitted_proof_of_delivery"]), | ||
| }, | ||
| "missing_documentation": ["face-to-face encounter within 6 months of proof of delivery"], | ||
| "review_conclusion": "insufficient documentation error", | ||
| "payment_action": "MAC recoups payment", | ||
| "source": { | ||
| "publisher": "Centers for Medicare & Medicaid Services", | ||
| "title": "Lower Limb Orthoses", | ||
| "url": SOURCE_URL, | ||
| "page_last_modified": "2026-02-11", | ||
| }, | ||
| "ranked_source_evidence": ranked, | ||
| "published_example": True, | ||
| "coverage_decision": None, | ||
| "medical_decision": None, | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Three "outcome" review fields are canned literals, not the extracted evidence — contradicts the "exact fragments" documentation.
build_review validates outcome["missing_documentation"], outcome["review_conclusion"], and outcome["payment_action"] for required substrings via _require_tokens, but then discards that text and emits fixed literal strings in the returned review instead — unlike submitted_documentation, which genuinely echoes the extracted text. The README states that "Ordinary code then maps the exact ranked CMS fragments to the review fields," which is not true for these three fields, and evaluate.py's corresponding checks then compare against the same hardcoded literals, so they pass tautologically whenever build_review succeeds rather than verifying the model's actual output content.
examples/prior-authorization-review-agent/prior_authorization/review.py#L389-L415: populatemissing_documentation,review_conclusion, andpayment_actionfromoutcome[...](the validated extracted text) instead of literal strings, e.g."review_conclusion": str(outcome["review_conclusion"]).examples/prior-authorization-review-agent/README.md#L26-L32: if the literals are intentionally kept as normalized/canonical output, revise "maps the exact ranked CMS fragments to the review fields" to clarify which fields are verbatim extracted text vs. canonicalized conclusions.examples/prior-authorization-review-agent/prior_authorization/evaluate.py#L50-L60: once the fields carry real extracted text, switch these checks to substring/token assertions (mirroring_require_tokens) instead of exact-equality against fixed strings, so they meaningfully verify model output.
📍 Affects 3 files
examples/prior-authorization-review-agent/prior_authorization/review.py#L389-L415(this comment)examples/prior-authorization-review-agent/README.md#L26-L32examples/prior-authorization-review-agent/prior_authorization/evaluate.py#L50-L60
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`
around lines 389 - 415, Update review.py lines 389-415 to populate
missing_documentation, review_conclusion, and payment_action from the validated
outcome fields rather than literals; update README.md lines 26-32 to accurately
describe verbatim versus canonical fields; update evaluate.py lines 50-60 to use
token/substring assertions matching _require_tokens instead of fixed-string
equality.
Source: Path instructions
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
examples/prior-authorization-review-agent/prior_authorization/review.py (1)
455-457: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueClient timeout duplicates the configured value as a literal.
timeoutis read fromprovision_timeout_son Line 455, then Line 457 hardcodestimeout_s=900, so editingconfig.yamlsilently leaves the socket timeout behind.♻️ Proposed refactor
- with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: + with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=timeout) as client:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/prior_authorization/review.py` around lines 455 - 457, Use the existing provision timeout value from config["cluster"]["provision_timeout_s"] when constructing SIEClient in the review flow, replacing the hardcoded timeout_s=900 while preserving the surrounding client setup.examples/prior-authorization-review-agent/README.md (1)
70-82: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueEvidence bundle listing omits the per-row GLiNER artifacts.
review.pyLine 598 writesraw/entities-requirement-<n>.jsonandraw/entities-case-<n>.jsonper ranked row, which aren't documented here. Readers comparing the run directory against this table will see unlisted files.As per path instructions, "Review runnable source, tests, configuration, and documentation. Check that commands match the implementation".
📝 Proposed doc addition
runs/<run-id>/raw/entities.json combined GLiNER entity spans +runs/<run-id>/raw/entities-requirement-*.json per-passage GLiNER requirement spans +runs/<run-id>/raw/entities-case-*.json per-passage GLiNER case spans🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/README.md` around lines 70 - 82, Update the evidence bundle listing in the README to include the per-ranked-row artifacts written by review.py, specifically raw/entities-requirement-<n>.json and raw/entities-case-<n>.json. Place them with the other raw GLiNER/entity outputs and describe their per-row purpose accurately.Source: Path instructions
examples/prior-authorization-review-agent/tests/test_review.py (1)
278-305: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNo test exercises the evaluator's failing path.
Every assertion here drives
evaluate_review/evaluate_runtoTrue; nothing covers a review that fails a check, so thefailrendering and theSystemExit(1)inmainare untested.As per path instructions, "Check that commands match the implementation and that tests cover failure paths".
💚 Proposed test
def test_evaluator_fails_on_a_mutated_review(tmp_path: Path) -> None: review = build_valid_review() review["overdue_by_months"] = 0 (tmp_path / "review.json").write_text(json.dumps(review), encoding="utf-8") assert evaluate_run(tmp_path) is False evaluation = json.loads((tmp_path / "evaluation.json").read_text(encoding="utf-8")) assert evaluation["passed"] is False assert any(check["name"] == "one-month-gap" and not check["passed"] for check in evaluation["checks"])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/tests/test_review.py` around lines 278 - 305, Add a failure-path test alongside the existing evaluator tests, using a mutated review with overdue_by_months set to 0. Assert evaluate_run returns False, evaluation.json is written with passed set to False, and the one-month-gap check is recorded as failed; also cover main’s SystemExit(1) behavior if that entry point is exposed.Source: Path instructions
examples/prior-authorization-review-agent/prior_authorization/evaluate.py (1)
105-106: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUnhelpful traceback when the run directory is wrong.
eval-paon a missing or incomplete run dir raises a bareFileNotFoundError. A guarded message keeps the example approachable.♻️ Proposed refactor
def evaluate_run(run_dir: Path) -> bool: - review = json.loads((run_dir / "review.json").read_text(encoding="utf-8")) + review_path = run_dir / "review.json" + if not review_path.exists(): + raise SystemExit(f"No review.json in {run_dir}. Run `uv run review-pa` first.") + review = json.loads(review_path.read_text(encoding="utf-8"))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/prior_authorization/evaluate.py` around lines 105 - 106, Update evaluate_run to guard loading review.json and raise a clear, user-facing error when the run directory is missing or incomplete, instead of exposing a bare FileNotFoundError. Preserve normal JSON loading for valid run directories and include the expected review.json location in the message.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/financial-filing-agent/README.md`:
- Around line 60-75: Update the “Evidence bundle” artifact list in the README to
document the per-stage files written by review.py:
raw/entities-original-10q.json, raw/entities-restated-10ka.json, and
raw/entities-status.json, alongside the existing combined raw/entities.json
entry.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 555-563: Update the evidence terms passed to _select_evidence_rows
in the outcome_rows selection so the face-to-face documentation phrase uses an
apostrophe-free substring that matches both straight and typographic
apostrophes. Leave the other outcome terms and selection behavior unchanged.
- Around line 448-457: Update run to handle FileExistsError from raw_dir.mkdir
when the fixed run_id already exists, and provide a clear user-facing message
instructing the user to choose a new run ID or delete the existing run
directory. Preserve the current behavior for new run IDs and keep the directory
non-overwriting.
---
Nitpick comments:
In `@examples/prior-authorization-review-agent/prior_authorization/evaluate.py`:
- Around line 105-106: Update evaluate_run to guard loading review.json and
raise a clear, user-facing error when the run directory is missing or
incomplete, instead of exposing a bare FileNotFoundError. Preserve normal JSON
loading for valid run directories and include the expected review.json location
in the message.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 455-457: Use the existing provision timeout value from
config["cluster"]["provision_timeout_s"] when constructing SIEClient in the
review flow, replacing the hardcoded timeout_s=900 while preserving the
surrounding client setup.
In `@examples/prior-authorization-review-agent/README.md`:
- Around line 70-82: Update the evidence bundle listing in the README to include
the per-ranked-row artifacts written by review.py, specifically
raw/entities-requirement-<n>.json and raw/entities-case-<n>.json. Place them
with the other raw GLiNER/entity outputs and describe their per-row purpose
accurately.
In `@examples/prior-authorization-review-agent/tests/test_review.py`:
- Around line 278-305: Add a failure-path test alongside the existing evaluator
tests, using a mutated review with overdue_by_months set to 0. Assert
evaluate_run returns False, evaluation.json is written with passed set to False,
and the one-month-gap check is recorded as failed; also cover main’s
SystemExit(1) behavior if that entry point is exposed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 998e61b4-151c-4ea8-ab11-12532d8edcef
⛔ Files ignored due to path filters (66)
.coderabbit.yamlis excluded by none and included by noneexamples/financial-filing-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/financial-filing-agent/fixtures/pathward-filing-packet.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/fixtures/pathward-q3-fy2023.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/financial-filing-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/maintenance-triage-agent/fixtures/STRUCTURED_OUTPUT_PROBE.mdis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/east-palestine-bearing-spread.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-request.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-response.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/maintenance-triage-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-cause.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-derailment.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-east-palestine.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-engineer.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-salem.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-sebring.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/prior-authorization-review-agent/fixtures/cms-l1851-insufficient-documentation.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/prior-authorization-review-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/prior-authorization-review-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-3.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-4.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-requirement-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-requirements.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-submission.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (28)
examples/README.mdexamples/financial-filing-agent/.env.exampleexamples/financial-filing-agent/.gitignoreexamples/financial-filing-agent/README.mdexamples/financial-filing-agent/config.yamlexamples/financial-filing-agent/financial_filing/__init__.pyexamples/financial-filing-agent/financial_filing/evaluate.pyexamples/financial-filing-agent/financial_filing/review.pyexamples/financial-filing-agent/pyproject.tomlexamples/financial-filing-agent/tests/test_review.pyexamples/maintenance-triage-agent/.env.exampleexamples/maintenance-triage-agent/.gitignoreexamples/maintenance-triage-agent/README.mdexamples/maintenance-triage-agent/config.yamlexamples/maintenance-triage-agent/maintenance_triage/__init__.pyexamples/maintenance-triage-agent/maintenance_triage/evaluate.pyexamples/maintenance-triage-agent/maintenance_triage/review.pyexamples/maintenance-triage-agent/pyproject.tomlexamples/maintenance-triage-agent/tests/test_review.pyexamples/prior-authorization-review-agent/.env.exampleexamples/prior-authorization-review-agent/.gitignoreexamples/prior-authorization-review-agent/README.mdexamples/prior-authorization-review-agent/config.yamlexamples/prior-authorization-review-agent/prior_authorization/__init__.pyexamples/prior-authorization-review-agent/prior_authorization/evaluate.pyexamples/prior-authorization-review-agent/prior_authorization/review.pyexamples/prior-authorization-review-agent/pyproject.tomlexamples/prior-authorization-review-agent/tests/test_review.py
Comment on lines
+60
to
+75
| ## Evidence bundle | ||
|
|
||
| Every run writes: | ||
|
|
||
| ```text | ||
| runs/<run-id>/manifest.json endpoint, model IDs, fixture hashes, latency | ||
| runs/<run-id>/raw/parse.json complete Docling response | ||
| runs/<run-id>/raw/retrieve.json embeddings and cosine ranking | ||
| runs/<run-id>/raw/rerank.json complete reranker response | ||
| runs/<run-id>/raw/entities.json combined entity spans | ||
| runs/<run-id>/raw/gliner2-*.json raw GLiNER2 source-span responses | ||
| runs/<run-id>/raw/mapped.json validated record mapped from Docling table coordinates | ||
| runs/<run-id>/parsed.md parsed packet used downstream | ||
| runs/<run-id>/review.json source-versioned result and calculated delta | ||
| runs/<run-id>/evaluation.json deterministic checks | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Artifact bundle list omits per-stage entity extraction files.
review.py also writes raw/entities-original-10q.json, raw/entities-restated-10ka.json, and raw/entities-status.json (per-stage GLiNER responses) in addition to the combined raw/entities.json, but only the combined file is documented here.
📝 Proposed doc fix
-runs/<run-id>/raw/entities.json combined entity spans
+runs/<run-id>/raw/entities.json combined entity spans
+runs/<run-id>/raw/entities-*.json per-stage GLiNER responses (original 10-Q, restated 10-K/A, filing status)As per path instructions, examples documentation should be checked so "commands match the implementation."
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ## Evidence bundle | |
| Every run writes: | |
| ```text | |
| runs/<run-id>/manifest.json endpoint, model IDs, fixture hashes, latency | |
| runs/<run-id>/raw/parse.json complete Docling response | |
| runs/<run-id>/raw/retrieve.json embeddings and cosine ranking | |
| runs/<run-id>/raw/rerank.json complete reranker response | |
| runs/<run-id>/raw/entities.json combined entity spans | |
| runs/<run-id>/raw/gliner2-*.json raw GLiNER2 source-span responses | |
| runs/<run-id>/raw/mapped.json validated record mapped from Docling table coordinates | |
| runs/<run-id>/parsed.md parsed packet used downstream | |
| runs/<run-id>/review.json source-versioned result and calculated delta | |
| runs/<run-id>/evaluation.json deterministic checks | |
| ``` | |
| ## Evidence bundle | |
| Every run writes: | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/financial-filing-agent/README.md` around lines 60 - 75, Update the
“Evidence bundle” artifact list in the README to document the per-stage files
written by review.py: raw/entities-original-10q.json,
raw/entities-restated-10ka.json, and raw/entities-status.json, alongside the
existing combined raw/entities.json entry.
Source: Path instructions
Comment on lines
+448
to
+457
| def run(run_id: str) -> Path: | ||
| config = load_config() | ||
| parse_model = str(config["models"]["parse"]) | ||
| run_dir = RUNS_DIR / run_id | ||
| raw_dir = run_dir / "raw" | ||
| raw_dir.mkdir(parents=True, exist_ok=False) | ||
| calls: list[dict[str, Any]] = [] | ||
| timeout = config["cluster"]["provision_timeout_s"] | ||
|
|
||
| with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Re-running the documented quickstart command fails with a raw FileExistsError.
The README instructs uv run review-pa --run-id local, a fixed id, but exist_ok=False makes the second run abort with an unhandled traceback rather than a clear message. Catch it and explain the choice (pick a new id or delete the directory).
As per path instructions, "Check that commands match the implementation".
🛠️ Proposed fix
run_dir = RUNS_DIR / run_id
raw_dir = run_dir / "raw"
- raw_dir.mkdir(parents=True, exist_ok=False)
+ if run_dir.exists():
+ raise SystemExit(f"Run directory already exists: {run_dir}. Pass a new --run-id or remove it first.")
+ raw_dir.mkdir(parents=True, exist_ok=False)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def run(run_id: str) -> Path: | |
| config = load_config() | |
| parse_model = str(config["models"]["parse"]) | |
| run_dir = RUNS_DIR / run_id | |
| raw_dir = run_dir / "raw" | |
| raw_dir.mkdir(parents=True, exist_ok=False) | |
| calls: list[dict[str, Any]] = [] | |
| timeout = config["cluster"]["provision_timeout_s"] | |
| with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: | |
| run_dir = RUNS_DIR / run_id | |
| raw_dir = run_dir / "raw" | |
| if run_dir.exists(): | |
| raise SystemExit(f"Run directory already exists: {run_dir}. Pass a new --run-id or remove it first.") | |
| raw_dir.mkdir(parents=True, exist_ok=False) | |
| calls: list[dict[str, Any]] = [] | |
| timeout = config["cluster"]["provision_timeout_s"] | |
| with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`
around lines 448 - 457, Update run to handle FileExistsError from raw_dir.mkdir
when the fixed run_id already exists, and provide a clear user-facing message
instructing the user to choose a new run ID or delete the existing run
directory. Preserve the current behavior for new run IDs and keep the directory
non-overwriting.
Source: Path instructions
Comment on lines
+555
to
+563
| outcome_rows = _select_evidence_rows( | ||
| ranked, | ||
| ( | ||
| "didn't document the face-to-face", | ||
| "insufficient documentation error", | ||
| "mac recoups", | ||
| ), | ||
| "outcome", | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Apostrophe mismatch: "didn't document the face-to-face" never matches the CMS fixture.
The fixture uses the typographic apostrophe — test_review.py Lines 179 and 263 assert "The doctor didn’t document the face-to-face encounter…". _select_evidence_rows compares raw casefolded text with no apostrophe normalization (unlike _normalized_source_text), so this term is dead. The stage still passes because the other two any(...) terms match, but the missing-documentation passage is silently dropped from the GLiNER case inputs.
Match on the apostrophe-free substring instead.
🐛 Proposed fix
outcome_rows = _select_evidence_rows(
ranked,
(
- "didn't document the face-to-face",
+ "document the face-to-face encounter within 6 months",
"insufficient documentation error",
"mac recoups",
),
"outcome",
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| outcome_rows = _select_evidence_rows( | |
| ranked, | |
| ( | |
| "didn't document the face-to-face", | |
| "insufficient documentation error", | |
| "mac recoups", | |
| ), | |
| "outcome", | |
| ) | |
| outcome_rows = _select_evidence_rows( | |
| ranked, | |
| ( | |
| "document the face-to-face encounter within 6 months", | |
| "insufficient documentation error", | |
| "mac recoups", | |
| ), | |
| "outcome", | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`
around lines 555 - 563, Update the evidence terms passed to
_select_evidence_rows in the outcome_rows selection so the face-to-face
documentation phrase uses an apostrophe-free substring that matches both
straight and typographic apostrophes. Leave the other outcome terms and
selection behavior unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
4b5ea9cEvidence and provenance
docling-project/docling. Manifests show the internaldoclingID used by the direct local server during verification.Verification
Live NVIDIA L4 runs:
2026-07-25T02:28:01.848776+00:002026-07-25T02:35:39.667778+00:002026-07-25T02:35:45.228957+00:00Local package checks:
Summary by CodeRabbit