feat: Add self-verifying document extractor kit - #224
Closed
Krishhiv wants to merge 2 commits into
Closed
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (6)
WalkthroughChangesSelf-Verifying Document Extraction
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
🎉 All checks passed! This contribution follows the AgentKit structure. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/self-verifying-extractor/apps/.gitignore`:
- Around line 19-22: Update the environment-file ignore patterns in the
gitignore configuration to match all dotenv variants, including production and
development files, while preserving the existing exclusions for .env and
.env.local.
In `@kits/self-verifying-extractor/apps/actions/orchestrate.ts`:
- Line 22: Replace the config import in the actions orchestrate module so it
uses the parent kit’s ../../lamatic.config instead of the local
../orchestrate.js bridge, while preserving the existing config usage.
In `@kits/self-verifying-extractor/apps/app/api/parse-pdf/route.ts`:
- Around line 71-76: Update the blob cleanup in the finally block to retain
best-effort response behavior while logging any error from del(blobUrl). Replace
the silent catch with an error handler that records the blob URL and deletion
failure details for operator investigation.
- Around line 43-46: Update the blob upload options in the parse-PDF route to
prevent predictable public URLs: prefer private access with a short-lived signed
URL if supported by the configured `@vercel/blob` setup; otherwise retain public
access but enable addRandomSuffix so uploaded document URLs are unguessable
during parsing.
In `@kits/self-verifying-extractor/apps/app/page.tsx`:
- Line 64: Rename the state pair in the component from document/setDocument to
documentText/setDocumentText, and update every corresponding reference
throughout the component, including the listed handlers and JSX usages. Preserve
the existing state behavior while avoiding shadowing the global DOM document.
In `@kits/self-verifying-extractor/apps/lib/environment.ts`:
- Around line 60-62: Update the endpoint validation in the environment
configuration flow to reject http:// URLs outside development, while continuing
to allow both http:// and https:// when NODE_ENV is development. Keep https://
valid in all environments and preserve the existing ConfigurationError for
invalid protocols.
In `@kits/self-verifying-extractor/apps/lib/pdf.ts`:
- Around line 54-61: Update safeBlobName so filenames whose sanitized value is
empty or begins with a dot use a non-hidden default base name before appending
the .pdf extension. Preserve the existing sanitization and timestamped
self-verifying-extractor path for normal filenames.
In `@kits/self-verifying-extractor/apps/orchestrate.js`:
- Around line 1-56: Convert the orchestrator config bridge from JavaScript to
TypeScript by renaming orchestrate.js to orchestrate.ts, then update the import
in apps/actions/orchestrate.ts to reference the TypeScript module through the
existing .js runtime import convention.
In `@kits/self-verifying-extractor/apps/package.json`:
- Around line 22-42: Update the dependencies for the Next.js app by adding
react-hook-form and zod, plus the shadcn/ui support packages used by the app
such as `@radix-ui/`* components, class-variance-authority, clsx, and
tailwind-merge. If the app contains no form-based inputs, retain only the
shadcn/ui dependencies and confirm that react-hook-form and zod are unnecessary.
- Around line 27-29: Update the dependencies in the app package manifest to
comply with the kit version policy: use a supported Next.js 14–15 release and
React 18, and align the corresponding React types, Next.js types, and ESLint
configuration packages with those versions. Do not retain the current Next.js 16
and React 19 combination unless the policy is explicitly updated instead.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 33bb95dd-1c35-43ed-bad7-75a9e9374ee1
⛔ Files ignored due to path filters (2)
kits/self-verifying-extractor/apps/package-lock.jsonis excluded by!**/package-lock.jsonkits/self-verifying-extractor/assets/sample-invoice.pdfis excluded by!**/*.pdf
📒 Files selected for processing (44)
kits/self-verifying-extractor/.env.examplekits/self-verifying-extractor/.gitignorekits/self-verifying-extractor/README.mdkits/self-verifying-extractor/agent.mdkits/self-verifying-extractor/apps/.env.examplekits/self-verifying-extractor/apps/.gitignorekits/self-verifying-extractor/apps/.npmrckits/self-verifying-extractor/apps/README.mdkits/self-verifying-extractor/apps/actions/orchestrate.tskits/self-verifying-extractor/apps/app/api/parse-pdf/route.tskits/self-verifying-extractor/apps/app/globals.csskits/self-verifying-extractor/apps/app/layout.tsxkits/self-verifying-extractor/apps/app/page.tsxkits/self-verifying-extractor/apps/eslint.config.mjskits/self-verifying-extractor/apps/lib/environment.tskits/self-verifying-extractor/apps/lib/lamatic-client.tskits/self-verifying-extractor/apps/lib/pdf.tskits/self-verifying-extractor/apps/lib/pipeline.tskits/self-verifying-extractor/apps/next.config.mjskits/self-verifying-extractor/apps/orchestrate.jskits/self-verifying-extractor/apps/package.jsonkits/self-verifying-extractor/apps/postcss.config.mjskits/self-verifying-extractor/apps/tests/environment.test.tskits/self-verifying-extractor/apps/tests/pdf.test.tskits/self-verifying-extractor/apps/tests/pipeline.test.tskits/self-verifying-extractor/apps/tsconfig.jsonkits/self-verifying-extractor/assets/sample-financial-snippet.txtkits/self-verifying-extractor/assets/sample-invoice.txtkits/self-verifying-extractor/constitutions/default.mdkits/self-verifying-extractor/flows/README.mdkits/self-verifying-extractor/flows/extract.tskits/self-verifying-extractor/flows/parse-pdf.tskits/self-verifying-extractor/flows/report.tskits/self-verifying-extractor/flows/verify.tskits/self-verifying-extractor/lamatic.config.tskits/self-verifying-extractor/model-configs/extract_extract-fields.tskits/self-verifying-extractor/model-configs/verify_verify-fields.tskits/self-verifying-extractor/prompts/extract_extract-fields_system.mdkits/self-verifying-extractor/prompts/extract_extract-fields_user.mdkits/self-verifying-extractor/prompts/verify_verify-fields_system.mdkits/self-verifying-extractor/prompts/verify_verify-fields_user.mdkits/self-verifying-extractor/scripts/extract_parse-json.tskits/self-verifying-extractor/scripts/parse-pdf_collate.tskits/self-verifying-extractor/scripts/report_route.ts
Author
|
Thanks, CodeRabbit, addressed 6, declining 4 with reasons: Applied: expanded .gitignore env patterns; log blob del() failures; addRandomSuffix: true for unguessable temp URLs; renamed document→documentText to avoid shadowing the DOM global; require https:// in production; safeBlobName fallback for empty/dotfile names (+test). Declining: Import ../../lamatic.config instead of ../orchestrate.js — these are different modules. orchestrate.js is the runtime flow-config bridge (config.flows/config.api); lamatic.config.ts is kit metadata (name/type/steps). Swapping breaks the app, and this matches the reference kits (content-generation, deep-search). |
Contributor
|
Hi @Krishhiv! 👋 Before this PR can be reviewed by maintainers, please resolve all comments and requested changes from the CodeRabbit automated review. Steps to follow:
This helps keep the review process efficient for everyone. Thank you! 🙏 |
Contributor
|
/validate |
Contributor
|
📡 Running Studio validation — results will appear here shortly. |
Contributor
Studio Runtime Validation (Phase 2)❌ Studio validation failed. The kit was rejected by Lamatic Studio. Errorsself-verifying-extractor
Please fix the errors above and push a new commit to re-run validation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A Kit that pulls the key details out of everyday documents — invoices, bills, receipts, contracts — then independently re-verifies each field against the source text and flags anything it can't prove. Hence, it never quietly hands you the wrong due date or amount.
Ordinary extraction tools (including the existing
invoice-summariser,resume-parser, anddocument-parsingkits) do a single confident pass and assert whatever they read. This kit treats extraction as two stages: pull the data, then prove each value against the original text before asserting it. Anything that it can't ground to an exact span is flagged for review instead of being guessed.The problem it solves
A single misread number in a document has real consequences — a transposed digit in a due date costs a late fee; a total that was never actually stated triggers a wrong payment. LLM extractors have no notion of "I'm not sure about this one." This kit adds exactly that.
How it works
extractdocument_type,vendor_or_sender,total_amount,due_date,account_or_invoice_number,key_terms[]). Deliberately lean.verifysupported/ambiguous/unsupported+ confidence +source_quote.key_termsis split, and each item is grounded independently.reportWhy a separate verify flow? Running verification as its own reasoning pass — with the document and the extraction as inputs — lets it genuinely disagree with the extractor. And because an LLM can still return an overconfident verdict or a reconstructed quote, a deterministic code gate has the final say: confidence is advisory; exact evidence is mandatory.
What's included
type: "kit") with a runnable Next.js app.extract,verify,report) + 1 optional (parse-pdf).@referenced.flows/README.md— exact Studio recreation steps + smoke tests.parse-pdfflow + upload route that validates (extension/MIME/size/%PDF-signature), stores in short-lived Vercel Blob, parses, and deletes the blob in afinally. Verified fields getsource_page. Text-based PDFs only.Testing & quality
npm run checkpasses: 32 unit tests (evidence gate,key_termssplitting, not-found routing, malformed JSON, flow-error handling, PDF validation, page attribution), ESLint clean,tsc --noEmitclean,next buildsucceeds. Node ≥ 20.9,lamaticSDK pinned.Notes for reviewers
.env.example. PDF is optional; the app runs on pasted text withoutDOC_PARSE_PDF_FLOW/BLOB_READ_WRITE_TOKEN.key_termsverification is deterministic exact-presence by design — the app owns that check rather than deferring to the model's fuzzy "is this a key term" judgment.parse-pdfrunsextractFromFileNodewith Join Pages on for Lamatic deploy compatibility → page attribution is exact for single-page docs and collapses to p.1 for multi-page (documented in the flow + README).1. Select Contribution Type
2. General Requirements
3. File Structure
lamatic.config.ts(current format;config.jsonis deprecated per CLAUDE.md)flows/<name>.tsfiles (current format) — the oldflows/<name>/config.json + inputs.json + meta.jsonlayout no longer applies.env.examplewith placeholder values only (root + apps/).tsfiles represent those graphs4. Validation
npm install && npm run devworks locally (UI runs);npm run check(lint + typecheck + 32 tests + build) is greenfeat:prefixkits/self-verifying-extractor/README.mdkits/self-verifying-extractor/agent.mdkits/self-verifying-extractor/constitutions/default.mdkits/self-verifying-extractor/flows/README.mdkits/self-verifying-extractor/.env.examplekits/self-verifying-extractor/.gitignorekits/self-verifying-extractor/apps/.env.examplekits/self-verifying-extractor/apps/.gitignorekits/self-verifying-extractor/apps/.npmrckits/self-verifying-extractor/lamatic.config.tskits/self-verifying-extractor/flows/extract.ts(Trigger:graphqlNode; Nodes:triggerNode,dynamicNode; Edges:defaultEdge,responseEdge)kits/self-verifying-extractor/flows/verify.ts(Trigger:graphqlNode; Nodes:triggerNode,dynamicNode; Edges:defaultEdge,responseEdge)kits/self-verifying-extractor/flows/report.ts(Trigger:graphqlNode; Nodes:triggerNode,dynamicNode; Edges:defaultEdge,responseEdge)kits/self-verifying-extractor/flows/parse-pdf.ts(Trigger:graphqlNode; Nodes:triggerNode,dynamicNode; Edges:defaultEdge,responseEdge)flow.jsonwas found in the repo; node types were taken directly fromflows/*.ts.kits/self-verifying-extractor/prompts/extract_extract-fields_system.mdkits/self-verifying-extractor/prompts/extract_extract-fields_user.mdkits/self-verifying-extractor/prompts/verify_verify-fields_system.mdkits/self-verifying-extractor/prompts/verify_verify-fields_user.mdkits/self-verifying-extractor/model-configs/extract_extract-fields.tskits/self-verifying-extractor/model-configs/verify_verify-fields.tskits/self-verifying-extractor/scripts/extract_parse-json.tskits/self-verifying-extractor/scripts/parse-pdf_collate.tskits/self-verifying-extractor/scripts/report_route.tskits/self-verifying-extractor/assets/sample-invoice.txtkits/self-verifying-extractor/assets/sample-invoice.pdfkits/self-verifying-extractor/assets/sample-financial-snippet.txtkits/self-verifying-extractor/apps/README.mdkits/self-verifying-extractor/apps/package.jsonkits/self-verifying-extractor/apps/tsconfig.jsonkits/self-verifying-extractor/apps/next.config.mjskits/self-verifying-extractor/apps/eslint.config.mjskits/self-verifying-extractor/apps/postcss.config.mjskits/self-verifying-extractor/apps/app/layout.tsxkits/self-verifying-extractor/apps/app/globals.csskits/self-verifying-extractor/apps/app/page.tsxkits/self-verifying-extractor/apps/app/api/parse-pdf/route.tskits/self-verifying-extractor/apps/actions/orchestrate.tskits/self-verifying-extractor/apps/orchestrate.jskits/self-verifying-extractor/apps/lib/environment.ts(fail-closed env validation)kits/self-verifying-extractor/apps/lib/lamatic-client.ts(cached client)kits/self-verifying-extractor/apps/lib/pdf.ts(PDF validation + safe blob naming)kits/self-verifying-extractor/apps/lib/pipeline.ts(deterministic extraction verification, evidence grounding, routing, and report consistency)kits/self-verifying-extractor/apps/tests/environment.test.tskits/self-verifying-extractor/apps/tests/pdf.test.tskits/self-verifying-extractor/apps/tests/pipeline.test.tstriggerNode (graphqlNode)→dynamicNode (LLM extract fields)→dynamicNode (parse JSON)→responseEdgewith{ extraction }.triggerNode→dynamicNode (LLM verify fields with exact-span grounding)→responseEdgewith{ verifications }.triggerNode→dynamicNode (deterministic route & build report)→responseEdgewithverified / needs_review / not_found + report + summary.triggerNode→dynamicNode (extractFromPDF with joinPages)→dynamicNode (collate pages into--- Page N ---markers)→responseEdgewith{ text, page_count }.