You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Textract grafting strips visible text on some PDFs #336
If we track processing jobs in the database, we could fall back to the result of the last successful job (ex. OCR job fails, fall back to upload job result).
We'll need to report these errors more clearly to users, so they know what went wrong and what steps they need to take next to get the result they want.
Right now, we show the error state but don't do any user-friendly fixes for the error message.
It breaks on digitally created PDFs, the kind exported from Word, Excel or a reporting system, or filled-in forms like the TIF report originally noted in the issue where the visible text is real text objects.
Cause: graft_ocr_in_pdf in documents/processing/info_and_image/main.py applies a full-page text redaction to every page before overlaying the Textract layer. This is fine when existing text is invisible, but it deletes real text. The Django graft path guards against this with _check_visible_text; the processing path doesn’t. The original PDF is overwritten, so affected documents need re-uploading.
I will add an environment variable in Lambda to turn grafting off completely and also add this _check_visible_text path that the Django grafting path does. That will allow me to see if this fixes this on staging.
Checking if the page had visible text and skipping those pages did fix this issue. It means those pages with visible text don't get the text grafted back in, but it at least doesn't destroy the PDF. It is a path to remedy this before we build out time to properly address this.
The longer term fix for this I think is to treat the pages differently depending on if they have underlying text or not.
No visible text: keep the current behavior. Redact the hidden layer and overlay Textract. This already works.
Visible text: Don’t redact at all. Overlay only the Textract words that don’t overlap existing visible text. The real text stays untouched, and OCR fills in what the PDF’s text layer lacks, such as text inside a pasted screenshot, a scanned signature page or a chart image. This requires some level of heuristics. We need a deduplication filter that drops Textract words that overlap with existing visible text so that we don't ruin PDFs. This requires a lot of testing, so for now we must just disable grafting on pages with visible text.
changed the title [-]Force OCR with Textract strips visible text on some PDFs[/-][+]Textract grafting strips visible text on some PDFs[/+]on Nov 3, 2026
Activity
allanlasser commentedon Oct 6, 2025
If we track processing jobs in the database, we could fall back to the result of the last successful job (ex. OCR job fails, fall back to upload job result).
We'll need to report these errors more clearly to users, so they know what went wrong and what steps they need to take next to get the result they want.
Right now, we show the error state but don't do any user-friendly fixes for the error message.
eyeseast commentedon Sep 15, 2026
Ran into this in a demo: https://www.documentcloud.org/documents/28623996-pressure-cooking-demo/
We should disable Textract in the UI until we can solve this. We can keep testing using the API.
duckduckgrayduck commentedon Nov 3, 2026
I spent some time debugging this as the issue intersects with a push to disable grafting for now.
This issue seems to occur on digitally created PDFs such as this one:
https://www.documentcloud.org/documents/28865374-requests20to20follow20up20-20sheet1/
This is a document that was originally an excel spreadsheet that I exported as a PDF in excel.
This is what it is supposed to look like, before Textract -> Force OCR breaks it:
https://www.documentcloud.org/documents/28865400-0001-requests20to20follow20up20-20sheet1/
I created a project on production that I will replicate on staging that is comprised of various edge cases:
https://www.documentcloud.org/projects/226005-test-ocr/
These kinds of documents seem to work fine:
https://www.documentcloud.org/documents/28865371-damaged-yanukovych-leaks/
https://www.documentcloud.org/documents/28865368-gpo-j6-doc-ctrl0000917052-00001-gcv/?mode=document
https://www.documentcloud.org/documents/28865370-acknowledgement-letter/
It breaks on digitally created PDFs, the kind exported from Word, Excel or a reporting system, or filled-in forms like the TIF report originally noted in the issue where the visible text is real text objects.
[-]Error in grafting ocr'd text back into PDF[/-][+]Force OCR with Textract strips visible text on some PDFs[/+]duckduckgrayduck commentedon Nov 3, 2026
I had Claude go look at the processing code and it flagged this:
documentcloud/documentcloud/documents/processing/info_and_image/main.py
Line 571 in 39e07b1
Cause: graft_ocr_in_pdf in documents/processing/info_and_image/main.py applies a full-page text redaction to every page before overlaying the Textract layer. This is fine when existing text is invisible, but it deletes real text. The Django graft path guards against this with _check_visible_text; the processing path doesn’t. The original PDF is overwritten, so affected documents need re-uploading.
duckduckgrayduck commentedon Nov 3, 2026
I will add an environment variable in Lambda to turn grafting off completely and also add this _check_visible_text path that the Django grafting path does. That will allow me to see if this fixes this on staging.
duckduckgrayduck commentedon Nov 3, 2026
Checking if the page had visible text and skipping those pages did fix this issue. It means those pages with visible text don't get the text grafted back in, but it at least doesn't destroy the PDF. It is a path to remedy this before we build out time to properly address this.
duckduckgrayduck commentedon Nov 3, 2026
The longer term fix for this I think is to treat the pages differently depending on if they have underlying text or not.
No visible text: keep the current behavior. Redact the hidden layer and overlay Textract. This already works.
Visible text: Don’t redact at all. Overlay only the Textract words that don’t overlap existing visible text. The real text stays untouched, and OCR fills in what the PDF’s text layer lacks, such as text inside a pasted screenshot, a scanned signature page or a chart image. This requires some level of heuristics. We need a deduplication filter that drops Textract words that overlap with existing visible text so that we don't ruin PDFs. This requires a lot of testing, so for now we must just disable grafting on pages with visible text.
I've captured this in this issue: #531
For now, the data destruction concern from this issue is closed in pending PR #529
[-]Force OCR with Textract strips visible text on some PDFs[/-][+]Textract grafting strips visible text on some PDFs[/+]