Skip to content

Turn off grafting and partially address the Textract overwrite problem - #529

Merged
duckduckgrayduck merged 3 commits into
masterfrom
turn_off_grafting
Nov 5, 2026
Merged

duckduckgrayduck merged 3 commits into
masterfrom
turn_off_grafting

Conversation

@duckduckgrayduck

@duckduckgrayduck duckduckgrayduck commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Adds a DISABLE_GRAFTING environment variable (default False )in the Django app so grafting can be turned off so that the memory issues in the workers can be addressed. This path runs when someone updates a document’s page text through the API with word positions, and grafts them into the PDF when the pages have no visible text layer. Position JSON is still always written. The flag check now comes before the PDF is downloaded and opened, so a disabled run skips that work entirely.

  • Adds a parameter in lambda (AWS SSM) called DISABLE_TEXTRACT_GRAFTING to disable the textract grafting branch entirely, which can be useful if we catch issues on production later. This code path is called when someone runs Force OCR -> Textract in the front-end. We've never had an issue with grafting with Tesseract, which is the legacy code path, so I didn't feel the need to disable that path.

  • In the textract grafting branch that runs on lambda, we now check if the page we are grafting into has a visible text layer so that we don't overwrite it. This is the root cause of Textract grafting strips visible text on some PDFs #336
    This now leaves the PDF in a usable state and allows us more time to fix the larger issue captured in that issue on staging while allowing for Textract to work on production.

  • While I was here Claude noticed another issue captured here, which I patched in this PR as it is an trivial oversight and aligns with the implementation in the Django path.

base_page = base_pdf.pages[page_number]
base_page.add_overlay(overlay_pdf.pages[0], Rectangle(*base_page.trimbox))

This matches Django’s _merge_overlay, which already uses base_page.trimbox here:

base_page.add_overlay(overlay_page, Rectangle(*base_page.trimbox))

What I tested on staging

Related issues closed

#336
https://github.com/MuckRock/issues/issues/96

Later

The data destruction concerns captured in #336 are fixed in this issue by bypassing grafting if visible text is detected, aligning this behavior with what the other path does already. This allows for Textract to continue working on production, but the ultimate fix requires more scoping and is captured in #531. That is out of scope for this PR which seeks to address immediate memory and data destruction concerns.

@duckduckgrayduck duckduckgrayduck changed the title Turn off grafting, partially address the Textract overwrite problem Turn off grafting and partially address the Textract overwrite problem Nov 3, 2026
@duckduckgrayduck
duckduckgrayduck merged commit d5f6a31 into master Nov 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants