Fix pdf text extraction multipage

[codewithfourtix]

What

Fix a regression in textcode/pdf.py where PDF text extraction silently stops after the first page, and can return None instead of a list of lines.

The bug

The refactoring in #4606 (commit fa63f1a7d, "Improve package scan performance") removed one level of with contextlib.closing(...) nesting in get_text_lines() and dedented the function body — except the final two lines, which kept their old absolute indentation and therefore silently moved inside the page loop:

for page_num, page in enumerate(pages, 1):
    interpreter.process_page(page)
    if max_pages and page_num == max_pages:
        break
    extracted_text.seek(0)
    return extracted_text.readlines()   # returns after processing page 1

Consequences:

• With the default max_pages=5, the function returns right after processing page 1, so text on pages 2–5 is never extracted. Copyright, license, email and URL detection silently miss anything past the first page of every PDF.
• When max_pages is reached (via break), or the PDF yields no pages, the function returns None. The caller textcode.analysis.unicode_text_lines_from_pdf then crashes with TypeError: 'NoneType' object is not iterable.

Before #4606, both lines executed after the loop. This PR restores that placement.

Why existing tests did not catch it

The PDF test files in tests/textcode/data/pdf/ only assert content from the first page, so the suite passes both with and without the regression.

Tests added

A 7-page test PDF (multi_page.pdf) with distinct text on each page, plus three regression tests asserting that:

• text is extracted from all pages up to the default max_pages=5 (and not beyond);
• reaching max_pages returns the extracted lines and not None;
• max_pages=0 extracts all pages.

Verified against the pinned pdfminer.six==20260107: all existing PDF tests still pass, and the new tests fail on develop and pass with this fix.

Related issues

I checked open issues and PRs — none cover this regression. (Open issue #3794 is a different, older TypeError coming from inside pdfminer on a malformed PDF, predating #4606.)

[Copilot]

Pull request overview

Fixes a regression in PDF text extraction where textcode/pdf.py:get_text_lines() returned after processing only the first page (and could return None in some paths), causing downstream text analysis to miss content beyond page 1 and potentially crash.

Changes:

• Move extracted_text.seek(0) and return extracted_text.readlines() outside the per-page loop so extraction continues through all pages up to max_pages.
• Add regression tests covering multi-page extraction, max_pages=1 (no None), and max_pages=0 (unlimited pages).

Reviewed changes

Copilot reviewed 1 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/textcode/pdf.py	Fixes control flow so text extraction does not stop after the first processed page.
tests/textcode/test_pdf.py	Adds multi-page regression tests validating correct max_pages behavior and non-None returns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[AyanSinhaMahapatra]

Updating the docstring like suggested here is actually a good idea.

[AyanSinhaMahapatra]

Thanks @codewithfourtix please see the test failure that you need to fix, this is related directly to your changes.

[codewithfourtix]

@AyanSinhaMahapatra All green now.

Turns out those extra licenses weren't real, they got matched off the truncated first page. Checked the pdf and it's 7 pages of plain SISSL 1.1, no cpal or proprietary wording anywhere in it. Both were added to the yml in #4606 itself, so I just put the file back to what it was before that.

Docstring updated too.

Also noticed #4606 trimmed apache_fop_expected in test_pdf.py from 14 lines to 8. It passes either way since it's an in check, so I left it, but can restore it in a follow-up if you want.