From 97806d16350fa92474a365a4dc8c6745b799301d Mon Sep 17 00:00:00 2001 From: Halleluyah Oludele Date: Sun, 2 Aug 2026 21:52:29 +0100 Subject: [PATCH] test: font-coverage corpus, and honour explicit spaces by default The suite ran on 26 words in a single font, which is how two font-metric bugs survived four releases. Adds a corpus that actually exercises the font layer, and fixes the first bug it found. Fixtures are hand-built rather than produced by a PDF library, because the point is to exercise what a producer normally hides: the standard-14 pages carry no /Widths and no /FontDescriptor, which is spec-legal and is exactly the case that hid both metric bugs. A reportlab fixture would embed the metrics and quietly test nothing. Three sizes per page, since a descender scaled by the wrong factor is indistinguishable from a wrong constant at a single size. They split by whether pdfplumber is a valid oracle. The 12 Latin fonts go to testdata/golden and are diffed against it automatically -- 144 words now match to 0.01pt on both axes. Symbol, ZapfDingbats and /Differences go to testdata/fonts with no generated golden and direct Go assertions instead, because pdfplumber decodes Symbol with StandardEncoding and returns "abgdep" where the answer is Greek. Generating a golden there would pin the wrong answer. The corpus immediately found a real divergence, which is the point. DefaultWordOpts did not set UseExplicitSpaces, so pdftable dropped space glyphs and re-inferred boundaries from the gap alone. pdfplumber ends a word AT a whitespace glyph before any gap test runs. At 8pt the space is 2.22pt wide, under the 3pt XTolerance, so a whole line collapsed into one run. Body type is routinely 8-9pt, so this was not exotic. Turning it on reproduces pdfplumber word-for-word and coordinate-for-coordinate. That change then exposed two flaws in MergeSplitTokens, both found by re-running against a real filing rather than the unit tests: - boundarySplitsToken measured gaps without excluding whitespace glyphs. A space is flush against its neighbours, so every space read as a zero-gap intra-word join and real words were welded together. Now a space sitting in the gap settles the question outright -- but only in the gap, since a space elsewhere in either cell says nothing about this boundary and blocking on it stops every merge. - the join was unconditionally separator-less. Whether to merge is a table-wide decision so the grid stays rectangular, but the SEPARATOR has to be per-row: the same boundary can cut a token on one row and fall between two real words on another. "(Dollars in millions," and "except per share amount)" became "millions,except" because a split several rows above had dropped the boundary. 3M 2018 10-K re-checked: 103/103 negatives intact, grid rectangular, spacing correct. --- CHANGELOG.md | 112 +- fonts_fixture_test.go | 170 ++ merge_split_test.go | 36 +- page.go | 39 +- scripts/gen_font_fixtures.py | 195 +++ testdata/fonts/differences.pdf | Bin 0 -> 796 bytes testdata/fonts/symbol.pdf | Bin 0 -> 1144 bytes .../golden/fonts-standard14.expected.json | 1397 +++++++++++++++++ testdata/golden/fonts-standard14.pdf | Bin 0 -> 6577 bytes .../fonts-standard14.tables.expected.json | 77 + text.go | 11 + 11 files changed, 2028 insertions(+), 9 deletions(-) create mode 100644 fonts_fixture_test.go create mode 100644 scripts/gen_font_fixtures.py create mode 100644 testdata/fonts/differences.pdf create mode 100644 testdata/fonts/symbol.pdf create mode 100644 testdata/golden/fonts-standard14.expected.json create mode 100644 testdata/golden/fonts-standard14.pdf create mode 100644 testdata/golden/fonts-standard14.tables.expected.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b24e03..156a977 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed @@ -87,7 +101,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed @@ -207,7 +235,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed @@ -352,7 +394,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed @@ -586,7 +642,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed @@ -942,7 +1012,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed @@ -1340,7 +1424,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Real font metrics for the 14 standard PDF fonts. No public API change. +Real font metrics for the 14 standard PDF fonts, plus a font-coverage +test corpus. One behaviour change to a default, noted below. + +### Changed (behaviour) + +- **`DefaultWordOpts()` now enables `UseExplicitSpaces`.** pdfplumber's + `WordExtractor` ends a word *at* a whitespace glyph, before any gap + test runs; pdftable dropped spaces and re-inferred boundaries purely + from the gap. At small type that silently over-merges: an 8pt space is + 278/1000 x 8 = 2.22pt, under the 3pt `XTolerance`, so + `Wim illegible 3,142 (16,048)` came back as one run while pdfplumber + returned four words. Body type in real documents is routinely 8-9pt, so + this was not an edge case. With it on, pdftable matches pdfplumber + word-for-word and coordinate-for-coordinate across all 12 Latin + standard fonts at 8/12/24pt. ### Fixed diff --git a/fonts_fixture_test.go b/fonts_fixture_test.go new file mode 100644 index 0000000..4e6d779 --- /dev/null +++ b/fonts_fixture_test.go @@ -0,0 +1,170 @@ +// Copyright (c) 2026 Halleluyah Oludele +// Licensed under the MIT License. + +package pdftable_test + +import ( + "strings" + "testing" + + "github.com/hallelx2/pdftable" +) + +// The fixtures under testdata/fonts deliberately have NO pdfplumber +// golden. pdfplumber decodes Symbol and ZapfDingbats with +// StandardEncoding and returns Latin letters where the correct answer is +// Greek and dingbats, so a generated golden would pin the wrong answer. +// Here we assert the right answer directly. +// +// Regenerate the PDFs with: python scripts/gen_font_fixtures.py + +func openPage(t *testing.T, path string, n int) pdftable.Page { + t.Helper() + doc, err := pdftable.OpenFile(path) + if err != nil { + t.Fatalf("OpenFile(%s): %v", path, err) + } + t.Cleanup(func() { doc.Close() }) + p, err := doc.Page(n) + if err != nil { + t.Fatalf("Page(%d): %v", n, err) + } + return p +} + +// TestSymbolFixtureDecoding is the end-to-end proof for HAL-481: a real +// PDF using Symbol and ZapfDingbats, parsed through the full pipeline. +// +// Both fonts carry their own built-in encoding and neither declares +// /Encoding, so a consumer that reaches for StandardEncoding reads Latin +// letters off a page of Greek. pdfplumber 0.11.9 still does exactly that +// — it returns "abgdep" for the Symbol page and "123" for the dingbats — +// which is why this fixture is asserted here rather than against it. +func TestSymbolFixtureDecoding(t *testing.T) { + t.Run("Symbol", func(t *testing.T) { + p := openPage(t, "testdata/fonts/symbol.pdf", 1) + text, err := p.ExtractText(pdftable.DefaultTextOpts()) + if err != nil { + t.Fatalf("ExtractText: %v", err) + } + // The content stream bytes are literally "abgdep". In Symbol's + // encoding those codes are alpha beta gamma delta epsilon pi. + const want = "αβγδεπ" + if !strings.Contains(text, want) { + t.Errorf("Symbol text = %q, want it to contain %q.\n"+ + "Getting \"abgdep\" back means StandardEncoding was applied "+ + "to a font that ships its own encoding.", text, want) + } + if strings.Contains(text, "abgdep") { + t.Error("Symbol decoded as Latin \"abgdep\" — built-in encoding not applied") + } + }) + + t.Run("ZapfDingbats", func(t *testing.T) { + p := openPage(t, "testdata/fonts/symbol.pdf", 2) + text, err := p.ExtractText(pdftable.DefaultTextOpts()) + if err != nil { + t.Fatalf("ExtractText: %v", err) + } + // Codes 0x31..0x33 are the a-names, not the digits 1..3. + if strings.Contains(text, "123") { + t.Errorf("ZapfDingbats text = %q, decoded as digits — built-in encoding not applied", text) + } + for _, r := range []rune{'✑', '✒', '✓'} { + if !strings.ContainsRune(text, r) { + t.Errorf("ZapfDingbats text = %q, missing %q", text, r) + } + } + }) +} + +// TestDifferencesFixtureDecoding covers the resolver split from HAL-481 +// through a real /Differences array. +// +// Symbol's glyph names are genuine Adobe Glyph List entries, so they mean +// the same thing in any font — a Helvetica whose /Differences names +// "Alpha" really is asking for U+0391. ZapfDingbats' "aNN" names are +// font-specific and must NOT resolve here: a Latin font naming "a1" means +// its own glyph, not U+2701 SCISSORS. Resolving those globally would +// silently corrupt text in any document that happens to use the name. +func TestDifferencesFixtureDecoding(t *testing.T) { + p := openPage(t, "testdata/fonts/differences.pdf", 1) + text, err := p.ExtractText(pdftable.DefaultTextOpts()) + if err != nil { + t.Fatalf("ExtractText: %v", err) + } + + // A=/Alpha B=/universal C=/club E=/summation F=/partialdiff + for _, tc := range []struct { + glyph string + want rune + }{ + {"Alpha", 'Α'}, + {"universal", '∀'}, + {"club", '♣'}, + {"summation", '∑'}, + {"partialdiff", '∂'}, + } { + if !strings.ContainsRune(text, tc.want) { + t.Errorf("/Differences name %q did not resolve to %q; text = %q", + tc.glyph, tc.want, tc.want) + } + } + + // D=/a1 — a ZapfDingbats name in a Helvetica font. It must not become + // the scissors dingbat. + if strings.ContainsRune(text, '✁') { + t.Errorf("/Differences name \"a1\" resolved to U+2701 in a Latin font; "+ + "dingbat names must stay font-scoped. text = %q", text) + } +} + +// TestStandard14FixtureCoversEveryLatinFont guards the corpus itself. +// The suite previously ran on 26 words in a single font, which is how two +// font-metric bugs survived four releases. If someone trims this fixture, +// the coverage silently collapses again — so assert the breadth, not just +// the output. +func TestStandard14FixtureCoversEveryLatinFont(t *testing.T) { + doc, err := pdftable.OpenFile("testdata/golden/fonts-standard14.pdf") + if err != nil { + t.Fatalf("OpenFile: %v", err) + } + defer doc.Close() + + want := []string{ + "Helvetica", "Helvetica-Bold", "Helvetica-Oblique", "Helvetica-BoldOblique", + "Times-Roman", "Times-Bold", "Times-Italic", "Times-BoldItalic", + "Courier", "Courier-Bold", "Courier-Oblique", "Courier-BoldOblique", + } + if doc.NumPages() != len(want) { + t.Fatalf("fixture has %d pages, want %d (one per Latin standard font)", + doc.NumPages(), len(want)) + } + + seen := map[string]bool{} + sizes := map[float64]bool{} + for n := 1; n <= doc.NumPages(); n++ { + p, err := doc.Page(n) + if err != nil { + t.Fatalf("Page(%d): %v", n, err) + } + chars, err := p.Chars() + if err != nil || len(chars) == 0 { + t.Fatalf("page %d: no chars (%v)", n, err) + } + for _, c := range chars { + seen[c.FontName] = true + sizes[c.FontSize] = true + } + } + for _, f := range want { + if !seen[f] { + t.Errorf("fixture never renders %s", f) + } + } + // Multiple sizes matter: a descent scaled by the wrong factor is + // indistinguishable from a wrong constant at a single size. + if len(sizes) < 3 { + t.Errorf("fixture uses %d font sizes, want >= 3", len(sizes)) + } +} diff --git a/merge_split_test.go b/merge_split_test.go index 021cf81..cce927f 100644 --- a/merge_split_test.go +++ b/merge_split_test.go @@ -115,6 +115,40 @@ func TestMergeSplitTokensLeavesRealColumnsAlone(t *testing.T) { } } +// TestBoundarySplitsTokenRespectsExplicitSpaces guards against welding +// real words together. +// +// The space glyph is itself a Char, sitting flush against its neighbours. +// Measuring gaps without excluding it makes every space look like an +// intra-word join — on a real 10-K that turned "(Dollars in millions," +// plus "except per share amount)" into "millions,except". A whitespace +// glyph across the boundary is the PDF stating outright that a word ends +// there, and it outranks any geometry. +func TestBoundarySplitsTokenRespectsExplicitSpaces(t *testing.T) { + left := BBox{X0: 90, X1: 118, Y0: 559, Y1: 569} + right := BBox{X0: 118, X1: 180, Y0: 559, Y1: 569} + + // No space: adjacent glyphs really are one token. + noSpace := []Char{ + glyph(",", 110, 116), + glyph("e", 116.02, 124), + } + if !boundarySplitsToken(noSpace, left, right, 3) { + t.Error("adjacent glyphs with no space should read as a split token") + } + + // Same geometry, but with the space glyph the producer actually + // emitted. Now it is two words and must not be merged. + withSpace := []Char{ + glyph(",", 110, 116), + glyph(" ", 116, 118.5), + glyph("e", 118.52, 126), + } + if boundarySplitsToken(withSpace, left, right, 3) { + t.Error("an explicit space across the boundary means two words — must not merge") + } +} + // TestMergeSplitTokensKeepsTableRectangular is the property that makes // the feature safe to use on a real table. // @@ -155,7 +189,7 @@ func TestMergeSplitTokensKeepsTableRectangular(t *testing.T) { } // The boundary split a token in row 0, so it goes for the whole table — // row 1 merges too, keeping the grid rectangular. - want := [][]string{{"Dec31,"}, {"A9"}} + want := [][]string{{"Dec31,"}, {"A 9"}} if !reflect.DeepEqual(gotRows, want) { t.Errorf("rows = %q, want %q", gotRows, want) } diff --git a/page.go b/page.go index 5e6d1a3..4131535 100644 --- a/page.go +++ b/page.go @@ -791,7 +791,24 @@ func mergeSplitTokens(rows [][]string, cells [][]BBox, chars []Char, tol float64 // so it belongs to the cell before it. if ci > 0 && drop[ci-1] && len(rowText) > 0 { n := len(rowText) - 1 - rowText[n] += text + // Whether to merge is a table-wide decision, so the grid + // stays rectangular. The SEPARATOR is per-row, because the + // same boundary can cut a token on one row and fall + // between two real words on another. + // + // Joining blind produced "millions,except" from + // "(Dollars in millions," + "except per share amount)" — + // the boundary was dropped for a split several rows above, + // and this row paid for it. + sep := " " + prevCell := rowCells[n] + if text == "" || rowText[n] == "" { + sep = "" + } else if !cell.IsZero() && !prevCell.IsZero() && + boundarySplitsToken(chars, prevCell, cell, tol) { + sep = "" // this row really is one token cut in half + } + rowText[n] += sep + text if !cell.IsZero() { if rowCells[n].IsZero() { rowCells[n] = cell @@ -818,7 +835,11 @@ func boundarySplitsToken(chars []Char, left, right BBox, tol float64) bool { if math.Abs(left.X1-right.X0) > 0.5 { return false } + // Nearest ink either side of the boundary, ignoring whitespace: a + // space glyph is flush against its neighbours, so counting it as ink + // would make every space look like a zero-gap intra-word join. var lastLeft, firstRight *Char + var spaces []*Char for i := range chars { c := &chars[i] hMid := (c.X0 + c.X1) / 2 @@ -826,6 +847,10 @@ func boundarySplitsToken(chars []Char, left, right BBox, tol float64) bool { if vMid < left.Y0 || vMid >= left.Y1 { continue } + if strings.TrimSpace(c.Text) == "" { + spaces = append(spaces, c) + continue + } if hMid >= left.X0 && hMid < left.X1 { if lastLeft == nil || c.X1 > lastLeft.X1 { lastLeft = c @@ -840,6 +865,18 @@ func boundarySplitsToken(chars []Char, left, right BBox, tol float64) bool { if lastLeft == nil || firstRight == nil { return false } + + // A space sitting IN the gap settles it: the PDF is stating outright + // that a word ends here, so whatever the geometry says, this is not + // one token cut in half. Only the gap is checked — a space elsewhere + // in either cell says nothing about this boundary, and treating it as + // if it did stops every merge from happening at all. + for _, sp := range spaces { + if sp.X1 > lastLeft.X1 && sp.X0 < firstRight.X0 { + return false + } + } + gap := firstRight.X0 - lastLeft.X1 return gap >= 0 && gap < tol } diff --git a/scripts/gen_font_fixtures.py b/scripts/gen_font_fixtures.py new file mode 100644 index 0000000..ab9b49d --- /dev/null +++ b/scripts/gen_font_fixtures.py @@ -0,0 +1,195 @@ +"""Generate font-coverage fixture PDFs for pdftable's tests. + + python scripts/gen_font_fixtures.py + python scripts/gen_golden.py # regenerate pdfplumber goldens + +Writes minimal, uncompressed PDFs. They are built by hand rather than +with a PDF library on purpose: the point is to exercise the paths a +*producer* would normally hide from us. + +The standard-14 pages carry **no /Widths and no /FontDescriptor**, which +is spec-legal (PDF 1.7 S9.6.2.2) and is exactly the case that hid two +font-metric bugs for four releases — a consumer has to supply the Adobe +AFM metrics itself or every glyph lands in the wrong place. A fixture +written by reportlab would embed the metrics and quietly test nothing. + +Three font sizes per page, because one of those bugs was a missing +font-size factor on the descender: at a single size a wrong scale factor +is indistinguishable from a wrong constant. 8pt also catches word +over-merging, since an 8pt space is only 2.2pt wide. + +Output splits by whether pdfplumber is a valid oracle: + + testdata/golden/ — the 12 Latin fonts. pdfplumber agrees, so + gen_golden.py generates the expected output and the + normal parity test picks them up automatically. + + testdata/fonts/ — Symbol, ZapfDingbats, /Differences. pdfplumber gets + these WRONG (it decodes Symbol with StandardEncoding + and returns "abgdep" where the answer is Greek), so a + generated golden would pin the wrong answer. These + are asserted directly in Go instead. +""" + +from __future__ import annotations + +import os +import sys + +GOLDEN = os.path.join("testdata", "golden") +FONTS = os.path.join("testdata", "fonts") + +# Glyphs whose AFM widths differ sharply (Helvetica i=222 vs m=833), so a +# flat-width fallback shows up immediately. The parenthesised negative is +# the shape that lost its sign in HAL-520. +LATIN_SAMPLE = "Wim illegible 3,142 (16,048)" + +LATIN_12 = [ + ("Helvetica", LATIN_SAMPLE), + ("Helvetica-Bold", LATIN_SAMPLE), + ("Helvetica-Oblique", LATIN_SAMPLE), + ("Helvetica-BoldOblique", LATIN_SAMPLE), + ("Times-Roman", LATIN_SAMPLE), + ("Times-Bold", LATIN_SAMPLE), + ("Times-Italic", LATIN_SAMPLE), + ("Times-BoldItalic", LATIN_SAMPLE), + ("Courier", LATIN_SAMPLE), + ("Courier-Bold", LATIN_SAMPLE), + ("Courier-Oblique", LATIN_SAMPLE), + ("Courier-BoldOblique", LATIN_SAMPLE), +] + +# These byte codes are alpha beta gamma delta epsilon pi in Symbol's own +# encoding, NOT Latin letters. A consumer applying StandardEncoding +# decodes "abgdep" — the bug HAL-481 fixed. +SYMBOL_FONTS = [ + ("Symbol", "abgdep"), + ("ZapfDingbats", "\\061\\062\\063"), +] + +SIZES = (8, 12, 24) + + +def esc(s: str) -> str: + """Escape parens for a PDF literal, leaving \\NNN octal intact.""" + return s.replace("(", r"\(").replace(")", r"\)") + + +def serialise(objs: list[bytes]) -> bytes: + out = bytearray(b"%PDF-1.7\n") + offsets = [0] * (len(objs) + 1) + for i, body in enumerate(objs, start=1): + offsets[i] = len(out) + out += f"{i} 0 obj\n".encode("latin-1") + body + b"\nendobj\n" + xref_at = len(out) + out += f"xref\n0 {len(objs) + 1}\n".encode("latin-1") + out += b"0000000000 65535 f \n" + for i in range(1, len(objs) + 1): + out += f"{offsets[i]:010d} 00000 n \n".encode("latin-1") + out += ( + f"trailer\n<< /Size {len(objs) + 1} /Root 1 0 R >>\n" + f"startxref\n{xref_at}\n%%EOF\n" + ).encode("latin-1") + return bytes(out) + + +def build_pdf(pages: list[tuple[str, str]]) -> bytes: + """One page per (basefont, text) entry, three font sizes each.""" + objs: list[bytes] = [b"", b""] # 1=Catalog, 2=Pages, filled in below + + def add(s: str) -> int: + objs.append(s.encode("latin-1")) + return len(objs) + + kids: list[int] = [] + for basefont, text in pages: + font_num = add(f"<< /Type /Font /Subtype /Type1 /BaseFont /{basefont} >>") + lines, y = [], 700 + for size in SIZES: + lines.append(f"BT /F1 {size} Tf 72 {y} Td ({esc(text)}) Tj ET") + y -= 60 + stream = "\n".join(lines) + content_num = add(f"<< /Length {len(stream)} >>\nstream\n{stream}\nendstream") + kids.append( + add( + "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] " + f"/Resources << /Font << /F1 {font_num} 0 R >> " + f"/ProcSet [/PDF /Text] >> /Contents {content_num} 0 R >>" + ) + ) + + kid_refs = " ".join(f"{k} 0 R" for k in kids) + objs[0] = b"<< /Type /Catalog /Pages 2 0 R >>" + objs[1] = f"<< /Type /Pages /Kids [{kid_refs}] /Count {len(kids)} >>".encode( + "latin-1" + ) + return serialise(objs) + + +def build_differences_pdf() -> bytes: + """A Helvetica font whose /Differences array renames glyphs. + + Covers the resolver split from HAL-481: Symbol's names are real AGL + entries and must resolve in any font, while ZapfDingbats' "aNN" names + are font-specific and must NOT — a Latin font naming "a1" means its + own glyph, not U+2701 SCISSORS. + """ + stream = ( + "BT /F1 18 Tf 72 700 Td (\\101\\102\\103\\104) Tj ET\n" + "BT /F1 18 Tf 72 640 Td (\\105\\106) Tj ET" + ) + objs = [ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", + ( + "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] " + "/Resources << /Font << /F1 4 0 R >> /ProcSet [/PDF /Text] >> " + "/Contents 6 0 R >>" + ).encode("latin-1"), + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding 5 0 R >>", + # A=Alpha, B=universal, C=club (real AGL names, must resolve), + # D=a1 (dingbat name, must NOT resolve globally), + # E=summation, F=partialdiff. + ( + "<< /Type /Encoding /Differences " + "[65 /Alpha /universal /club /a1 /summation /partialdiff] >>" + ).encode("latin-1"), + f"<< /Length {len(stream)} >>\nstream\n{stream}\nendstream".encode("latin-1"), + ] + return serialise(objs) + + +def main() -> int: + os.makedirs(GOLDEN, exist_ok=True) + os.makedirs(FONTS, exist_ok=True) + written = [] + + def emit(path: str, data: bytes, note: str) -> None: + with open(path, "wb") as f: + f.write(data) + written.append((path, note)) + + emit( + os.path.join(GOLDEN, "fonts-standard14.pdf"), + build_pdf(LATIN_12), + f"{len(LATIN_12)} pages, pdfplumber golden", + ) + emit( + os.path.join(FONTS, "symbol.pdf"), + build_pdf(SYMBOL_FONTS), + "2 pages, asserted in Go (pdfplumber is wrong here)", + ) + emit( + os.path.join(FONTS, "differences.pdf"), + build_differences_pdf(), + "1 page, asserted in Go", + ) + + for path, note in written: + print(f"wrote {path} ({note}, {os.path.getsize(path)} bytes)") + print("\nnow run: python scripts/gen_golden.py") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/testdata/fonts/differences.pdf b/testdata/fonts/differences.pdf new file mode 100644 index 0000000000000000000000000000000000000000..ed9c49f2026fbe28b3b5528fd1e48cfd8b371432 GIT binary patch literal 796 zcmZWn(Q4Z;6n*zsoR`4%u*tG5JAqPI+JwP2RvN!$eNb#URdK8!$tBy@@5+hOrV$9F zd(S=Rp39v)+%LcIuSLWW!)#unn;Wq5?_Po3NiSQo1$&TN<&Y5I5x2LI>goUt|I30S zj{VTJ!#WM{Ct@*u=fU3NkEEMi>bPn@gvT zneVE7RbE#TY|&Mw*4-B318ewSvYw&rUN?;*!@f`%E#C~JKB z!(&c2p_|gpKBIiWqP)WA*%>bgmybAFhLgf+Ee=*SQH&yr&j+~>sX&9b*bGA!-CImd zuVS3j1jBiUOLE$u53w}6=F0?)WW;$Zb*t>K+$;T0p`b-FChX;Dshy{}4!w(68cimP H-^=Jfja<|} literal 0 HcmV?d00001 diff --git a/testdata/fonts/symbol.pdf b/testdata/fonts/symbol.pdf new file mode 100644 index 0000000000000000000000000000000000000000..9068217f6c317c9add325256e50f808500d2e614 GIT binary patch literal 1144 zcmcJPK~KUk6vyxT6z@W!2W`92jU~hbC>J#v*-7NkxiUv^C0j|*Pw#sLWHN$YT+(H0 z|L?u__ix*IydGVM%g7VJ!K8Dq-v^d_EH$v9vT9*+V2>)-HTWoaf~zY}ml+A^|7XBP z$8L+PhKY3Y9=)fS%dAR^DfZz(D|}>7!p5uAy7Dv$0c@aZy&d^@PtBr{qT5Jur_0>F zfe@}nZ7Z$by+HzOB%lY$3?d&Q&LPR*Or?3Im*MbYxxWna3ga`1b>$l!1g9?E7MHqfqYqB(8>@ zMxXHUBp>$m+PXltb=|mKL*l_X%oj=Af4dsCQpG}7?&261pBiG^xW-tZ8^x`%wyLsj OjWHKqr*rc#^1c9pcPSSD literal 0 HcmV?d00001 diff --git a/testdata/golden/fonts-standard14.expected.json b/testdata/golden/fonts-standard14.expected.json new file mode 100644 index 0000000..48fd1fa --- /dev/null +++ b/testdata/golden/fonts-standard14.expected.json @@ -0,0 +1,1397 @@ +{ + "name": "fonts-standard14", + "pages": [ + { + "number": 1, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 87.992, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 90.21600000000001, + "x1": 116.888, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 119.11200000000001, + "x1": 139.12800000000001, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 141.35200000000003, + "x1": 171.14400000000003, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 95.988, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 99.324, + "x1": 139.332, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 142.668, + "x1": 172.69199999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 176.02799999999996, + "x1": 220.71599999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 119.976, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 126.648, + "x1": 206.664, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 213.33599999999998, + "x1": 273.38399999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 280.0559999999999, + "x1": 369.43199999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 2, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 88.88799999999999, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 91.112, + "x1": 120.904, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 123.12799999999999, + "x1": 143.14399999999998, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 145.368, + "x1": 175.16, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 97.332, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.668, + "x1": 145.356, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 148.692, + "x1": 178.71599999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 182.05199999999996, + "x1": 226.73999999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 122.664, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 129.336, + "x1": 218.712, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 225.384, + "x1": 285.43199999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 292.1039999999999, + "x1": 381.47999999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 3, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 87.992, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 90.21600000000001, + "x1": 116.888, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 119.11200000000001, + "x1": 139.12800000000001, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 141.35200000000003, + "x1": 171.14400000000003, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 95.988, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 99.324, + "x1": 139.332, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 142.668, + "x1": 172.69199999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 176.02799999999996, + "x1": 220.71599999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 119.976, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 126.648, + "x1": 206.664, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 213.33599999999998, + "x1": 273.38399999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 280.0559999999999, + "x1": 369.43199999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 4, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 88.88799999999999, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 91.112, + "x1": 120.904, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 123.12799999999999, + "x1": 143.14399999999998, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 145.368, + "x1": 175.16, + "y0": 698.344, + "y1": 706.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 97.332, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.668, + "x1": 145.356, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 148.692, + "x1": 178.71599999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 182.05199999999996, + "x1": 226.73999999999998, + "y0": 637.516, + "y1": 649.516, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 122.664, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 129.336, + "x1": 218.712, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 225.384, + "x1": 285.43199999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 292.1039999999999, + "x1": 381.47999999999996, + "y0": 575.032, + "y1": 599.032, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 5, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 88.0, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 90.0, + "x1": 116.22400000000002, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 118.22400000000002, + "x1": 136.22400000000002, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 138.22400000000002, + "x1": 165.55200000000002, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 96.0, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 99.0, + "x1": 138.33599999999998, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 141.336, + "x1": 168.336, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 171.336, + "x1": 212.328, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 120.0, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 126.0, + "x1": 204.672, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 210.672, + "x1": 264.672, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 270.672, + "x1": 352.656, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 6, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 88.888, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 90.888, + "x1": 117.56, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 119.56, + "x1": 137.56, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 139.56, + "x1": 166.88799999999998, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 97.332, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.332, + "x1": 140.34, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 143.34, + "x1": 170.34, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 173.34, + "x1": 214.33200000000002, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 122.66399999999999, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 128.664, + "x1": 208.68, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 214.68, + "x1": 268.68, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 274.68, + "x1": 356.66400000000004, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 7, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 86.664, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 88.664, + "x1": 114.888, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 116.888, + "x1": 134.888, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 136.888, + "x1": 164.216, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 93.996, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 96.996, + "x1": 136.332, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 139.332, + "x1": 166.332, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 169.332, + "x1": 210.32399999999998, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 115.992, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 121.99199999999999, + "x1": 200.664, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 206.664, + "x1": 260.664, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 266.664, + "x1": 348.64799999999997, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 8, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 87.56, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 89.56, + "x1": 115.784, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 117.78399999999999, + "x1": 135.784, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 137.784, + "x1": 165.11199999999997, + "y0": 698.264, + "y1": 706.264, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 95.34, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 98.34, + "x1": 137.67600000000002, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 140.676, + "x1": 167.676, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 170.676, + "x1": 211.668, + "y0": 637.396, + "y1": 649.396, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 118.67999999999999, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 124.68, + "x1": 203.352, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 209.352, + "x1": 263.352, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 269.352, + "x1": 351.336, + "y0": 574.792, + "y1": 598.792, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 9, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 86.39999999999999, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 91.2, + "x1": 134.4, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 139.2, + "x1": 163.2, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 167.99999999999997, + "x1": 206.39999999999998, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 93.60000000000001, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.8, + "x1": 165.60000000000002, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 172.8, + "x1": 208.8, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 216.0, + "x1": 273.5999999999999, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 115.19999999999999, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 129.6, + "x1": 259.20000000000005, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 273.6, + "x1": 345.6, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 360.0, + "x1": 475.1999999999998, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 10, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 86.39999999999999, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 91.2, + "x1": 134.4, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 139.2, + "x1": 163.2, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 167.99999999999997, + "x1": 206.39999999999998, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 93.60000000000001, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.8, + "x1": 165.60000000000002, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 172.8, + "x1": 208.8, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 216.0, + "x1": 273.5999999999999, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 115.19999999999999, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 129.6, + "x1": 259.20000000000005, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 273.6, + "x1": 345.6, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 360.0, + "x1": 475.1999999999998, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 11, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 86.39999999999999, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 91.2, + "x1": 134.4, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 139.2, + "x1": 163.2, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 167.99999999999997, + "x1": 206.39999999999998, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 93.60000000000001, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.8, + "x1": 165.60000000000002, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 172.8, + "x1": 208.8, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 216.0, + "x1": 273.5999999999999, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 115.19999999999999, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 129.6, + "x1": 259.20000000000005, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 273.6, + "x1": 345.6, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 360.0, + "x1": 475.1999999999998, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + } + ] + }, + { + "number": 12, + "width": 612, + "height": 792, + "extract_text": "Wim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)\nWim illegible 3,142 (16,048)", + "extract_words": [ + { + "text": "Wim", + "x0": 72.0, + "x1": 86.39999999999999, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 91.2, + "x1": 134.4, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 139.2, + "x1": 163.2, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 167.99999999999997, + "x1": 206.39999999999998, + "y0": 698.448, + "y1": 706.448, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 93.60000000000001, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 100.8, + "x1": 165.60000000000002, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 172.8, + "x1": 208.8, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 216.0, + "x1": 273.5999999999999, + "y0": 637.672, + "y1": 649.672, + "upright": true, + "direction": "ltr" + }, + { + "text": "Wim", + "x0": 72.0, + "x1": 115.19999999999999, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "illegible", + "x0": 129.6, + "x1": 259.20000000000005, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "3,142", + "x0": 273.6, + "x1": 345.6, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + }, + { + "text": "(16,048)", + "x0": 360.0, + "x1": 475.1999999999998, + "y0": 575.344, + "y1": 599.344, + "upright": true, + "direction": "ltr" + } + ] + } + ] +} \ No newline at end of file diff --git a/testdata/golden/fonts-standard14.pdf b/testdata/golden/fonts-standard14.pdf new file mode 100644 index 0000000000000000000000000000000000000000..4a2be45761c5b610c66c9f966c4169951a2bf2ae GIT binary patch literal 6577 zcmds+&rjPh6vyxVD||r$36}U*en_Z?Zd9AL2~;ZW036cRbU|seCF#WW*YDZJtrJ(m zo5CeMH1KM_*ze=_eRgtk^I>{{-^C-0LR2oEMwgc;nE(DEQ1Cu$vZ7q0;3iv(8WCtP zL*wyCY*(_O{J-5nw+lY!s~X*L)jUz(SUR5}%=9Hz7ZU0V)t5+LQgxyFW>h<&%F6?c zsN8KEgo!l@*}FZ8X$ftD+ufp3vygMaD41lm(7${X#V^t1%gpj32WGr{5!-e19pQw@ z26a=3Y%`k7Q82|QLGwox6BLIbny=95w|s;0q7dtRQ3yoOF(c^y6i4SFOYYCk(EJHq z%}r+*x9%OW|L9HFOK+b1OK*VnGKCjxGNfjznzO_vAy)zdrvo2+6{|d(l+Wl+Qi@>c zaY}%|Ow{GBTEcRW>qG@q{eh9jI39zis$AZR2Hgd)wqP-d=jH)IDe-L^xJQljIyfvP z$-X@%Z-^I@vRGLp<=Qf?22N6Dg_&xfBy}eGk<_VtASu(gJtlvW*NYv*v`b5`EcGtOlciDTJxjL?__6F1PHao}z)`BL=UZqk z*85#glt!5iO5*Op{ZM{_je5Sc@{-uHJ#?6MJ?S$q6%JpGRS~xMU0ix)Y7QMff~z8K=jd}K9!DXf=_uTPP{cVr-F-v{8&nJV zLED3k!-P6EoWj<&VlIJfQ(gTXj9%G|z3VnBH0#F-?aGhdQ&#~%%)R=3AON#wRW5(COo z=N>eca^CTmBo0fOa;F``i4zw_<%@#0KgQ&fl8!B6Zo4>j;=+>HnH$HhKTaZNZE(s! z+S>Q!^2z<))>K(uh)TV0xy}E;+k+Gyua_mf-++UI%AdN)szzlGhfs}9POh$}qrXxs BzL5X` literal 0 HcmV?d00001 diff --git a/testdata/golden/fonts-standard14.tables.expected.json b/testdata/golden/fonts-standard14.tables.expected.json new file mode 100644 index 0000000..74f47b7 --- /dev/null +++ b/testdata/golden/fonts-standard14.tables.expected.json @@ -0,0 +1,77 @@ +{ + "name": "fonts-standard14", + "pages": [ + { + "number": 1, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 2, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 3, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 4, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 5, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 6, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 7, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 8, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 9, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 10, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 11, + "width": 612, + "height": 792, + "tables": [] + }, + { + "number": 12, + "width": 612, + "height": 792, + "tables": [] + } + ] +} \ No newline at end of file diff --git a/text.go b/text.go index a1c8c33..c841977 100644 --- a/text.go +++ b/text.go @@ -184,6 +184,17 @@ func DefaultWordOpts() WordOpts { HorizontalLTR: true, VerticalTTB: true, Expand: true, + // On by default because pdfplumber does the same, and leaving it + // off silently over-merges small type. pdfplumber's WordExtractor + // ends a word AT a whitespace glyph, before any gap test runs; + // only when there is no space glyph does it fall back to the gap. + // + // Without this, 8pt text merged into one run: the space is + // 278/1000 x 8 = 2.22pt wide, under the 3pt XTolerance, so the + // gap check saw no boundary even though the PDF had emitted an + // explicit space. Body type in real documents is routinely 8-9pt, + // so this was not an edge case. + UseExplicitSpaces: true, } }