diff --git a/README.md b/README.md index 4e49166..f1ef9b3 100644 --- a/README.md +++ b/README.md @@ -603,12 +603,38 @@ stdlib-only. and a `pdftable` CLI mirroring pdfplumber's surface. - `v0.4.x` — bundle the standard-14 AFM metrics so word bboxes (and therefore cell text) match pdfplumber on standard fonts. **Done**: the - Adobe Core 14 metrics ship, and Symbol/ZapfDingbats decode with their - own built-in encodings. The golden position envelope is still asserted - at 15pt pending a re-measure, so the "within 1 point" claim is not yet - evidenced. -- `v0.5.x` — performance pass: parser benchmarking against - pdfminer.six and pdfplumber on a representative document corpus. + Adobe Core 14 metrics ship, Symbol/ZapfDingbats decode with their own + built-in encodings, and measured position drift against pdfplumber is + **0.0000pt on both axes** — the golden envelope is now asserted at + 0.01pt. See + [the evaluation](docs/evaluations/2026-08-02-font-metrics-and-table-fidelity.md). +- `v0.5.x` — **table detection**. The ICDAR 2013 benchmark puts + end-to-end F1 at 0.362, level with pdfplumber's 0.370, and the + diagnostic is unambiguous: precision 0.865, recall 0.229. What we + extract is right; we miss three quarters of the tables, because the + `lines` strategy needs *intersecting* rulings and a horizontally-ruled + table produces none. See + [the evaluation](docs/evaluations/2026-08-02-icdar2013-table-structure.md). +- `v0.6.x` — performance pass: parser speed against pdfminer.six and + pdfplumber on a representative corpus. + +## Repository layout + +`pdftable` is a single Go package, so its source lives in the repository +root — that is the import path, and Go keeps `_test.go` files next to the +code they cover. + +| | | +| --- | --- | +| `*.go` | the library | +| `internal/pdf/` | content-stream interpreter, fonts, encodings | +| `internal/layout/` | edge/intersection/cell geometry | +| `cmd/pdftable/` | CLI | +| `bench/` | accuracy benchmarks against public datasets — a **separate Go module**, so it never adds a dependency here | +| `docs/evaluations/` | dated measurement reports, with their caveats | +| `scripts/` | fixture and golden-file generators | +| `testdata/golden/` | fixtures compared against pdfplumber | +| `testdata/fonts/` | fixtures where pdfplumber is wrong, asserted directly | ## License diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 0000000..d79e8d4 --- /dev/null +++ b/bench/README.md @@ -0,0 +1,33 @@ +# Benchmarks + +Accuracy benchmarks against public datasets with published ground truth. + +These are **not** `go test` benchmarks (those live next to the code as +`*_bench_test.go` and measure speed). These measure **correctness** against +an external reference, and they are deliberately outside the library's Go +module so their dependencies never reach a consumer of `pdftable`. + +Datasets are **not committed** — they are large and separately licensed. +Each harness downloads what it needs into a scratch directory. + +| Benchmark | Dataset | Measures | Report | +| --- | --- | --- | --- | +| [`icdar2013/`](icdar2013/) | ICDAR 2013 Table Competition (125 PDFs) | table detection + structure | [2026-08-02](../docs/evaluations/2026-08-02-icdar2013-table-structure.md) | + +## Why the numbers live in `docs/evaluations/` + +A benchmark result is only meaningful with its date, the code version it +ran against, and the caveats on the metric. A bare number in a README rots +the moment either side changes. Every run gets a dated report; the table +above links to the most recent. + +## Adding a benchmark + +1. `bench//` with its own `go.mod` if it needs Go. +2. A `run.py` that fetches the dataset and prints a result table. +3. A dated report in `docs/evaluations/`, and a row in the table above. + +State the metric precisely, and say what it is **not** comparable to. +Published table-extraction scores in particular are usually +structure-only — the system is handed the table location — and are not +comparable to an end-to-end number. diff --git a/bench/go.mod b/bench/go.mod new file mode 100644 index 0000000..f592b91 --- /dev/null +++ b/bench/go.mod @@ -0,0 +1,11 @@ +// Separate module on purpose: the benchmark harness must never add a +// dependency to the library it measures. Nested modules are excluded from +// the parent's ./... , so go build ./... at the repo root ignores this +// entirely and a consumer of pdftable never fetches it. +module github.com/hallelx2/pdftable/bench + +go 1.25.0 + +require github.com/hallelx2/pdftable v0.0.0 + +replace github.com/hallelx2/pdftable => ../ diff --git a/bench/icdar2013/README.md b/bench/icdar2013/README.md new file mode 100644 index 0000000..fb7531c --- /dev/null +++ b/bench/icdar2013/README.md @@ -0,0 +1,57 @@ +# ICDAR 2013 Table Competition benchmark + +Measures **table detection + structure recognition** end to end against +125 born-digital PDFs from EU and US government sources, with per-cell +ground truth. + +```sh +pip install pdfplumber +python bench/icdar2013/run.py # full run, ~10 min +python bench/icdar2013/run.py --limit 5 # quick check while iterating +python bench/icdar2013/run.py --diag # detection vs structure breakdown +``` + +The dataset (~12 MB) downloads to `~/.cache/pdftable-bench` (override with +`PDFTABLE_BENCH_DIR`). It is not committed. + +## Metric: adjacency relations + +From Göbel et al., the metric the competition itself used. For every +non-empty cell, take its nearest non-empty neighbour to the right and +below; each pair is a relation `(text_a, text_b, direction)`. Score the +detected multiset against ground truth. + +Cell-by-cell grid comparison would be the obvious alternative and is +worse: two tools can grid the same table differently — one emits a spacer +column, the other does not — and still convey identical structure. A grid +diff calls that a failure. Adjacency asks the question a reader actually +cares about: *is this value next to that label?* It is unforgiving about +genuinely wrong structure and forgiving about harmless disagreements. + +Documents are scored whole, with relations unioned across pages and +tables. That sidesteps matching detected tables to ground-truth tables, a +step which would need its own arbitrary thresholds and would make the +number depend on them. + +## Reading the result + +**This is an end-to-end number and is NOT comparable to published ICDAR +2013 scores of 0.85–0.95.** Those evaluate structure recognition with the +table region *given*; this has to find the table first. Compare runs of +this harness to each other, and to the pdfplumber column, not to papers. + +`--diag` splits the two failure modes, which need opposite work: + +- **the table was never detected** — a detection problem +- **it was detected but gridded wrongly** — a structure problem + +Latest results: [`docs/evaluations/`](../../docs/evaluations/). + +## Files + +| | | +| --- | --- | +| `run.py` | fetches the dataset, builds the extractor, runs everything | +| `extract.go` | dumps every table pdftable finds as JSON; built as its own module so the benchmark adds no dependency to the library | +| `score.py` | parses ground-truth XML, computes precision/recall/F1 | +| `diag.py` | separates detection failures from structure failures | diff --git a/bench/icdar2013/diag.py b/bench/icdar2013/diag.py new file mode 100644 index 0000000..c8d5dc8 --- /dev/null +++ b/bench/icdar2013/diag.py @@ -0,0 +1,86 @@ +"""Separate the two ways table extraction can fail on ICDAR 2013. + +Low recall has two very different causes and the headline F1 cannot tell +them apart: + + (a) the table was never detected at all — a detection problem + (b) it was detected but gridded wrongly — a structure problem + +(a) is fixed by better "is this a table?" logic; (b) by better cell +geometry. They need opposite work, so it is worth knowing which one we +have before touching anything. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from collections import Counter + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from score import gt_relations, norm, relations_from_grid, score # noqa: E402 + + +def main() -> int: + root, exe = sys.argv[1], sys.argv[2] + pairs = [] + for dirpath, _, files in os.walk(root): + for f in sorted(files): + if f.endswith("-str.xml"): + pdf = os.path.join(dirpath, f.replace("-str.xml", ".pdf")) + if os.path.exists(pdf): + pairs.append((pdf, os.path.join(dirpath, f))) + pairs.sort() + + zero_detect = 0 + gt_regions_total = 0 + detected_tables_total = 0 + pages_total = 0 + per_doc = [] + + for pdf, xml in pairs: + gt = gt_relations(xml) + out = subprocess.run( + [exe, "-strategy", "lines", pdf], capture_output=True, timeout=120 + ).stdout + tables = json.loads(out or b"[]") + detected_tables_total += len(tables) + rels: Counter = Counter() + for t in tables: + rels += relations_from_grid([[norm(c) for c in r] for r in t["rows"]]) + c, nd, ng = score(gt, rels) + if nd == 0: + zero_detect += 1 + gt_regions_total += 1 + per_doc.append((os.path.basename(pdf), ng, nd, c, len(tables))) + + print(f"documents : {len(pairs)}") + print(f"documents where we found NO table: {zero_detect}" + f" ({100*zero_detect/len(pairs):.0f}%)") + print(f"tables detected (total) : {detected_tables_total}") + print() + + # Restrict to documents where we DID detect something: if structure is + # good, precision AND recall should both be high on this subset. + tot = [0, 0, 0] + for _, ng, nd, c, _ in per_doc: + if nd: + tot[0] += c + tot[1] += nd + tot[2] += ng + p = tot[0] / tot[1] if tot[1] else 0 + r = tot[0] / tot[2] if tot[2] else 0 + f = 2 * p * r / (p + r) if (p + r) else 0 + print("--- documents where a table WAS detected ---") + print(f" precision {p:.3f} recall {r:.3f} F1 {f:.3f}") + print() + print("worst 8 documents by missed relations:") + for name, ng, nd, c, nt in sorted(per_doc, key=lambda x: x[1] - x[3])[-8:]: + print(f" {name:<18} gt={ng:<6} detected={nd:<6} correct={c:<6} tables={nt}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/icdar2013/extract.go b/bench/icdar2013/extract.go new file mode 100644 index 0000000..ba010bd --- /dev/null +++ b/bench/icdar2013/extract.go @@ -0,0 +1,70 @@ +// Emit every table pdftable finds in a PDF, as JSON, for benchmarking. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "os" + + "github.com/hallelx2/pdftable" +) + +type tableOut struct { + Page int `json:"page"` + Rows [][]string `json:"rows"` +} + +func main() { + strategy := flag.String("strategy", "lines", "lines | text | fallback") + merge := flag.Bool("merge", false, "TableSettings.MergeSplitTokens") + flag.Parse() + + doc, err := pdftable.OpenFile(flag.Arg(0)) + if err != nil { + // A file we cannot open scores zero rather than aborting the run; + // the harness needs a result for every document. + json.NewEncoder(os.Stdout).Encode([]tableOut{}) + fmt.Fprintln(os.Stderr, "open:", err) + return + } + defer doc.Close() + + mk := func(v, h pdftable.TableStrategy) pdftable.TableSettings { + s := pdftable.DefaultTableSettings() + s.VerticalStrategy, s.HorizontalStrategy = v, h + s.MergeSplitTokens = *merge + return s + } + lines := mk(pdftable.StrategyLines, pdftable.StrategyLines) + text := mk(pdftable.StrategyText, pdftable.StrategyText) + + var attempts []pdftable.TableSettings + switch *strategy { + case "lines": + attempts = []pdftable.TableSettings{lines} + case "text": + attempts = []pdftable.TableSettings{text} + default: // fallback: ruled cells first, whitespace alignment if none + attempts = []pdftable.TableSettings{lines, text} + } + + out := []tableOut{} + for i := 1; i <= doc.NumPages(); i++ { + p, err := doc.Page(i) + if err != nil { + continue + } + for _, s := range attempts { + tables, err := p.ExtractTables(s) + if err != nil || len(tables) == 0 { + continue + } + for _, t := range tables { + out = append(out, tableOut{Page: i, Rows: t.Rows}) + } + break + } + } + json.NewEncoder(os.Stdout).Encode(out) +} diff --git a/bench/icdar2013/run.py b/bench/icdar2013/run.py new file mode 100644 index 0000000..7b578f7 --- /dev/null +++ b/bench/icdar2013/run.py @@ -0,0 +1,112 @@ +"""Run the ICDAR 2013 table benchmark end to end. + + pip install pdfplumber + python bench/icdar2013/run.py + +Downloads the dataset (~12 MB) into a scratch directory, builds the Go +extractor, scores pdftable against the ground truth, and prints a result +table. pdfplumber is scored alongside as a reference point — the question +"is 0.36 good?" is unanswerable without a baseline, and pdfplumber is the +implementation pdftable is a port of. + +Pass --limit N to score only the first N documents while iterating. +""" + +from __future__ import annotations + +import argparse +import os +import shutil +import subprocess +import sys +import tarfile +import urllib.request + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..")) + +# Brandon Smock's corrected edition of the ICDAR 2013 competition set. +# The original release had known ground-truth errors; this is the version +# current table-structure papers evaluate against. +URL = ( + "https://huggingface.co/datasets/bsmock/" + "ICDAR-2013-Table-Competition-Corrected/resolve/main/" + "ICDAR-2013-Table-Competition-Corrected.tar.gz" +) +DIRNAME = "ICDAR-2013-Table-Competition-Corrected" + + +def scratch() -> str: + d = os.environ.get("PDFTABLE_BENCH_DIR") or os.path.join( + os.path.expanduser("~"), ".cache", "pdftable-bench" + ) + os.makedirs(d, exist_ok=True) + return d + + +def fetch(dest: str) -> str: + root = os.path.join(dest, DIRNAME) + if os.path.isdir(root): + print(f"dataset already present: {root}") + return root + tgz = os.path.join(dest, "icdar2013.tar.gz") + if not os.path.exists(tgz): + print(f"downloading {URL}") + urllib.request.urlretrieve(URL, tgz) + print("extracting...") + with tarfile.open(tgz) as t: + t.extractall(dest) + return root + + +def build_extractor(dest: str) -> str: + exe = os.path.join(dest, "bench-extract.exe" if os.name == "nt" else "bench-extract") + mod = os.path.join(dest, "extractor") + os.makedirs(mod, exist_ok=True) + shutil.copy(os.path.join(HERE, "extract.go"), os.path.join(mod, "main.go")) + + # Its own module, pointed at the working tree: the benchmark must never + # add a dependency to the library it is measuring. + gomod = os.path.join(mod, "go.mod") + if not os.path.exists(gomod): + subprocess.run(["go", "mod", "init", "benchextract"], cwd=mod, check=True, + capture_output=True) + subprocess.run( + ["go", "mod", "edit", "-replace", + f"github.com/hallelx2/pdftable={REPO.replace(os.sep, '/')}"], + cwd=mod, check=True, capture_output=True) + subprocess.run( + ["go", "mod", "edit", "-require", "github.com/hallelx2/pdftable@v0.0.0"], + cwd=mod, check=True, capture_output=True) + subprocess.run(["go", "mod", "tidy"], cwd=mod, check=True, capture_output=True) + subprocess.run(["go", "build", "-o", exe, "."], cwd=mod, check=True) + print(f"built {exe}") + return exe + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--limit", type=int, default=0, + help="score only the first N documents") + ap.add_argument("--diag", action="store_true", + help="also run the detection-vs-structure diagnostic") + args = ap.parse_args() + + dest = scratch() + root = fetch(dest) + exe = build_extractor(dest) + + cmd = [sys.executable, os.path.join(HERE, "score.py"), root, exe] + if args.limit: + cmd.append(str(args.limit)) + subprocess.run(cmd, check=True) + + if args.diag: + print() + subprocess.run( + [sys.executable, os.path.join(HERE, "diag.py"), root, exe], check=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/icdar2013/score.py b/bench/icdar2013/score.py new file mode 100644 index 0000000..1e3fdb8 --- /dev/null +++ b/bench/icdar2013/score.py @@ -0,0 +1,184 @@ +"""Benchmark table extraction on the ICDAR 2013 table competition set. + +Metric: adjacency relations (Goebel et al., ICDAR 2013) — the standard for +this dataset. For every non-empty cell, take its nearest non-empty +neighbour to the right and below; each such pair is a relation +(text_a, text_b, direction). Score the detected relation multiset against +ground truth. + +Why this metric rather than comparing grids cell-by-cell: two tools can +grid a table differently — one emits a spacer column, another does not — +and still convey exactly the same structure. Adjacency asks the question +that actually matters for a reader: "is this value next to that label?" +It is unforgiving about genuinely wrong structure and forgiving about +harmless disagreements over gridding. + +Documents are scored as a whole (relations unioned across pages/tables), +which sidesteps having to match detected tables to ground-truth tables — +a matching step that would itself need arbitrary thresholds. +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import sys +import xml.etree.ElementTree as ET +from collections import Counter + +WS = re.compile(r"\s+") + + +def norm(s: str | None) -> str: + return WS.sub(" ", (s or "")).strip() + + +def relations_from_grid(grid: list[list[str]]) -> Counter: + """Adjacency relations from a dense row/col grid.""" + rels: Counter = Counter() + nrows = len(grid) + for r in range(nrows): + ncols = len(grid[r]) + for c in range(ncols): + a = grid[r][c] + if not a: + continue + # nearest non-empty to the right, skipping blanks + for c2 in range(c + 1, ncols): + if grid[r][c2]: + rels[(a, grid[r][c2], "H")] += 1 + break + # nearest non-empty below + for r2 in range(r + 1, nrows): + row2 = grid[r2] + if c < len(row2) and row2[c]: + rels[(a, row2[c], "V")] += 1 + break + return rels + + +def gt_relations(xml_path: str) -> Counter: + """Ground truth: build a dense grid per region from start/end row+col.""" + rels: Counter = Counter() + root = ET.parse(xml_path).getroot() + for region in root.iter("region"): + cells = [] + for cell in region.findall("cell"): + sr = int(cell.get("start-row", 0)) + sc = int(cell.get("start-col", 0)) + er = int(cell.get("end-row", sr)) + ec = int(cell.get("end-col", sc)) + content = cell.find("content") + text = norm(content.text if content is not None else "") + cells.append((sr, sc, er, ec, text)) + if not cells: + continue + nrows = max(c[2] for c in cells) + 1 + ncols = max(c[3] for c in cells) + 1 + grid = [["" for _ in range(ncols)] for _ in range(nrows)] + for sr, sc, er, ec, text in cells: + # A spanning cell occupies every position it covers, which is + # what makes adjacency work across merged headers. + for r in range(sr, er + 1): + for c in range(sc, ec + 1): + if r < nrows and c < ncols: + grid[r][c] = text + rels += relations_from_grid(grid) + return rels + + +def score(gt: Counter, got: Counter) -> tuple[int, int, int]: + correct = sum((gt & got).values()) + return correct, sum(got.values()), sum(gt.values()) + + +def prf(correct: int, ndet: int, ngt: int) -> tuple[float, float, float]: + p = correct / ndet if ndet else 0.0 + r = correct / ngt if ngt else 0.0 + f = 2 * p * r / (p + r) if (p + r) else 0.0 + return p, r, f + + +def run_pdftable(exe: str, pdf: str, strategy: str, merge: bool) -> Counter: + cmd = [exe, "-strategy", strategy] + if merge: + cmd.append("-merge") + cmd.append(pdf) + try: + out = subprocess.run(cmd, capture_output=True, timeout=120).stdout + tables = json.loads(out or b"[]") + except Exception: + return Counter() + rels: Counter = Counter() + for t in tables: + rels += relations_from_grid([[norm(c) for c in row] for row in t["rows"]]) + return rels + + +def run_pdfplumber(pdf: str, strategy: str) -> Counter: + import pdfplumber + + settings = {"vertical_strategy": strategy, "horizontal_strategy": strategy} + rels: Counter = Counter() + try: + with pdfplumber.open(pdf) as doc: + for page in doc.pages: + for tbl in page.extract_tables(settings): + rels += relations_from_grid( + [[norm(c) for c in row] for row in tbl] + ) + except Exception: + return Counter() + return rels + + +def main() -> int: + root = sys.argv[1] + exe = sys.argv[2] + limit = int(sys.argv[3]) if len(sys.argv) > 3 else 0 + + pairs = [] + for dirpath, _, files in os.walk(root): + for f in sorted(files): + if not f.endswith("-str.xml"): + continue + pdf = os.path.join(dirpath, f.replace("-str.xml", ".pdf")) + if os.path.exists(pdf): + pairs.append((pdf, os.path.join(dirpath, f))) + pairs.sort() + if limit: + pairs = pairs[:limit] + print(f"scoring {len(pairs)} documents\n", flush=True) + + systems = { + "pdftable (lines)": lambda p: run_pdftable(exe, p, "lines", False), + "pdftable (fallback)": lambda p: run_pdftable(exe, p, "fallback", False), + "pdftable (fallback+merge)": lambda p: run_pdftable(exe, p, "fallback", True), + "pdfplumber (lines)": lambda p: run_pdfplumber(p, "lines"), + "pdfplumber (text)": lambda p: run_pdfplumber(p, "text"), + } + totals = {k: [0, 0, 0] for k in systems} + + for i, (pdf, xml) in enumerate(pairs, 1): + gt = gt_relations(xml) + for name, fn in systems.items(): + c, nd, ng = score(gt, fn(pdf)) + totals[name][0] += c + totals[name][1] += nd + totals[name][2] += ng + if i % 10 == 0: + print(f" ...{i}/{len(pairs)}", flush=True) + + print(f"\n{'system':<28} {'precision':>10} {'recall':>10} {'F1':>10}") + print("-" * 62) + for name in systems: + p, r, f = prf(*totals[name]) + print(f"{name:<28} {p:>10.3f} {r:>10.3f} {f:>10.3f}") + print(f"\nground-truth relations: {totals['pdftable (lines)'][2]}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..41c9e96 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,39 @@ +# Documentation + +| | | +| --- | --- | +| [`evaluations/`](evaluations/) | dated accuracy findings — what was measured, against what, and what it does not prove | + +User-facing documentation lives in the root [`README.md`](../README.md); +release history in [`CHANGELOG.md`](../CHANGELOG.md). + +## Evaluations + +| date | subject | headline | +| --- | --- | --- | +| [2026-08-02](evaluations/2026-08-02-icdar2013-table-structure.md) | ICDAR 2013 table detection + structure | F1 0.362 end-to-end; the bottleneck is **detection**, not cell accuracy | +| [2026-08-02](evaluations/2026-08-02-font-metrics-and-table-fidelity.md) | font metrics and table fidelity | position drift 11.99pt → **0.0000pt**; negative-sign loss 19% → **0%** | + +### Conventions + +Each report states: + +- the **date** and the **commit** it ran against, +- the **dataset** and the **external reference** used as an oracle, +- the **metric**, precisely, and **what the number is not comparable to**, +- **what remains untested**. + +That last section is the point. A benchmark result without its scope reads +as a general claim, and every number here is narrower than it looks — +these results say nothing about scanned documents or CJK text, for +instance. + +Reports are append-only. Re-running a benchmark adds a new dated file +rather than editing an old one, so a regression is visible as a diff +between two reports instead of vanishing into a rewrite. + +### Why findings live here and not only in the issue tracker + +An issue records that something was *decided*. A report records what was +*measured*, so the next person can tell whether a number still holds +without re-deriving it — and can see the caveats that made it honest. diff --git a/docs/evaluations/2026-08-02-font-metrics-and-table-fidelity.md b/docs/evaluations/2026-08-02-font-metrics-and-table-fidelity.md new file mode 100644 index 0000000..f2ede0c --- /dev/null +++ b/docs/evaluations/2026-08-02-font-metrics-and-table-fidelity.md @@ -0,0 +1,124 @@ +# Font metrics and table fidelity — audit and fixes + +**Date:** 2026-08-02 +**Commits:** `325e628` … `0ca65ca` (PRs #9, #13–#19) +**References:** rendered pages via `pdftoppm`, poppler `pdftotext -layout`, pdfplumber 0.11.9 +**Real-world corpus:** 3M Company 2018 Form 10-K (FinanceBench), pages 56–60 + +## Headline + +| | before | after | +|---|---|---| +| Horizontal (X) drift vs pdfplumber | 11.99pt max / 4.79pt mean | **0.0000pt** | +| Vertical (Y) drift vs pdfplumber | 4.968pt @24pt, 2.484pt @12pt | **0.0000pt** | +| Golden position tolerance | 15pt | **0.01pt, both axes** | +| Negative signs preserved (3M, 5 statements) | 83/103 (81%) | **103/103** | +| Font coverage in the test corpus | 1 font, 26 words | **12 fonts, 3 sizes, 170 words** | + +## Defects found and fixed + +### 1. Flat 500 width for the standard 14 fonts + +PDF 1.7 §9.6.2.2 lets the 14 standard fonts omit `/Widths`; a consumer is +expected to already know their metrics. pdftable did not, and fell through +to a flat 500/1000 guess for every glyph — `i` (222) and `m` (833) got +identical widths, and the error accumulated along each line. + +Because the `text` and `lines_strict` strategies infer column boundaries +from word positions, this was upstream of table structure, not merely +cosmetic. + +### 2. Symbol and ZapfDingbats decoded as Latin + +Both fonts carry their own built-in encoding and neither declares +`/Encoding`, so `readFont` handed them StandardEncoding. Symbol code +`0x61` decoded as `a` rather than `alpha` — a *text*-correctness failure, +not a metrics one. Any document using Symbol for Greek extracted nonsense. + +pdfplumber 0.11.9 still exhibits this; pdftable now does not. + +### 3. Glyph boxes sat `descent × size` too high + +Two independent causes, the second of which would have survived fixing the +first: + +- The same exemption that omits `/Widths` also omits `/FontDescriptor`, + so `Ascent`/`Descent` were 0 and the glyph box collapsed to + `[baseline, baseline+size]`. +- Descent was scaled by `0.001` but **not** by the font size, so even + fonts that *did* supply a descriptor were short by a factor of the font + size — 12× at 12pt. + +Measured error was exactly `0.207 × size`, matching Helvetica's −207/1000 +descender. Rows are derived from word Y extents, so this could merge or +split table rows. + +### 4. 19% of negative numbers lost their sign + +Cell assignment gives a glyph to whichever cell contains its centre. That +is correct at an *interior* boundary — it settles which of two candidates +owns a straddling glyph. At the table's *outer* edge there is no competing +cell, so the rule stopped disambiguating and started deleting. + +On 3M page 58 the `)` of `(16,048)` had its centre at x=537.879 while the +last column ended at x=537.871 — **a shortfall of 0.008pt**, about a +nine-thousandth of an inch. Accounting notation for −16,048 was read back +as +16,048. + +Across the five financial statements this flipped the sign of **20 of 103 +negatives (19%)** while leaving every magnitude correct — which is what +made it dangerous. A missing value is detectable downstream; a plausible +wrong sign is not. + +**pdfplumber has the same defect.** On the same row it returns `16,048` +without the closing paren. pdftable is now strictly better than its +reference here. + +### 5. Small type over-merged into single runs + +`DefaultWordOpts()` did not honour explicit space glyphs, so word +boundaries were inferred from inter-glyph gaps alone. At 8pt a space is +`278/1000 × 8 = 2.22pt`, under the 3pt `XTolerance`, so a whole line +collapsed into one word. Body type in real documents is routinely 8–9pt. + +pdfplumber ends a word *at* a whitespace glyph before any gap test runs. +Matching that reproduces its output word-for-word and coordinate-for-coordinate. + +## A finding that was wrong + +`ExtractText` returns `"Consolidated Balance Shee t"` for 3M page 58. This +was filed as a defect. **It is not** — that gap is in the source document. +The rendered page shows it, and poppler extracts it identically. 3M's +filing agent typeset it that way, which is common in SEC EDGAR documents. + +The error was methodological: the original comparison was pdftable's table +output against pdftable's *own* text output. Internal agreement proves the +two code paths disagree, not which one is right. Correcting it required an +external reference — a rendered page and a second extractor. + +**Any fidelity claim needs an oracle the project did not write.** + +## Verification method + +Findings were confirmed against three independent sources before being +accepted: + +1. the page rendered to PNG (`pdftoppm -r 150`) and read directly, +2. poppler `pdftotext -layout`, +3. pdfplumber 0.11.9. + +The citation geometry (`BBox.Viewport` / `BBox.Normalized`) was verified by +projecting real cell bboxes onto the rendered page and confirming the +highlights land on the intended rows. + +## What remains untested + +- **scanned PDFs** — no text layer at all; the geometric approach cannot + apply and an OCR path is required +- **CID/Type0 fonts** — CJK and modern subsetted fonts +- **embedded subset fonts** with their own `/Widths` — the common + production case +- **rotated and multi-column layouts** + +And separately, table *structure* accuracy, measured in the +[ICDAR 2013 evaluation](2026-08-02-icdar2013-table-structure.md). diff --git a/docs/evaluations/2026-08-02-icdar2013-table-structure.md b/docs/evaluations/2026-08-02-icdar2013-table-structure.md new file mode 100644 index 0000000..13ecccd --- /dev/null +++ b/docs/evaluations/2026-08-02-icdar2013-table-structure.md @@ -0,0 +1,90 @@ +# ICDAR 2013 — table detection and structure + +**Date:** 2026-08-02 +**Commit:** `0ca65ca` (after the font-metric and table-fidelity work of HAL-480/481/510/520/548/511) +**Harness:** [`bench/icdar2013`](../../bench/icdar2013/) — `python bench/icdar2013/run.py` +**Dataset:** ICDAR 2013 Table Competition, Smock's corrected edition — 125 born-digital PDFs, 39,524 ground-truth adjacency relations +**Reference:** pdfplumber 0.11.9 + +## Result + +| system | precision | recall | F1 | +| --- | --- | --- | --- | +| pdftable (`lines`) | 0.865 | 0.229 | **0.362** | +| pdfplumber (`lines`) | 0.868 | 0.235 | **0.370** | +| pdftable (fallback `lines`→`text`) | 0.223 | 0.557 | 0.318 | +| pdftable (fallback + `MergeSplitTokens`) | 0.471 | 0.275 | 0.347 | +| pdfplumber (`text`) | 0.167 | 0.679 | 0.267 | + +## Two findings + +### 1. Parity holds at the table level + +0.362 vs 0.370, with near-identical precision (0.865 vs 0.868). Everything +before this validated *text* fidelity — word positions, glyph widths, +signs. This is the first measurement of *table* behaviour, and pdftable +tracks its reference implementation there too. + +### 2. The bottleneck is detection, not structure + +``` +documents : 125 +documents where NO table was found : 28 (22%) +tables detected (total) : 306 + +restricted to documents where a table WAS detected: + precision 0.865 recall 0.409 F1 0.556 +``` + +Precision 0.865 says **what we extract is right**. Recall 0.229 says **we +miss three quarters of it**. Those are very different problems and they +need opposite work. + +The cause is structural. The `lines` strategy builds cells from +**intersecting** rulings. A table ruled only horizontally — booktabs +style, ubiquitous in government and academic documents — yields no +intersections and is invisible. Confirmed on `us-018.pdf`: 46 ruling lines +on page 1, zero rectangles, zero tables detected. + +The `text` strategy sees those tables (recall 0.56–0.68) but fabricates +tables on prose pages (precision 0.17–0.22). Neither library has a usable +"is this actually a table?" decision, which is why the naive fallback +scores *worse* overall than `lines` alone. + +## Caveat on the number + +**0.36 is not comparable to the 0.85–0.95 reported in table-structure +papers.** Those benchmarks hand the system the table region and score only +the gridding. This runs the harder end-to-end task. The 0.556 figure on +documents where detection succeeded is closer to a structure-only +comparison, and still contains intra-document detection misses. + +The honest one-line summary: **cell-level extraction is strong, table +detection is weak.** + +## What this implies + +Detection — *where is the table* — is a vision problem, and the thing +CV/VLM models are built for. Content fidelity — the values, signs and +coordinates — is a text-layer problem, where the deterministic parser is +provably better (it preserves accounting minus signs that pdfplumber +drops; see the [font-metrics evaluation](2026-08-02-font-metrics-and-table-fidelity.md)). + +That argues for splitting the work by which half is weak, rather than +replacing either: + +- **layout/VLM model → table region and row/column structure** (attacks + the 0.229 recall) +- **pdftable text layer → cell contents and coordinates** (keeps exact + values and citation geometry, which a generative model cannot provide) + +Two cheaper wins should be tried first, since that is where the points +are: + +1. Support horizontally-ruled-only tables — derive column edges from word + alignment *within* the rule band instead of requiring vertical rulings. +2. Give the `text` strategy a confidence signal, so a fallback can reject + prose instead of emitting a table for every page. + +Tracked as HAL-568. Re-run this harness after each and record a new dated +report here.