diff --git a/bench/README.md b/bench/README.md index d79e8d4..9af9a6f 100644 --- a/bench/README.md +++ b/bench/README.md @@ -13,6 +13,7 @@ Each harness downloads what it needs into a scratch directory. | Benchmark | Dataset | Measures | Report | | --- | --- | --- | --- | | [`icdar2013/`](icdar2013/) | ICDAR 2013 Table Competition (125 PDFs) | table detection + structure | [2026-08-02](../docs/evaluations/2026-08-02-icdar2013-table-structure.md) | +| [`icdar2013/oracle.py`](icdar2013/oracle.py) | same, with ground-truth boundaries | the ceiling a layout model could reach | [2026-08-03](../docs/evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md) | ## Why the numbers live in `docs/evaluations/` diff --git a/bench/icdar2013/oracle.py b/bench/icdar2013/oracle.py new file mode 100644 index 0000000..5c3308f --- /dev/null +++ b/bench/icdar2013/oracle.py @@ -0,0 +1,215 @@ +"""Measure the CEILING of a layout-model hybrid. + +Feeds pdftable the ground-truth row and column boundaries — the output a +perfect layout model would produce — and scores what comes back. pdftable +then does only the part it is good at: filling cells from the text layer +and reporting exact coordinates. + +This sizes the hybrid before anyone builds it. If extraction with perfect +boundaries scores near 1.0, every remaining point is a detection/structure +problem and a layout model buys all of it. If it scores 0.6, the ceiling +is far below the pitch and the geometry needs work first. + + python bench/icdar2013/oracle.py [limit] + +## Deriving the grid + +The obvious approach — collect every cell bounding-box edge and use them +all — does NOT work. Cells in different rows have slightly different +extents, so it yields dozens of near-duplicate boundaries and shreds the +table into fragments. The first version of this scored 0.119 F1 with +PERFECT input, i.e. it measured itself. + +Instead the grid comes from the ground truth's own logical indices. Each +cell carries start-col/end-col and start-row/end-row, so column c's extent +is the span of the cells that begin and end in it, and the boundary +between adjacent columns is the midpoint of the gap between them. That +yields exactly ncols+1 lines, which is what a grid actually is. + +Ground-truth boxes are in PDF points with the origin bottom-left — the +same space pdftable reports — verified against pdfplumber word positions +(GT y1=619.0 vs word y0=616.9 on eu-002; a box-vs-glyph difference, not a +flip). No conversion is needed. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import xml.etree.ElementTree as ET +from collections import Counter + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from score import norm, prf, relations_from_grid, score # noqa: E402 + + +def _boundaries(lo: dict[int, float], hi: dict[int, float]) -> list[float]: + """Grid lines from per-index extents: outer edges plus gap midpoints.""" + idx = sorted(set(lo) | set(hi)) + if len(idx) < 1: + return [] + out = [lo[idx[0]]] + for a, b in zip(idx, idx[1:]): + # Midpoint of the gutter between two adjacent bands. Using either + # edge alone would clip whichever side is wider on some rows. + out.append((hi[a] + lo[b]) / 2) + out.append(hi[idx[-1]]) + return sorted(out) + + +def oracle_edges(xml_path: str) -> dict[str, dict[str, list[float]]]: + root = ET.parse(xml_path).getroot() + + # One region per page only. Merging several tables on a page into one + # edge set produces a mega-grid spanning the gap between them. + per_page: Counter = Counter(r.get("page", "1") for r in root.iter("region")) + + pages: dict[str, dict[str, list[float]]] = {} + for region in root.iter("region"): + page = region.get("page", "1") + if per_page[page] != 1: + continue + + col_lo: dict[int, float] = {} + col_hi: dict[int, float] = {} + row_lo: dict[int, float] = {} + row_hi: dict[int, float] = {} + + for cell in region.findall("cell"): + bb = cell.find("bounding-box") + if bb is None: + continue + x1, x2 = float(bb.get("x1")), float(bb.get("x2")) + y1, y2 = float(bb.get("y1")), float(bb.get("y2")) + sc = int(cell.get("start-col", 0)) + ec = int(cell.get("end-col", sc)) + sr = int(cell.get("start-row", 0)) + er = int(cell.get("end-row", sr)) + + # Only cells that BEGIN in a band define its near edge, and + # only cells that END in it define the far edge — a spanning + # cell says nothing about the bands it merely crosses. + col_lo[sc] = min(col_lo.get(sc, x1), x1) + col_hi[ec] = max(col_hi.get(ec, x2), x2) + + # Row index 0 is the TOP row, and y grows upward, so row order + # is the reverse of y order. Negate to keep the index-ordered + # helper honest, then flip back. + row_lo[sr] = min(row_lo.get(sr, -y2), -y2) + row_hi[er] = max(row_hi.get(er, -y1), -y1) + + v = _boundaries(col_lo, col_hi) + h = sorted(-y for y in _boundaries(row_lo, row_hi)) + if len(v) >= 2 and len(h) >= 2: + pages[page] = {"v": v, "h": h} + return pages + + +def gt_relations_single_region(xml_path: str) -> Counter: + """Ground truth restricted to the pages the oracle actually covers. + + Scoring against ALL regions while only feeding boundaries for + single-region pages deflates recall with tables the experiment never + attempted — it would report the multi-region exclusion as an + extraction failure. Same class of mistake as the edge-clustering bug + above: the harness measuring itself. + """ + rels: Counter = Counter() + root = ET.parse(xml_path).getroot() + per_page: Counter = Counter(r.get("page", "1") for r in root.iter("region")) + for region in root.iter("region"): + if per_page[region.get("page", "1")] != 1: + continue + cells = [] + for cell in region.findall("cell"): + sr = int(cell.get("start-row", 0)) + sc = int(cell.get("start-col", 0)) + er = int(cell.get("end-row", sr)) + ec = int(cell.get("end-col", sc)) + content = cell.find("content") + cells.append((sr, sc, er, ec, + norm(content.text if content is not None else ""))) + if not cells: + continue + nrows = max(c[2] for c in cells) + 1 + ncols = max(c[3] for c in cells) + 1 + grid = [["" for _ in range(ncols)] for _ in range(nrows)] + for sr, sc, er, ec, text in cells: + for r in range(sr, er + 1): + for c in range(sc, ec + 1): + if r < nrows and c < ncols: + grid[r][c] = text + rels += relations_from_grid(grid) + return rels + + +def main() -> int: + root, exe = sys.argv[1], sys.argv[2] + limit = int(sys.argv[3]) if len(sys.argv) > 3 else 0 + + pairs = [] + for dirpath, _, files in os.walk(root): + for f in sorted(files): + if f.endswith("-str.xml"): + pdf = os.path.join(dirpath, f.replace("-str.xml", ".pdf")) + if os.path.exists(pdf): + pairs.append((pdf, os.path.join(dirpath, f))) + pairs.sort() + if limit: + pairs = pairs[:limit] + + # Scored over the pages the oracle can actually describe (single-region + # pages). Documents with none contribute nothing either way, so the + # number is not diluted by pages the experiment says nothing about. + print(f"scoring {len(pairs)} documents with ORACLE boundaries\n", flush=True) + + variants = {"oracle": [], "oracle + MergeSplitTokens": ["-merge"]} + totals = {k: [0, 0, 0] for k in variants} + scored_docs = 0 + + for i, (pdf, xml) in enumerate(pairs, 1): + edges = oracle_edges(xml) + if not edges: + continue + scored_docs += 1 + gt = gt_relations_single_region(xml) + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh: + json.dump(edges, fh) + path = fh.name + try: + for name, extra in variants.items(): + try: + out = subprocess.run( + [exe, "-oracle", path] + extra + [pdf], + capture_output=True, timeout=180).stdout + tables = json.loads(out or b"[]") + except Exception: + tables = [] + rels: Counter = Counter() + for t in tables: + rels += relations_from_grid( + [[norm(c) for c in r] for r in t["rows"]]) + c, nd, ng = score(gt, rels) + totals[name][0] += c + totals[name][1] += nd + totals[name][2] += ng + finally: + os.unlink(path) + if i % 20 == 0: + print(f" ...{i}/{len(pairs)}", flush=True) + + print(f"\n{'system':<40} {'precision':>10} {'recall':>10} {'F1':>10}") + print("-" * 74) + for name in variants: + p, r, f = prf(*totals[name]) + print(f"{'pdftable + ' + name:<40} {p:>10.3f} {r:>10.3f} {f:>10.3f}") + print(f"\ndocuments scored: {scored_docs} (single-region pages only)") + print("This is the ceiling a perfect layout model could hand pdftable.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/README.md b/docs/README.md index dceb3ac..37e9184 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,7 @@ release history in [`CHANGELOG.md`](../CHANGELOG.md). | date | subject | headline | | --- | --- | --- | +| [2026-08-03](evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md) | hybrid ceiling with oracle boundaries | **0.362 → 0.935.** Given a correct grid, extraction is near-perfect — structure is the whole gap | | [2026-08-02](evaluations/2026-08-02-icdar2013-table-structure.md) | ICDAR 2013 table detection + structure | F1 0.362 end-to-end; the bottleneck is **detection**, not cell accuracy | | [2026-08-02](evaluations/2026-08-02-strategy-auto-negative-result.md) | `StrategyAuto` for one-axis-ruled tables | **negative result** — detection 22%→18% missed, but F1 0.362→0.358. Shipped opt-in only | | [2026-08-02](evaluations/2026-08-02-font-metrics-and-table-fidelity.md) | font metrics and table fidelity | position drift 11.99pt → **0.0000pt**; negative-sign loss 19% → **0%** | diff --git a/docs/evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md b/docs/evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md new file mode 100644 index 0000000..93a3830 --- /dev/null +++ b/docs/evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md @@ -0,0 +1,71 @@ +# The hybrid ceiling — pdftable with perfect table boundaries + +**Date:** 2026-08-03 +**Commit:** `0b39b95` +**Harness:** [`bench/icdar2013/oracle.py`](../../bench/icdar2013/oracle.py) +**Question:** if a layout model supplied correct rows and columns, how good would extraction be? That number decides whether the model is worth deploying. + +## Result + +| system | precision | recall | F1 | +| --- | --- | --- | --- | +| pdftable, current best (`lines`) | 0.865 | 0.229 | **0.362** | +| **pdftable + ORACLE boundaries** | **0.948** | **0.922** | **0.935** | +| pdftable + oracle + `MergeSplitTokens` | 0.806 | 0.660 | 0.726 | + +107 documents (single-region pages). + +## What it means + +**0.362 → 0.935.** Given the right grid, pdftable extracts almost perfectly. + +So essentially the entire gap in the end-to-end score is **table structure**, not text extraction. The cell-filling, the coordinates, the text fidelity — all the work of the last two days — is not the limiting factor. Finding the rows and columns is. + +That is a clear verdict on the hybrid: **a layout model that outputs row/column structure is worth deploying.** It converts almost the whole gap. + +It also confirms the division of labour. pdftable keeps what a generative model cannot give: exact cell text and exact coordinates for citation highlighting. The model supplies only geometry. + +## `MergeSplitTokens` must be OFF when boundaries come from a model + +0.935 → 0.726 with it on. That is not a small regression and the reason is structural: the setting exists to repair columns that a *geometric* boundary guess cut through a value. Given a correct grid there is nothing to repair, so every merge it performs destroys a correct cell. + +**Rule: explicit boundaries from a model ⇒ `MergeSplitTokens = false`.** It remains useful for the pure-geometry path where boundaries are inferred. + +## Integration point — already present + +No new dependency and no HTTP client in the library: + +```go +s := pdftable.DefaultTableSettings() +s.VerticalStrategy = pdftable.StrategyExplicit +s.HorizontalStrategy = pdftable.StrategyExplicit +s.ExplicitVerticalLines = colBoundaries // from the layout model +s.ExplicitHorizontalLines = rowBoundaries +s.MergeSplitTokens = false // see above +tables, _ := page.ExtractTables(s) +``` + +The caller owns the model call. pdftable stays deterministic and offline. + +## Two harness bugs, both worth recording + +The first version of this experiment reported **0.119 F1 with perfect input** — near-random, and obviously measuring itself rather than the extractor. Two causes, and both are the same mistake in different clothes: + +1. **Every cell bounding-box edge was treated as a grid line.** Cells in adjacent rows have slightly different extents, so this produced dozens of near-duplicate boundaries and shredded each table into fragments. Fixed by deriving the grid from the ground truth's own logical `start-col`/`end-col` indices: a column's extent is the span of the cells that begin and end in it, and the boundary between two columns is the midpoint of the gutter. That yields exactly `ncols+1` lines. +2. **Ground truth was counted for pages the oracle never attempted.** Boundaries were only supplied for single-region pages, but recall was scored against every region in the document — reporting an exclusion the experiment had chosen as an extraction failure. Fixed by restricting ground truth to the same pages. + +Fixing (1) moved 0.119 → 0.782; fixing (2) moved 0.782 → 0.935. + +A third suspicion turned out to be unfounded: the ground-truth Y origin was checked against pdfplumber word positions and is bottom-left, the same space pdftable reports (GT `y1=619.0` vs word `y0=616.9` on `eu-002` — a box-versus-glyph difference, not a flip). No conversion needed. + +**The lesson is the same one from the font work:** a measurement that disagrees violently with expectation is far more likely to be a broken measurement than a broken system. Both times, checking the harness against a known-good case found the fault in the harness. + +## Scope + +Single-region pages only — 107 of 125 documents. Pages carrying several tables are excluded because merging their regions into one edge set produces a grid spanning the gap between them, which measures the harness again. A real layout model would emit one region per table, so this is a fair proxy for the hybrid, but it is not a measurement of multi-table pages. + +## Reproduce + +```sh +python bench/icdar2013/oracle.py ~/.cache/pdftable-bench/ICDAR-2013-Table-Competition-Corrected +```