bench: measure the hybrid ceiling — 0.362 to 0.935 with correct boundaries

bench/README.md

@@ -13,6 +13,7 @@ Each harness downloads what it needs into a scratch directory.
 | Benchmark | Dataset | Measures | Report |
 | --- | --- | --- | --- |
 | [`icdar2013/`](icdar2013/) | ICDAR 2013 Table Competition (125 PDFs) | table detection + structure | [2026-08-02](../docs/evaluations/2026-08-02-icdar2013-table-structure.md) |
+| [`icdar2013/oracle.py`](icdar2013/oracle.py) | same, with ground-truth boundaries | the ceiling a layout model could reach | [2026-08-03](../docs/evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md) |
 
 ## Why the numbers live in `docs/evaluations/`
 

bench/icdar2013/oracle.py

@@ -0,0 +1,215 @@
+"""Measure the CEILING of a layout-model hybrid.
+
+Feeds pdftable the ground-truth row and column boundaries — the output a
+perfect layout model would produce — and scores what comes back. pdftable
+then does only the part it is good at: filling cells from the text layer
+and reporting exact coordinates.
+
+This sizes the hybrid before anyone builds it. If extraction with perfect
+boundaries scores near 1.0, every remaining point is a detection/structure
+problem and a layout model buys all of it. If it scores 0.6, the ceiling
+is far below the pitch and the geometry needs work first.
+
+    python bench/icdar2013/oracle.py <dataset-root> <extractor-exe> [limit]
+
+## Deriving the grid
+
+The obvious approach — collect every cell bounding-box edge and use them
+all — does NOT work. Cells in different rows have slightly different
+extents, so it yields dozens of near-duplicate boundaries and shreds the
+table into fragments. The first version of this scored 0.119 F1 with
+PERFECT input, i.e. it measured itself.
+
+Instead the grid comes from the ground truth's own logical indices. Each
+cell carries start-col/end-col and start-row/end-row, so column c's extent
+is the span of the cells that begin and end in it, and the boundary
+between adjacent columns is the midpoint of the gap between them. That
+yields exactly ncols+1 lines, which is what a grid actually is.
+
+Ground-truth boxes are in PDF points with the origin bottom-left — the
+same space pdftable reports — verified against pdfplumber word positions
+(GT y1=619.0 vs word y0=616.9 on eu-002; a box-vs-glyph difference, not a
+flip). No conversion is needed.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import xml.etree.ElementTree as ET
+from collections import Counter
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from score import norm, prf, relations_from_grid, score  # noqa: E402
+
+
+def _boundaries(lo: dict[int, float], hi: dict[int, float]) -> list[float]:
+    """Grid lines from per-index extents: outer edges plus gap midpoints."""
+    idx = sorted(set(lo) | set(hi))
+    if len(idx) < 1:
+        return []
+    out = [lo[idx[0]]]
+    for a, b in zip(idx, idx[1:]):
+        # Midpoint of the gutter between two adjacent bands. Using either
+        # edge alone would clip whichever side is wider on some rows.
+        out.append((hi[a] + lo[b]) / 2)
+    out.append(hi[idx[-1]])
+    return sorted(out)
+
+
+def oracle_edges(xml_path: str) -> dict[str, dict[str, list[float]]]:
+    root = ET.parse(xml_path).getroot()
+
+    # One region per page only. Merging several tables on a page into one
+    # edge set produces a mega-grid spanning the gap between them.
+    per_page: Counter = Counter(r.get("page", "1") for r in root.iter("region"))
+
+    pages: dict[str, dict[str, list[float]]] = {}
+    for region in root.iter("region"):
+        page = region.get("page", "1")
+        if per_page[page] != 1:
+            continue
+
+        col_lo: dict[int, float] = {}
+        col_hi: dict[int, float] = {}
+        row_lo: dict[int, float] = {}
+        row_hi: dict[int, float] = {}
+
+        for cell in region.findall("cell"):
+            bb = cell.find("bounding-box")
+            if bb is None:
+                continue
+            x1, x2 = float(bb.get("x1")), float(bb.get("x2"))
+            y1, y2 = float(bb.get("y1")), float(bb.get("y2"))
+            sc = int(cell.get("start-col", 0))
+            ec = int(cell.get("end-col", sc))
+            sr = int(cell.get("start-row", 0))
+            er = int(cell.get("end-row", sr))
+
+            # Only cells that BEGIN in a band define its near edge, and
+            # only cells that END in it define the far edge — a spanning
+            # cell says nothing about the bands it merely crosses.
+            col_lo[sc] = min(col_lo.get(sc, x1), x1)
+            col_hi[ec] = max(col_hi.get(ec, x2), x2)
+
+            # Row index 0 is the TOP row, and y grows upward, so row order
+            # is the reverse of y order. Negate to keep the index-ordered
+            # helper honest, then flip back.
+            row_lo[sr] = min(row_lo.get(sr, -y2), -y2)
+            row_hi[er] = max(row_hi.get(er, -y1), -y1)
+
+        v = _boundaries(col_lo, col_hi)
+        h = sorted(-y for y in _boundaries(row_lo, row_hi))
+        if len(v) >= 2 and len(h) >= 2:
+            pages[page] = {"v": v, "h": h}
+    return pages
+
+
+def gt_relations_single_region(xml_path: str) -> Counter:
+    """Ground truth restricted to the pages the oracle actually covers.
+
+    Scoring against ALL regions while only feeding boundaries for
+    single-region pages deflates recall with tables the experiment never
+    attempted — it would report the multi-region exclusion as an
+    extraction failure. Same class of mistake as the edge-clustering bug
+    above: the harness measuring itself.
+    """
+    rels: Counter = Counter()
+    root = ET.parse(xml_path).getroot()
+    per_page: Counter = Counter(r.get("page", "1") for r in root.iter("region"))
+    for region in root.iter("region"):
+        if per_page[region.get("page", "1")] != 1:
+            continue
+        cells = []
+        for cell in region.findall("cell"):
+            sr = int(cell.get("start-row", 0))
+            sc = int(cell.get("start-col", 0))
+            er = int(cell.get("end-row", sr))
+            ec = int(cell.get("end-col", sc))
+            content = cell.find("content")
+            cells.append((sr, sc, er, ec,
+                          norm(content.text if content is not None else "")))
+        if not cells:
+            continue
+        nrows = max(c[2] for c in cells) + 1
+        ncols = max(c[3] for c in cells) + 1
+        grid = [["" for _ in range(ncols)] for _ in range(nrows)]
+        for sr, sc, er, ec, text in cells:
+            for r in range(sr, er + 1):
+                for c in range(sc, ec + 1):
+                    if r < nrows and c < ncols:
+                        grid[r][c] = text
+        rels += relations_from_grid(grid)
+    return rels
+
+
+def main() -> int:
+    root, exe = sys.argv[1], sys.argv[2]
+    limit = int(sys.argv[3]) if len(sys.argv) > 3 else 0
+
+    pairs = []
+    for dirpath, _, files in os.walk(root):
+        for f in sorted(files):
+            if f.endswith("-str.xml"):
+                pdf = os.path.join(dirpath, f.replace("-str.xml", ".pdf"))
+                if os.path.exists(pdf):
+                    pairs.append((pdf, os.path.join(dirpath, f)))
+    pairs.sort()
+    if limit:
+        pairs = pairs[:limit]
+
+    # Scored over the pages the oracle can actually describe (single-region
+    # pages). Documents with none contribute nothing either way, so the
+    # number is not diluted by pages the experiment says nothing about.
+    print(f"scoring {len(pairs)} documents with ORACLE boundaries\n", flush=True)
+
+    variants = {"oracle": [], "oracle + MergeSplitTokens": ["-merge"]}
+    totals = {k: [0, 0, 0] for k in variants}
+    scored_docs = 0
+
+    for i, (pdf, xml) in enumerate(pairs, 1):
+        edges = oracle_edges(xml)
+        if not edges:
+            continue
+        scored_docs += 1
+        gt = gt_relations_single_region(xml)
+        with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh:
+            json.dump(edges, fh)
+            path = fh.name
+        try:
+            for name, extra in variants.items():
+                try:
+                    out = subprocess.run(
+                        [exe, "-oracle", path] + extra + [pdf],
+                        capture_output=True, timeout=180).stdout
+                    tables = json.loads(out or b"[]")
+                except Exception:
+                    tables = []
+                rels: Counter = Counter()
+                for t in tables:
+                    rels += relations_from_grid(
+                        [[norm(c) for c in r] for r in t["rows"]])
+                c, nd, ng = score(gt, rels)
+                totals[name][0] += c
+                totals[name][1] += nd
+                totals[name][2] += ng
+        finally:
+            os.unlink(path)
+        if i % 20 == 0:
+            print(f"  ...{i}/{len(pairs)}", flush=True)
+
+    print(f"\n{'system':<40} {'precision':>10} {'recall':>10} {'F1':>10}")
+    print("-" * 74)
+    for name in variants:
+        p, r, f = prf(*totals[name])
+        print(f"{'pdftable + ' + name:<40} {p:>10.3f} {r:>10.3f} {f:>10.3f}")
+    print(f"\ndocuments scored: {scored_docs} (single-region pages only)")
+    print("This is the ceiling a perfect layout model could hand pdftable.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docs/README.md

@@ -11,6 +11,7 @@ release history in [`CHANGELOG.md`](../CHANGELOG.md).
 
 | date | subject | headline |
 | --- | --- | --- |
+| [2026-08-03](evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md) | hybrid ceiling with oracle boundaries | **0.362 → 0.935.** Given a correct grid, extraction is near-perfect — structure is the whole gap |
 | [2026-08-02](evaluations/2026-08-02-icdar2013-table-structure.md) | ICDAR 2013 table detection + structure | F1 0.362 end-to-end; the bottleneck is **detection**, not cell accuracy |
 | [2026-08-02](evaluations/2026-08-02-strategy-auto-negative-result.md) | `StrategyAuto` for one-axis-ruled tables | **negative result** — detection 22%→18% missed, but F1 0.362→0.358. Shipped opt-in only |
 | [2026-08-02](evaluations/2026-08-02-font-metrics-and-table-fidelity.md) | font metrics and table fidelity | position drift 11.99pt → **0.0000pt**; negative-sign loss 19% → **0%** |

docs/evaluations/2026-08-03-hybrid-ceiling-oracle-boundaries.md

@@ -0,0 +1,71 @@
+# The hybrid ceiling — pdftable with perfect table boundaries
+
+**Date:** 2026-08-03
+**Commit:** `0b39b95`
+**Harness:** [`bench/icdar2013/oracle.py`](../../bench/icdar2013/oracle.py)
+**Question:** if a layout model supplied correct rows and columns, how good would extraction be? That number decides whether the model is worth deploying.
+
+## Result
+
+| system | precision | recall | F1 |
+| --- | --- | --- | --- |
+| pdftable, current best (`lines`) | 0.865 | 0.229 | **0.362** |
+| **pdftable + ORACLE boundaries** | **0.948** | **0.922** | **0.935** |
+| pdftable + oracle + `MergeSplitTokens` | 0.806 | 0.660 | 0.726 |
+
+107 documents (single-region pages).
+
+## What it means
+
+**0.362 → 0.935.** Given the right grid, pdftable extracts almost perfectly.
+
+So essentially the entire gap in the end-to-end score is **table structure**, not text extraction. The cell-filling, the coordinates, the text fidelity — all the work of the last two days — is not the limiting factor. Finding the rows and columns is.
+
+That is a clear verdict on the hybrid: **a layout model that outputs row/column structure is worth deploying.** It converts almost the whole gap.
+
+It also confirms the division of labour. pdftable keeps what a generative model cannot give: exact cell text and exact coordinates for citation highlighting. The model supplies only geometry.
+
+## `MergeSplitTokens` must be OFF when boundaries come from a model
+
+0.935 → 0.726 with it on. That is not a small regression and the reason is structural: the setting exists to repair columns that a *geometric* boundary guess cut through a value. Given a correct grid there is nothing to repair, so every merge it performs destroys a correct cell.
+
+**Rule: explicit boundaries from a model ⇒ `MergeSplitTokens = false`.** It remains useful for the pure-geometry path where boundaries are inferred.
+
+## Integration point — already present
+
+No new dependency and no HTTP client in the library:
+
+```go
+s := pdftable.DefaultTableSettings()
+s.VerticalStrategy   = pdftable.StrategyExplicit
+s.HorizontalStrategy = pdftable.StrategyExplicit
+s.ExplicitVerticalLines   = colBoundaries // from the layout model
+s.ExplicitHorizontalLines = rowBoundaries
+s.MergeSplitTokens = false                // see above
+tables, _ := page.ExtractTables(s)
+```
+
+The caller owns the model call. pdftable stays deterministic and offline.
+
+## Two harness bugs, both worth recording
+
+The first version of this experiment reported **0.119 F1 with perfect input** — near-random, and obviously measuring itself rather than the extractor. Two causes, and both are the same mistake in different clothes:
+
+1. **Every cell bounding-box edge was treated as a grid line.** Cells in adjacent rows have slightly different extents, so this produced dozens of near-duplicate boundaries and shredded each table into fragments. Fixed by deriving the grid from the ground truth's own logical `start-col`/`end-col` indices: a column's extent is the span of the cells that begin and end in it, and the boundary between two columns is the midpoint of the gutter. That yields exactly `ncols+1` lines.
+2. **Ground truth was counted for pages the oracle never attempted.** Boundaries were only supplied for single-region pages, but recall was scored against every region in the document — reporting an exclusion the experiment had chosen as an extraction failure. Fixed by restricting ground truth to the same pages.
+
+Fixing (1) moved 0.119 → 0.782; fixing (2) moved 0.782 → 0.935.
+
+A third suspicion turned out to be unfounded: the ground-truth Y origin was checked against pdfplumber word positions and is bottom-left, the same space pdftable reports (GT `y1=619.0` vs word `y0=616.9` on `eu-002` — a box-versus-glyph difference, not a flip). No conversion needed.
+
+**The lesson is the same one from the font work:** a measurement that disagrees violently with expectation is far more likely to be a broken measurement than a broken system. Both times, checking the harness against a known-good case found the fault in the harness.
+
+## Scope
+
+Single-region pages only — 107 of 125 documents. Pages carrying several tables are excluded because merging their regions into one edge set produces a grid spanning the gap between them, which measures the harness again. A real layout model would emit one region per table, so this is a fair proxy for the hybrid, but it is not a measurement of multi-table pages.
+
+## Reproduce
+
+```sh
+python bench/icdar2013/oracle.py ~/.cache/pdftable-bench/ICDAR-2013-Table-Competition-Corrected <extractor>
+```