docs: add bench/ and docs/evaluations/, with the ICDAR 2013 result

README.md

@@ -603,12 +603,38 @@ stdlib-only.
   and a `pdftable` CLI mirroring pdfplumber's surface.
 - `v0.4.x` — bundle the standard-14 AFM metrics so word bboxes (and
   therefore cell text) match pdfplumber on standard fonts. **Done**: the
-  Adobe Core 14 metrics ship, and Symbol/ZapfDingbats decode with their
-  own built-in encodings. The golden position envelope is still asserted
-  at 15pt pending a re-measure, so the "within 1 point" claim is not yet
-  evidenced.
-- `v0.5.x` — performance pass: parser benchmarking against
-  pdfminer.six and pdfplumber on a representative document corpus.
+  Adobe Core 14 metrics ship, Symbol/ZapfDingbats decode with their own
+  built-in encodings, and measured position drift against pdfplumber is
+  **0.0000pt on both axes** — the golden envelope is now asserted at
+  0.01pt. See
+  [the evaluation](docs/evaluations/2026-08-02-font-metrics-and-table-fidelity.md).
+- `v0.5.x` — **table detection**. The ICDAR 2013 benchmark puts
+  end-to-end F1 at 0.362, level with pdfplumber's 0.370, and the
+  diagnostic is unambiguous: precision 0.865, recall 0.229. What we
+  extract is right; we miss three quarters of the tables, because the
+  `lines` strategy needs *intersecting* rulings and a horizontally-ruled
+  table produces none. See
+  [the evaluation](docs/evaluations/2026-08-02-icdar2013-table-structure.md).
+- `v0.6.x` — performance pass: parser speed against pdfminer.six and
+  pdfplumber on a representative corpus.
+
+## Repository layout
+
+`pdftable` is a single Go package, so its source lives in the repository
+root — that is the import path, and Go keeps `_test.go` files next to the
+code they cover.
+
+| | |
+| --- | --- |
+| `*.go` | the library |
+| `internal/pdf/` | content-stream interpreter, fonts, encodings |
+| `internal/layout/` | edge/intersection/cell geometry |
+| `cmd/pdftable/` | CLI |
+| `bench/` | accuracy benchmarks against public datasets — a **separate Go module**, so it never adds a dependency here |
+| `docs/evaluations/` | dated measurement reports, with their caveats |
+| `scripts/` | fixture and golden-file generators |
+| `testdata/golden/` | fixtures compared against pdfplumber |
+| `testdata/fonts/` | fixtures where pdfplumber is wrong, asserted directly |
 
 ## License
 

bench/README.md

@@ -0,0 +1,33 @@
+# Benchmarks
+
+Accuracy benchmarks against public datasets with published ground truth.
+
+These are **not** `go test` benchmarks (those live next to the code as
+`*_bench_test.go` and measure speed). These measure **correctness** against
+an external reference, and they are deliberately outside the library's Go
+module so their dependencies never reach a consumer of `pdftable`.
+
+Datasets are **not committed** — they are large and separately licensed.
+Each harness downloads what it needs into a scratch directory.
+
+| Benchmark | Dataset | Measures | Report |
+| --- | --- | --- | --- |
+| [`icdar2013/`](icdar2013/) | ICDAR 2013 Table Competition (125 PDFs) | table detection + structure | [2026-08-02](../docs/evaluations/2026-08-02-icdar2013-table-structure.md) |
+
+## Why the numbers live in `docs/evaluations/`
+
+A benchmark result is only meaningful with its date, the code version it
+ran against, and the caveats on the metric. A bare number in a README rots
+the moment either side changes. Every run gets a dated report; the table
+above links to the most recent.
+
+## Adding a benchmark
+
+1. `bench/<name>/` with its own `go.mod` if it needs Go.
+2. A `run.py` that fetches the dataset and prints a result table.
+3. A dated report in `docs/evaluations/`, and a row in the table above.
+
+State the metric precisely, and say what it is **not** comparable to.
+Published table-extraction scores in particular are usually
+structure-only — the system is handed the table location — and are not
+comparable to an end-to-end number.

bench/go.mod

@@ -0,0 +1,11 @@
+// Separate module on purpose: the benchmark harness must never add a
+// dependency to the library it measures. Nested modules are excluded from
+// the parent's ./... , so go build ./... at the repo root ignores this
+// entirely and a consumer of pdftable never fetches it.
+module github.com/hallelx2/pdftable/bench
+
+go 1.25.0
+
+require github.com/hallelx2/pdftable v0.0.0
+
+replace github.com/hallelx2/pdftable => ../

bench/icdar2013/README.md

@@ -0,0 +1,57 @@
+# ICDAR 2013 Table Competition benchmark
+
+Measures **table detection + structure recognition** end to end against
+125 born-digital PDFs from EU and US government sources, with per-cell
+ground truth.
+
+```sh
+pip install pdfplumber
+python bench/icdar2013/run.py            # full run, ~10 min
+python bench/icdar2013/run.py --limit 5  # quick check while iterating
+python bench/icdar2013/run.py --diag     # detection vs structure breakdown
+```
+
+The dataset (~12 MB) downloads to `~/.cache/pdftable-bench` (override with
+`PDFTABLE_BENCH_DIR`). It is not committed.
+
+## Metric: adjacency relations
+
+From Göbel et al., the metric the competition itself used. For every
+non-empty cell, take its nearest non-empty neighbour to the right and
+below; each pair is a relation `(text_a, text_b, direction)`. Score the
+detected multiset against ground truth.
+
+Cell-by-cell grid comparison would be the obvious alternative and is
+worse: two tools can grid the same table differently — one emits a spacer
+column, the other does not — and still convey identical structure. A grid
+diff calls that a failure. Adjacency asks the question a reader actually
+cares about: *is this value next to that label?* It is unforgiving about
+genuinely wrong structure and forgiving about harmless disagreements.
+
+Documents are scored whole, with relations unioned across pages and
+tables. That sidesteps matching detected tables to ground-truth tables, a
+step which would need its own arbitrary thresholds and would make the
+number depend on them.
+
+## Reading the result
+
+**This is an end-to-end number and is NOT comparable to published ICDAR
+2013 scores of 0.85–0.95.** Those evaluate structure recognition with the
+table region *given*; this has to find the table first. Compare runs of
+this harness to each other, and to the pdfplumber column, not to papers.
+
+`--diag` splits the two failure modes, which need opposite work:
+
+- **the table was never detected** — a detection problem
+- **it was detected but gridded wrongly** — a structure problem
+
+Latest results: [`docs/evaluations/`](../../docs/evaluations/).
+
+## Files
+
+| | |
+| --- | --- |
+| `run.py` | fetches the dataset, builds the extractor, runs everything |
+| `extract.go` | dumps every table pdftable finds as JSON; built as its own module so the benchmark adds no dependency to the library |
+| `score.py` | parses ground-truth XML, computes precision/recall/F1 |
+| `diag.py` | separates detection failures from structure failures |

bench/icdar2013/diag.py

@@ -0,0 +1,86 @@
+"""Separate the two ways table extraction can fail on ICDAR 2013.
+
+Low recall has two very different causes and the headline F1 cannot tell
+them apart:
+
+  (a) the table was never detected at all — a detection problem
+  (b) it was detected but gridded wrongly — a structure problem
+
+(a) is fixed by better "is this a table?" logic; (b) by better cell
+geometry. They need opposite work, so it is worth knowing which one we
+have before touching anything.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+from collections import Counter
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from score import gt_relations, norm, relations_from_grid, score  # noqa: E402
+
+
+def main() -> int:
+    root, exe = sys.argv[1], sys.argv[2]
+    pairs = []
+    for dirpath, _, files in os.walk(root):
+        for f in sorted(files):
+            if f.endswith("-str.xml"):
+                pdf = os.path.join(dirpath, f.replace("-str.xml", ".pdf"))
+                if os.path.exists(pdf):
+                    pairs.append((pdf, os.path.join(dirpath, f)))
+    pairs.sort()
+
+    zero_detect = 0
+    gt_regions_total = 0
+    detected_tables_total = 0
+    pages_total = 0
+    per_doc = []
+
+    for pdf, xml in pairs:
+        gt = gt_relations(xml)
+        out = subprocess.run(
+            [exe, "-strategy", "lines", pdf], capture_output=True, timeout=120
+        ).stdout
+        tables = json.loads(out or b"[]")
+        detected_tables_total += len(tables)
+        rels: Counter = Counter()
+        for t in tables:
+            rels += relations_from_grid([[norm(c) for c in r] for r in t["rows"]])
+        c, nd, ng = score(gt, rels)
+        if nd == 0:
+            zero_detect += 1
+        gt_regions_total += 1
+        per_doc.append((os.path.basename(pdf), ng, nd, c, len(tables)))
+
+    print(f"documents                     : {len(pairs)}")
+    print(f"documents where we found NO table: {zero_detect}"
+          f"  ({100*zero_detect/len(pairs):.0f}%)")
+    print(f"tables detected (total)       : {detected_tables_total}")
+    print()
+
+    # Restrict to documents where we DID detect something: if structure is
+    # good, precision AND recall should both be high on this subset.
+    tot = [0, 0, 0]
+    for _, ng, nd, c, _ in per_doc:
+        if nd:
+            tot[0] += c
+            tot[1] += nd
+            tot[2] += ng
+    p = tot[0] / tot[1] if tot[1] else 0
+    r = tot[0] / tot[2] if tot[2] else 0
+    f = 2 * p * r / (p + r) if (p + r) else 0
+    print("--- documents where a table WAS detected ---")
+    print(f"  precision {p:.3f}   recall {r:.3f}   F1 {f:.3f}")
+    print()
+    print("worst 8 documents by missed relations:")
+    for name, ng, nd, c, nt in sorted(per_doc, key=lambda x: x[1] - x[3])[-8:]:
+        print(f"  {name:<18} gt={ng:<6} detected={nd:<6} correct={c:<6} tables={nt}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

bench/icdar2013/extract.go

@@ -0,0 +1,70 @@
+// Emit every table pdftable finds in a PDF, as JSON, for benchmarking.
+package main
+
+import (
+	"encoding/json"
+	"flag"
+	"fmt"
+	"os"
+
+	"github.com/hallelx2/pdftable"
+)
+
+type tableOut struct {
+	Page int        `json:"page"`
+	Rows [][]string `json:"rows"`
+}
+
+func main() {
+	strategy := flag.String("strategy", "lines", "lines | text | fallback")
+	merge := flag.Bool("merge", false, "TableSettings.MergeSplitTokens")
+	flag.Parse()
+
+	doc, err := pdftable.OpenFile(flag.Arg(0))
+	if err != nil {
+		// A file we cannot open scores zero rather than aborting the run;
+		// the harness needs a result for every document.
+		json.NewEncoder(os.Stdout).Encode([]tableOut{})
+		fmt.Fprintln(os.Stderr, "open:", err)
+		return
+	}
+	defer doc.Close()
+
+	mk := func(v, h pdftable.TableStrategy) pdftable.TableSettings {
+		s := pdftable.DefaultTableSettings()
+		s.VerticalStrategy, s.HorizontalStrategy = v, h
+		s.MergeSplitTokens = *merge
+		return s
+	}
+	lines := mk(pdftable.StrategyLines, pdftable.StrategyLines)
+	text := mk(pdftable.StrategyText, pdftable.StrategyText)
+
+	var attempts []pdftable.TableSettings
+	switch *strategy {
+	case "lines":
+		attempts = []pdftable.TableSettings{lines}
+	case "text":
+		attempts = []pdftable.TableSettings{text}
+	default: // fallback: ruled cells first, whitespace alignment if none
+		attempts = []pdftable.TableSettings{lines, text}
+	}
+
+	out := []tableOut{}
+	for i := 1; i <= doc.NumPages(); i++ {
+		p, err := doc.Page(i)
+		if err != nil {
+			continue
+		}
+		for _, s := range attempts {
+			tables, err := p.ExtractTables(s)
+			if err != nil || len(tables) == 0 {
+				continue
+			}
+			for _, t := range tables {
+				out = append(out, tableOut{Page: i, Rows: t.Rows})
+			}
+			break
+		}
+	}
+	json.NewEncoder(os.Stdout).Encode(out)
+}