Skip to content

share: a clearer board card that names its repo - #89

Merged
davanstrien merged 9 commits into
mainfrom
feat/board-card
Nov 9, 2026
Merged

davanstrien merged 9 commits into
mainfrom
feat/board-card

Conversation

@davanstrien

Copy link
Copy Markdown
Owner

Makes the board card that ocr-bench share writes easier to read and to use, after a first look at the v0 government-documents board.

Card

  • Starts with what the board is, a link to ocr-bench, and a short "how to read it" (rating, 95% interval, Pages, why not to compare wins).
  • Columns are Rating and 95% interval (a Bradley-Terry fit on an Elo-like scale), and Size.
  • Model names that are Hub repo ids link to their model page. baidu/Unlimited-OCR-2 (tiled) links the id and keeps the label as text.
  • The ⚠ note gives each model's number of error outputs; a FAILED row has a note.
  • Slice labels and the slice column are shown as code. The wording no longer says "slice board".
  • The settings section keeps the judge, the criteria in words, the sample, the comparisons and the judge date. The prompt hash, caps and fingerprint stay in board.json.
  • A "Use the data" section has load_dataset lines and a collapsed DuckDB block (overall board, top model per label) that name the board repo.

Behaviour

  • plan_share(..., board_repo=) is required; board.json records it as repo. push_share refuses another destination. A board repo that is the results repo or the source is refused when the plan is built, so a dry run never writes it.
  • The board repo must be an ASCII Hub repo id, also in a dry run (it appears in the card).
  • The private-id check allows a mention only inside a whole match of the board's own repo id (me/scans-board for source me/scans). An overlapping mention (public/me + alias public/me/private-source) is refused.
  • Size: models the registry does not know get the parameter count the Hub records for their safetensors files (hub_model_sizes, passed by the CLI). Not on the Hub, gated, or no safetensors: no size.
  • The sample line uses the drawn sample size from model_coverage; without it, --max-samples is shown as a cap.

Reviews

Codex and Fable reviewed 3068542. Codex: board-id stripping could hide an overlapping private id (P1), board names were not fully validated (P2), the sample cap was shown as the sample size (P2). All are fixed in 69fa50b with tests. Fable's copy and test-gap notes are addressed in the commits after that. Fresh reviews on the final head will follow.

Checks

pytest (1012+ tests), ruff, ty, rumdl pass. The DuckDB queries were run against the private v0 board.

🤖 Generated with Claude Code

https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh

davanstrien and others added 4 commits November 9, 2026 15:52
The card starts with what the board is and how to read it, calls the
rating a rating, links model names that are Hub repo ids, counts error
outputs in the warning note, and shows slice labels as code. The full
settings stay in board.json. A collapsed section has DuckDB queries.

plan_share takes board_repo; board.json records it and push_share
refuses another destination. The private-id check skips the board's
own whole repo id only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh
…le size

- Size: models the registry does not know get the Hub's safetensors
  parameter count (hub_model_sizes, passed by the CLI).
- A private-id mention is allowed only inside a whole board-id match;
  an overlapping mention is refused (Codex P1).
- The board repo must be an ASCII Hub repo id, also in a dry run (P2).
- The card uses the drawn sample size from model_coverage, else shows
  --max-samples as a cap (P2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh
- uv.lock: restore main's lock file. 69fa50b committed a lock that a
  local uv run had rewritten to an internal package mirror.
- Card copy: ties, interval overlap, error outputs "in the sample",
  pages in no board, the judge criteria in order, a load_dataset hint
  and a token note for private boards.
- Size: 10B and larger keep three significant digits; the Size cell
  is escaped.
- CLI: a note when the board repo name contains a private repo id.
- Tests: markup-shaped model names, the FAILED note, no sample cap,
  a slice column with markup, size formats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh
Comment thread src/ocr_bench/share.py Resolved
davanstrien and others added 5 commits November 9, 2026 16:09
…o_id

- The card links the judge prompts (judge.py at the release tag) and
  no longer paraphrases the criteria: a judge reason on the v0 board
  repeated " running headers and page number", and the page-text
  check refused the card in a real dry run.
- Size: the Hub's safetensors count for every public model on the Hub
  (read without a token), the registry's size otherwise, so sizes are
  measured the same way. The card says what Size counts.
- Board repo: huggingface_hub's validate_repo_id plus a namespace and
  no surrounding whitespace.
- The error-output note says the count is over the source dataset;
  the comparisons line says the auto-ties are included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh
…II board id

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh
@davanstrien
davanstrien merged commit cebfaa9 into main Nov 9, 2026
1 check passed
@davanstrien
davanstrien deleted the feat/board-card branch November 9, 2026 16:22
@davanstrien davanstrien mentioned this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant