Skip to content

refactor(mcp_readability): group the scorer modules into a package - #637

Merged
akangsha7 merged 2 commits into
mainfrom
evalbench/mcp-readability-scorers-package
Oct 1, 2026
Merged

akangsha7 merged 2 commits into
mainfrom
evalbench/mcp-readability-scorers-package

Conversation

@akangsha7

Copy link
Copy Markdown
Collaborator

Addresses the review comment on #623: the readability scorers had grown to four modules sitting flat in scorers/, with carry-forward and CUJ coverage still to come. This lays them out the way dataset_quality is, as a pure move so #623 stays a review of its own logic.

Changes

  1. scorers/mcp_readability/ (new package) — mcp_readability_scoring.py → scoring.py, mcp_fingerprint.py → fingerprint.py, mcp_style_readability.py → style_readability.py, mcp_tool_metrics.py → tool_metrics.py. Contents are unchanged apart from their own imports.
  2. scorers/mcp_readability/registry.py (new) — SCORER_REGISTRY moves out of the orchestrator, so the package is the one thing a caller imports. It is a registry rather than a single top-level scorer class like DatasetQualityScorer: mcp_tool_metrics and mcp_style_readability are separately registered names that appear under scorers: in the run config and as the comparator on their summary rows, so collapsing them behind one scorer would change the config surface and the dashboard rows.
  3. evaluator/mcp_readability/orchestrator.py — imports the registry instead of the scorer classes; docstring module paths updated.
  4. Tests — import paths only. The patch targets are strings (scorers.mcp_style_readability.get_generator), so those were swept for separately; a stale one would silently patch nothing rather than fail.

Scorer names in run_config.yaml are unchanged, so the google3 mirror and Guitar are unaffected.

PYTHONPATH=$PWD/evalbench:$PWD uv run --with pytest pytest evalbench/test — 783 passed. bigtable_test fails with a GCP PermissionDenied, which reproduces on main and is unrelated.

The readability scorers have grown to four modules sitting flat in
scorers/, with more coming. Move them under scorers/mcp_readability/ the
way dataset_quality is laid out, and give the package the scorer registry
so the orchestrator imports one entry point instead of each module.

Pure move: no behavior change, and the scorer names in the run config are
untouched.
@akangsha7
akangsha7 merged commit 4da3fae into main Oct 1, 2026
12 of 14 checks passed
@akangsha7
akangsha7 deleted the evalbench/mcp-readability-scorers-package branch October 1, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants