Skip to content

verifier: strip comments for code redundancy in linear time - #90

Open
reacher-z wants to merge 2 commits into
mainfrom
verifier/linear-comment-strip
Open

reacher-z wants to merge 2 commits into
mainfrom
verifier/linear-comment-strip

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Closes #61

Two changes to the code-redundancy measurement in redundancy.py, which is reported beside the score in eval.json.

Linear comment stripping

The block-comment alternatives (/\*.*?\*/ for C-style and SQL, <!--.*?--> for HTML) searched to the end of the text from every opener that never closes, so many unclosed openers took quadratic time.

An opener can close only if the last closer in the text starts after it. strip_comments now uses the full scanner before that point and the same scanner without its block-comment alternative after it, and attempts a match only at characters where one can start.

Input Before After
64 KiB of unclosed /* 3.96 s 0.006 s
1 MiB of unclosed /* (the per-file cap) ~20 min (extrapolated) 0.10 s
1.1 MB of the verifier's own sources 0.038 s 0.023 s

The output is identical to the old implementation on 360,000 generated inputs across all comment styles and on the verifier's sources.

Plain .css

LANGUAGES mapped .less, .sass and .scss to css but not .css, so plain stylesheets were not measured. The second commit adds it.

Release

All 70 copies of redundancy.py stay byte-identical. The change alters the tasks' content hashes, so it ships with the next task release.

Tests

tests/test_redundancy.py loads the module from tasks/medium: comment and string handling, unclosed openers left as code, a timing bound on 128 KiB of unclosed openers (the old code takes 15 s, over the 2 s bound), and a duplicated .css file pair measured at redundancy 0.5 (no css entry before the change).

  • uv run python -m unittest discover -s tests
  • uv run ruff check scripts src tests
  • uv run ruff format --check scripts src tests
  • uv run python scripts/check_tasks.py --code-only

strip_comments ran one regular expression whose block-comment
alternative (/\*.*?\*/ for C-style and SQL sources, <!--.*?--> for HTML)
searches to the end of the text from every opener that is never closed.
A file with many unclosed openers took quadratic time: 4 s for 64 KiB,
about 20 minutes extrapolated to the 1 MiB per-file cap, all before the
time budget is next checked.

An opener can only close when the last closer in the text starts after
it, so the scan now uses the full scanner before that point and the same
scanner without its block-comment alternative after it, and tries a match
only at characters where one can start. The output is unchanged (checked
against the old implementation on 360,000 generated inputs and on the
verifier's own sources); 1 MiB of unclosed openers now takes 0.1 s.

All 70 copies of redundancy.py stay byte-identical. The change alters the
tasks' content hashes, so it ships with the next task release.
LANGUAGES mapped .less, .sass and .scss to css but not .css, so plain
stylesheets, a large share of a reproduced website, were skipped by the
code-redundancy measurement. The css comment style was already defined.

The measurement is reported beside the score and does not enter it.
@reacher-z reacher-z added verifier Scoring, capture and check execution in tests/websitebench performance Speed and resource use labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Speed and resource use verifier Scoring, capture and check execution in tests/websitebench

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Code-redundancy comment stripping is quadratic, and plain .css is never measured

1 participant