Repository navigation
fix(cobol): terminate LINE_PREFIX_COMMENT scan at EOF (worker livelock) - #3000
Open
skynetdqdkfa wants to merge 1 commit into
Open
skynetdqdkfa wants to merge 1 commit into
skynetdqdkfa wants to merge 1 commit into
Conversation
The vendored COBOL external scanner's LINE_PREFIX_COMMENT loop advanced
while get_column() <= 5 without checking the lookahead. tree-sitter's
lexer advance() is a no-op at EOF -- the column never increments -- so
any input reaching EOF at column <= 5 spun the scanner forever at 100%
CPU: a .cob shorter than 6 bytes ("X" suffices), or a trailing newline
that leaves the lexer at column 0 of the virtual empty last line in the
right parser state. On a full GCC 16 index this burned nine 15-minute
quiet timeouts with extract-phase restarts, quarantining 7 .cob tests
and 1 .cbl per run (DeusData#2998).
Also stop emitting a zero-length COMMENT_ENTRY token when
start_with_word() bailed at "\n"/EOF without consuming input -- a
zero-length external token is a separate parser livelock class.
Reproduce-first regression tests parse the EOF shapes in a forked child
under a wall-clock deadline, so a regression is a RED row on the repro
board instead of a hung CI job; a control re-extracts PROGRAM-ID from
the canonical valid-program fixture. RED on unfixed main, GREEN here.
Fixes DeusData#2998
Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the index-worker livelock reported in #2998.
The vendored COBOL external scanner's
LINE_PREFIX_COMMENTloop advanced whileget_column() <= 5without ever checking the lookahead. tree-sitter'sadvance()is a no-op at EOF — the column never increments — so any input reaching EOF at column <= 5 spun the scanner forever at 100% CPU until the 15-minute quiet timeout killed the worker. A.cobfile shorter than 6 bytes (Xsuffices) triggers it directly; longer files trigger it when their parse leaves the scanner on the virtual empty line after the final\n(7 of ~90 GCC COBOL tests did). On a full GCC 16 index that meant nine quiet-timeout kills, extract-phase restarts from scratch, and 8 quarantined files per run (~3 h wall for ~6 min of real work).Two guards in
internal/cbm/vendored/grammars/cobol/scanner.c:LINE_PREFIX_COMMENTloop stops atlookahead == 0(progress is still guaranteed: the scan entry already rejects EOF, so at least one real character is consumed before the token is emitted);COMMENT_ENTRYis no longer emitted whenstart_with_word()consumed nothing — a zero-length external token, which is a separate parser-livelock class (its bail-at-\npath returned a token spanning zero bytes).Reproduce-first regression tests in
tests/repro/repro_extraction.c(#2998): the EOF shapes are parsed in a forked child under a 10 s wall-clock deadline, so a regression shows up as a RED row on the repro board instead of a hung CI job; a control re-extractsPROGRAM-IDfrom the canonical valid-program fixture (same source as the cobol row ingrammar_cases.h), guarding the guards.Verified locally:
main—repro2998_parse_with_deadline("X", …) == -1, expected 0(child killed at the deadline);CBM_REPRO_ONLY=repro_extraction build/c/test-repro-runner);.cobtests now index in ~1.7 s.Checklist
git commit -s) — required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test) — the touched suites run viamake -f Makefile.cbm build/c/test-repro-runner && CBM_REPRO_ONLY=repro_extraction build/c/test-repro-runner(the repro board is a separate non-gating runner by design; the full gating suite is left to CI — no first-party/gated source is touched)
make -f Makefile.cbm lint-ci) — both touched files are outside thelint scope (
vendored/grammars/**andtests/**are not inLINT_SRCS; the same vendored scanner already carries pre-existing clang-format deviations onmain)Fixes #2998