Skip to content

fix(cobol): terminate LINE_PREFIX_COMMENT scan at EOF (worker livelock) - #3000

Open
skynetdqdkfa wants to merge 1 commit into
DeusData:mainfrom
skynetdqdkfa:fix/cobol-scanner-eof-livelock
Open

skynetdqdkfa wants to merge 1 commit into
DeusData:mainfrom
skynetdqdkfa:fix/cobol-scanner-eof-livelock

Conversation

@skynetdqdkfa

Copy link
Copy Markdown

What does this PR do?

Fixes the index-worker livelock reported in #2998.

The vendored COBOL external scanner's LINE_PREFIX_COMMENT loop advanced while get_column() <= 5 without ever checking the lookahead. tree-sitter's advance() is a no-op at EOF — the column never increments — so any input reaching EOF at column <= 5 spun the scanner forever at 100% CPU until the 15-minute quiet timeout killed the worker. A .cob file shorter than 6 bytes (X suffices) triggers it directly; longer files trigger it when their parse leaves the scanner on the virtual empty line after the final \n (7 of ~90 GCC COBOL tests did). On a full GCC 16 index that meant nine quiet-timeout kills, extract-phase restarts from scratch, and 8 quarantined files per run (~3 h wall for ~6 min of real work).

Two guards in internal/cbm/vendored/grammars/cobol/scanner.c:

  • the LINE_PREFIX_COMMENT loop stops at lookahead == 0 (progress is still guaranteed: the scan entry already rejects EOF, so at least one real character is consumed before the token is emitted);
  • COMMENT_ENTRY is no longer emitted when start_with_word() consumed nothing — a zero-length external token, which is a separate parser-livelock class (its bail-at-\n path returned a token spanning zero bytes).

Reproduce-first regression tests in tests/repro/repro_extraction.c (#2998): the EOF shapes are parsed in a forked child under a 10 s wall-clock deadline, so a regression shows up as a RED row on the repro board instead of a hung CI job; a control re-extracts PROGRAM-ID from the canonical valid-program fixture (same source as the cobol row in grammar_cases.h), guarding the guards.

Verified locally:

  • RED on unfixed main — repro2998_parse_with_deadline("X", …) == -1, expected 0 (child killed at the deadline);
  • GREEN with this change — 3 passed, 0 failed (CBM_REPRO_ONLY=repro_extraction build/c/test-repro-runner);
  • end-to-end: a 120k-file GCC checkout full-indexes in ~6 min with an empty quarantine list (previously ~3 h with 9 hangs), and the 7 formerly quarantined .cob tests now index in ~1.7 s.

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test) — the touched suites run via
    make -f Makefile.cbm build/c/test-repro-runner && CBM_REPRO_ONLY=repro_extraction build/c/test-repro-runner
    (the repro board is a separate non-gating runner by design; the full gating suite is left to CI — no first-party/gated source is touched)
  • Lint passes (make -f Makefile.cbm lint-ci) — both touched files are outside the
    lint scope (vendored/grammars/** and tests/** are not in LINT_SRCS; the same vendored scanner already carries pre-existing clang-format deviations on main)
  • New behavior is covered by a test (reproduce-first for bug fixes)

Fixes #2998

The vendored COBOL external scanner's LINE_PREFIX_COMMENT loop advanced
while get_column() <= 5 without checking the lookahead. tree-sitter's
lexer advance() is a no-op at EOF -- the column never increments -- so
any input reaching EOF at column <= 5 spun the scanner forever at 100%
CPU: a .cob shorter than 6 bytes ("X" suffices), or a trailing newline
that leaves the lexer at column 0 of the virtual empty last line in the
right parser state. On a full GCC 16 index this burned nine 15-minute
quiet timeouts with extract-phase restarts, quarantining 7 .cob tests
and 1 .cbl per run (DeusData#2998).

Also stop emitting a zero-length COMMENT_ENTRY token when
start_with_word() bailed at "\n"/EOF without consuming input -- a
zero-length external token is a separate parser livelock class.

Reproduce-first regression tests parse the EOF shapes in a forked child
under a wall-clock deadline, so a regression is a RED row on the repro
board instead of a hung CI job; a control re-extracts PROGRAM-ID from
the canonical valid-program fixture. RED on unfixed main, GREEN here.

Fixes DeusData#2998

Signed-off-by: Леонид Юрьев (Leonid Yuriev) <leo@yuriev.ru>
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Index worker livelocks on tiny COBOL files — external scanner LINE_PREFIX_COMMENT loop never terminates at EOF

2 participants