S0054-cli-bulk-1: a documented input-size limit and incremental record output for parse - #46
Merged
Conversation
…d output for parse A legitimately large input used to walk into one of Node's own allocation ceilings, throw past every handler, and be reported as CLI_INTERNAL / exit 70: "an unexpected exception, i.e. a bug", about an input that is merely big. And multi-record parse output was accumulated in full and written once, so nothing reached stdout until the last byte of input had been read. The CLI now declares a limit of its own, 67108864 bytes (64 MiB), far below both platform ceilings, and checks it against the running byte count as the input arrives. An over-limit invocation is refused with a value-free CLI_INPUT_TOO_LARGE naming the limit and the data-error exit code (65), before anything allocates memory proportional to the oversized input. The number is rendered from one constant into `cosyte --help` and the command reference, and a test reds if the two ever disagree. Multi-record input (--ndjson and MLLP frames) is now read as chunks and emitted record by record as each record is parsed. Per-record isolation and the exit-code contract are unchanged. A fatal condition part way through keeps the lines already written and still resolves to that failure's own non-zero exit code, so a partial record stream is never presented as a complete one; a truncated MLLP stream is the visible case, where the frames that completed are now emitted before the truncation is detected at end of stream. A downstream consumer that closes the pipe is a value-free CLI_OUTPUT_WRITE_FAILED rather than an unhandled write error: one write per record is a failure surface a single write did not have. Spec: work/specs/S0054-cli-bulk-1/spec.md
…same path The size pre-check stat-ed the path and then read the path, which is a time-of-check to time-of-use race: the file measured and the file read need not be the same one. Code scanning flagged it as `js/file-system-race`, high severity, on the first push of this branch. The reader now opens the file once and does both operations on that handle, so the size that gates the read is the size of the bytes that get read. The value-free failure modes are unchanged: a missing path, a directory and an unreadable file are all still CLI_NO_INPUT / exit 66, and an oversized file is still refused before its bytes are read.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
cosyte parsegains a documented input-size limit and emits multi-record output as it is produced.The limit. One invocation reads at most 67108864 bytes (64 MiB). Past that it is a value-free
CLI_INPUT_TOO_LARGEnaming the limit and the data-error exit code (65), never theinternal-error code (
70). The check runs against the running byte count as the input arrives, sothe refusal lands before anything allocates memory proportional to the oversized input. Before this,
a legitimately large input crossed one of Node's own allocation ceilings
(
buffer.constants.MAX_LENGTH/MAX_STRING_LENGTH), threw past every handler and was reported asCLI_INTERNAL/ exit70: "an unexpected exception, i.e. a bug", about an input that is merely big.Incremental output. Multi-record input (
--ndjsonand MLLP frames) is read as chunks and eachrecord's NDJSON line is written as that record is parsed, instead of being accumulated and written
once at the end. Per-record isolation and the exit-code contract are unchanged.
Why 64 MiB. Strictly below the smaller platform ceiling (536870888) by roughly eight times,
because the bytes read are not the peak: a parsed model rendered as JSON is routinely several times
the size of the input. A power of two, so the byte count and the mebibyte rendering are both exact.
Behaviour changes worth reviewing
detected at end of stream, after those lines have been written. It was never a success and still is
not: exit
65, value-free diagnostic.test/formats.test.tswas updated for this and now assertsthe partial stream plus the non-zero exit, which is the contract the spec asks for.
cosyte parse big.ndjson | head -3) is a value-freeCLI_OUTPUT_WRITE_FAILED(exit70) rather than an unhandled write error. One write per record isa failure surface a single write did not have.
RunDepsgains three optional members (openFile,openStdin,writeStdout). A caller thatomits them gets identical output through the same code path, over a single chunk.
parse path only, so the documented number is the binding one rather than an undocumented one
underneath it.
inspectkeeps the library default.Evidence
pnpm run test524 passed / 33 files.pnpm run lint,pnpm run typecheck,pnpm run build,pnpm run format:check,pnpm attw,pnpm smoke,pnpm phi-scan,pnpm check:no-emdash,pnpm check:no-internal-refs,pnpm check:agent-notesall green. Coverage:core97.55 stmts /93.63 branch,
commands98.37 / 93.71, both over the 90 gate.The over-limit regressions run at the real limit against sources that would produce more bytes than a
single string can hold, and assert the source was never drained: a check that ran after assembling the
input could not pass them. The incremental tests were falsified against a deliberately
whole-input-buffering variant of the same code and failed, as intended.
Verified on the built binary:
--helpstates the limit; a 70 MB single message and a 70 MB--ndjsonstream on stdin both exit65with the limit named;parse | head -2prints two recordsand the value-free write failure with no stack trace; a 2000-record batch exits
0with 2000 lines.Follow-up for a maintainer
CLI_INPUT_TOO_LARGEandCLI_OUTPUT_WRITE_FAILEDare new stableCLI_*codes, so the standingcrew-skill and knowledgebase loop applies. Those live outside this repository and are not touched here.
Spec:
work/specs/S0054-cli-bulk-1/spec.md