Skip to content

S0054-cli-bulk-1: a documented input-size limit and incremental record output for parse - #46

Merged
NSchatz merged 2 commits into
mainfrom
sdd/S0054-cli-bulk-1
Aug 20, 2026
Merged

S0054-cli-bulk-1: a documented input-size limit and incremental record output for parse#46
NSchatz merged 2 commits into
mainfrom
sdd/S0054-cli-bulk-1

Conversation

@NSchatz

@NSchatz NSchatz commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What

cosyte parse gains a documented input-size limit and emits multi-record output as it is produced.

The limit. One invocation reads at most 67108864 bytes (64 MiB). Past that it is a value-free
CLI_INPUT_TOO_LARGE naming the limit and the data-error exit code (65), never the
internal-error code (70). The check runs against the running byte count as the input arrives, so
the refusal lands before anything allocates memory proportional to the oversized input. Before this,
a legitimately large input crossed one of Node's own allocation ceilings
(buffer.constants.MAX_LENGTH / MAX_STRING_LENGTH), threw past every handler and was reported as
CLI_INTERNAL / exit 70: "an unexpected exception, i.e. a bug", about an input that is merely big.

Incremental output. Multi-record input (--ndjson and MLLP frames) is read as chunks and each
record's NDJSON line is written as that record is parsed, instead of being accumulated and written
once at the end. Per-record isolation and the exit-code contract are unchanged.

Why 64 MiB. Strictly below the smaller platform ceiling (536870888) by roughly eight times,
because the bytes read are not the peak: a parsed model rendered as JSON is routinely several times
the size of the input. A power of two, so the byte count and the mebibyte rendering are both exact.

Behaviour changes worth reviewing

  • A truncated MLLP stream now emits the frames that completed before it fails. The truncation is
    detected at end of stream, after those lines have been written. It was never a success and still is
    not: exit 65, value-free diagnostic. test/formats.test.ts was updated for this and now asserts
    the partial stream plus the non-zero exit, which is the contract the spec asks for.
  • A downstream consumer that closes the pipe (cosyte parse big.ndjson | head -3) is a value-free
    CLI_OUTPUT_WRITE_FAILED (exit 70) rather than an unhandled write error. One write per record is
    a failure surface a single write did not have.
  • RunDeps gains three optional members (openFile, openStdin, writeStdout). A caller that
    omits them gets identical output through the same code path, over a single chunk.
  • The MLLP frame reader's own smaller default frame ceiling is raised to the CLI's limit on the
    parse path only
    , so the documented number is the binding one rather than an undocumented one
    underneath it. inspect keeps the library default.

Evidence

pnpm run test 524 passed / 33 files. pnpm run lint, pnpm run typecheck, pnpm run build,
pnpm run format:check, pnpm attw, pnpm smoke, pnpm phi-scan, pnpm check:no-emdash,
pnpm check:no-internal-refs, pnpm check:agent-notes all green. Coverage: core 97.55 stmts /
93.63 branch, commands 98.37 / 93.71, both over the 90 gate.

The over-limit regressions run at the real limit against sources that would produce more bytes than a
single string can hold, and assert the source was never drained: a check that ran after assembling the
input could not pass them. The incremental tests were falsified against a deliberately
whole-input-buffering variant of the same code and failed, as intended.

Verified on the built binary: --help states the limit; a 70 MB single message and a 70 MB
--ndjson stream on stdin both exit 65 with the limit named; parse | head -2 prints two records
and the value-free write failure with no stack trace; a 2000-record batch exits 0 with 2000 lines.

Follow-up for a maintainer

CLI_INPUT_TOO_LARGE and CLI_OUTPUT_WRITE_FAILED are new stable CLI_* codes, so the standing
crew-skill and knowledgebase loop applies. Those live outside this repository and are not touched here.

Spec: work/specs/S0054-cli-bulk-1/spec.md

…d output for parse

A legitimately large input used to walk into one of Node's own allocation
ceilings, throw past every handler, and be reported as CLI_INTERNAL / exit 70:
"an unexpected exception, i.e. a bug", about an input that is merely big. And
multi-record parse output was accumulated in full and written once, so nothing
reached stdout until the last byte of input had been read.

The CLI now declares a limit of its own, 67108864 bytes (64 MiB), far below
both platform ceilings, and checks it against the running byte count as the
input arrives. An over-limit invocation is refused with a value-free
CLI_INPUT_TOO_LARGE naming the limit and the data-error exit code (65), before
anything allocates memory proportional to the oversized input. The number is
rendered from one constant into `cosyte --help` and the command reference, and
a test reds if the two ever disagree.

Multi-record input (--ndjson and MLLP frames) is now read as chunks and emitted
record by record as each record is parsed. Per-record isolation and the
exit-code contract are unchanged. A fatal condition part way through keeps the
lines already written and still resolves to that failure's own non-zero exit
code, so a partial record stream is never presented as a complete one; a
truncated MLLP stream is the visible case, where the frames that completed are
now emitted before the truncation is detected at end of stream. A downstream
consumer that closes the pipe is a value-free CLI_OUTPUT_WRITE_FAILED rather
than an unhandled write error: one write per record is a failure surface a
single write did not have.

Spec: work/specs/S0054-cli-bulk-1/spec.md
…same path

The size pre-check stat-ed the path and then read the path, which is a
time-of-check to time-of-use race: the file measured and the file read need not
be the same one. Code scanning flagged it as `js/file-system-race`, high
severity, on the first push of this branch.

The reader now opens the file once and does both operations on that handle, so
the size that gates the read is the size of the bytes that get read. The
value-free failure modes are unchanged: a missing path, a directory and an
unreadable file are all still CLI_NO_INPUT / exit 66, and an oversized file is
still refused before its bytes are read.
@NSchatz
NSchatz merged commit fcdd00f into main Aug 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant