S0054-cli-bulk-1: a documented input-size limit and incremental record output for parse

.changeset/brave-pumas-stream.md

@@ -0,0 +1,20 @@
+---
+"@cosyte/cli": patch
+---
+
+Give `cosyte parse` a documented input-size limit and incremental multi-record output.
+
+An input past 67108864 bytes (64 MiB) is now refused with a value-free `CLI_INPUT_TOO_LARGE`
+diagnostic naming the limit and the data-error exit code (`65`), never the internal-error code a
+platform allocation failure used to produce. The check runs against the running byte count as the
+input arrives, so the refusal lands before anything allocates memory proportional to the oversized
+input. The limit is rendered from one constant into `cosyte --help` and the command reference, and a
+test reds if those two ever disagree.
+
+Multi-record output (`--ndjson` and MLLP frames) is emitted record by record as each record is
+parsed, rather than accumulated and written once at the end. Per-record isolation and the exit-code
+contract are unchanged. A fatal condition part way through keeps the lines already written and still
+resolves to that failure's own non-zero exit code, so a partial record stream is never presented as a
+complete one; a truncated MLLP stream is the visible case, where the frames that completed are now
+emitted before the truncation is detected. A downstream consumer that closes the pipe is a value-free
+`CLI_OUTPUT_WRITE_FAILED` rather than an unhandled write error.

CHANGELOG.md

@@ -80,8 +80,37 @@ still do. Each entry was assigned to the release whose tag first contains it, re
     gate's own OK line.** Both key on an actual NUL byte; the wider set is git's own binary
     classification, which is why neither gate may be reduced to `grep -I`.
 
+- **`parse` has a documented input-size limit: 67108864 bytes (64 MiB) per invocation, refused as a
+  data error.** Node has two hard allocation ceilings (`buffer.constants.MAX_LENGTH` and
+  `buffer.constants.MAX_STRING_LENGTH`), and a legitimately large input that crossed one of them threw
+  past every handler and was reported as `CLI_INTERNAL` / exit `70`, which says "this is a bug in the
+  tool" about an input that is merely big. The CLI now declares its own limit far below both, checks it
+  against the **running byte count as the input arrives**, and refuses with a value-free
+  `CLI_INPUT_TOO_LARGE` naming the limit and exit `65`. The number is rendered from one constant into
+  `cosyte --help` and the command reference, and a test reds if the two ever disagree.
+  - The refusal fires before anything allocates memory proportional to the oversized input, which is
+    what makes it a refusal rather than a slower way to reach the same crash. The regression suite
+    proves it with sources that would run past `MAX_STRING_LENGTH` if anything drained them.
+  - The MLLP frame reader's own smaller default ceiling is raised to the CLI's limit on this path, so
+    the documented number is the binding one rather than an undocumented number underneath it.
+
 ### Changed
 
+- **Multi-record `parse` output (`--ndjson` and MLLP) is now emitted record by record, as each record
+  is parsed**, instead of being accumulated and written once at the end. The first line reaches stdout
+  before the rest of the input has been read, so a bulk batch pipes into the next process instead of
+  waiting on the whole file. Per-record isolation and the exit-code contract are unchanged: a record
+  that fails to parse is still a value-free `{ record, error }` line, the stream still continues, and
+  any failed record still resolves the invocation to exit `65`.
+  - **A fatal condition part way through keeps the lines already written and still exits non-zero.**
+    A truncated MLLP stream is the visible case: the frames that completed have already been emitted
+    when the unterminated one is detected at end of stream, so `stdout` is no longer empty for that
+    input. It was never a success and still is not: the exit code carries the failure, and a partial
+    record stream is never reported as a complete one.
+  - **A downstream consumer that closes the pipe part way through is now a value-free
+    `CLI_OUTPUT_WRITE_FAILED`** rather than an unhandled write error. One write per record is a
+    failure surface a single write did not have.
+
 - **`CLAUDE.md` narrative was relocated into `documentation/agent-notes.md` to make room for the gate's
   rules.** The branch-protection, PHI-scanner-residual and em-dash blocks were compressed to their
   imperatives; every trap keeps a one-line rule and a pointer, and the reasoning each one compresses

README.md

@@ -146,7 +146,15 @@ astm | ncpdp | ccda | dicom | mllp`.
 **MLLP** stream, one record per frame, or any input under **`--ndjson`**, one record per non-empty
 line) streams as **NDJSON**, one `{ record, format, model, warnings }` line each, with per-record
 isolation: a record that fails to parse becomes a value-free `{ record, error }` line and the stream
-continues; the overall exit is a data error (`65`) if any record failed.
+continues; the overall exit is a data error (`65`) if any record failed. Each line is written **as
+its record is parsed**, before the rest of the input has been read, so a batch pipes straight into
+the next process.
+
+**Input size.** `parse` reads at most **67108864 bytes (64 MiB)** per invocation, the number
+`cosyte --help` prints. A larger input is a value-free `CLI_INPUT_TOO_LARGE` refusal naming that
+limit, with the data-error exit code (`65`) and never the internal-error code: split the input and
+re-run. Since records are emitted as they are parsed, a refusal can land after some records have
+already reached stdout; the exit code, not the output, says the run did not complete.
 
 Support is honest **per (format, operation)**: `x12`/`astm`/`ncpdp` support all of parse/inspect/fmt/
 validate; `ccda` supports inspect/fmt/validate (parse deferred); `dicom` supports inspect/validate
@@ -232,15 +240,15 @@ valid JSON or not a loadable ConceptMap is a `CLI_MAP_INVALID` data error (`65`)
 
 Every command is safe to branch on in CI. The exit code carries the outcome (`sysexits.h`):
 
-| Code | Meaning                                                    |
-| ---- | ---------------------------------------------------------- |
-| `0`  | success / **valid** (`validate`)                           |
-| `1`  | **invalid**: `validate` found a parseable-but-bad message  |
-| `2`  | usage error (unknown flag, missing argument)               |
-| `65` | data error (unparseable input, or format undetected)       |
-| `66` | no input (missing/unreadable file)                         |
-| `69` | unavailable (a capability is not yet built, e.g. `redact`) |
-| `70` | internal error (a bug)                                     |
+| Code | Meaning                                                                         |
+| ---- | ------------------------------------------------------------------------------- |
+| `0`  | success / **valid** (`validate`)                                                |
+| `1`  | **invalid**: `validate` found a parseable-but-bad message                       |
+| `2`  | usage error (unknown flag, missing argument)                                    |
+| `65` | data error (unparseable input, format undetected, or input past the size limit) |
+| `66` | no input (missing/unreadable file)                                              |
+| `69` | unavailable (a capability is not yet built, e.g. `redact`)                      |
+| `70` | internal error (a bug)                                                          |
 
 The load-bearing rule: the CLI **never prints a reassuring line and exits `0`** on input it could not
 handle, or on an invalid message.

docs-content/reference-commands.md

@@ -44,12 +44,27 @@ Parse a healthcare message to typed JSON on stdout (the data channel). Autodetec
 record per frame with per-record isolation (a failed record becomes a value-free `{ record, error }`
 line and the stream continues; any failure → exit `65`).
 
+Multi-record output is **written record by record, as each record is parsed**, so the first line
+reaches stdout long before the last byte of input has been read and a large batch can be piped
+straight into another process.
+
 ```bash
 cosyte parse adt.hl7
 cat adt.hl7 | cosyte parse -
 cosyte parse --ndjson bulk.ndjson | jq .
 ```
 
+#### Input size limit
+
+`parse` reads at most **67108864 bytes (64 MiB)** of input per invocation. A larger input is refused
+with a value-free `CLI_INPUT_TOO_LARGE` diagnostic naming that limit and the **data-error** exit code
+(`65`), never the internal-error code (`70`): a large input is not a bug in the tool. Split the input
+and re-run. The same number is printed by `cosyte --help`.
+
+Because records are emitted as they are parsed, a refusal can arrive **after** some records have
+already been written to stdout. The exit code, not the output, is the signal that the run did not
+complete: a partial record stream always carries a non-zero exit, never `0`.
+
 ### `cosyte validate <file\|-> [--profile <name>]`
 
 Validate a message; **the exit code carries the verdict**: `0` valid, `1` invalid (parseable but
@@ -104,7 +119,7 @@ tools over the same core. See [MCP server](./mcp).
 | `0`  | success / `validate` found the input **valid**                                          |
 | `1`  | operation-level failure: `validate` found the input **invalid** (a real CI signal)     |
 | `2`  | usage error: unknown command, bad flag, missing argument                               |
-| `65` | data error: input could not be parsed / format not detected / (format, op) unsupported |
+| `65` | data error: input could not be parsed / format not detected / (format, op) unsupported / input larger than the size limit |
 | `66` | no input: the file does not exist or is unreadable                                     |
 | `69` | unavailable: a capability is not yet built (e.g. `redact`, `--profile`)                |
 | `70` | internal error: an unexpected exception (a bug)                                        |

src/bin/cosyte.ts

@@ -10,12 +10,32 @@
  */
 
 /* v8 ignore start -- process wiring: argv/stdin/stdout/exit glue, exercised by the packaged bin smoke, not unit-covered */
-import { readFileBytes, readStreamBytes, type RunDeps } from "../core/io.js";
+import {
+  fileChunks,
+  readFileBytes,
+  readStreamBytes,
+  streamChunks,
+  type RunDeps,
+} from "../core/io.js";
 import { run } from "../core/run.js";
 
+// A consumer that closes the pipe (`cosyte parse big.ndjson | head -3`) makes the next write fail.
+// Node reports that as an 'error' event on the stream, which is fatal if nothing is listening, so
+// the sink below is marked closed instead and the command resolves it as a write failure.
+let stdoutOpen = true;
+process.stdout.on("error", () => {
+  stdoutOpen = false;
+});
+
 const deps: RunDeps = {
   readFile: (path) => readFileBytes(path),
   readStdin: () => readStreamBytes(process.stdin),
+  openFile: (path) => fileChunks(path),
+  openStdin: () => streamChunks(process.stdin),
+  writeStdout: (chunk) => {
+    if (!stdoutOpen) throw new Error("stdout closed");
+    process.stdout.write(chunk);
+  },
 };
 
 // The `cosyte mcp` subcommand starts the stdio MCP server (also reachable as the `cosyte-mcp` bin).