fix(sdk): reject malformed ZIP64 lengths and null assertion statements; refuse isEncrypted=false (DSPX-5102)

[dmihalcik-virtru]

Jira: https://virtru.atlassian.net/browse/DSPX-5102 (also resolves DSPX-4687)

Fixes #4202

Proposed Changes

sdk.IsValidTdf and sdk.LoadTDF classify untrusted files. Malformed input must produce an error, never panic the caller.

• ZIP64 length overflow (sdk: TDF reader panics and permissive validation on malformed archives (v0.34.0) #4202 part 1 / DSPX-4687). On current main this is already rejected by the DSPX-4590 bounds work (fix(sdk): bound zip64 reader offsets against the archive (DSPX-4590) #4043). Every ZIP64 size and offset is checked against the central-directory start and the archive length before it is narrowed to int64. The sdk: TDF reader panics and permissive validation on malformed archives (v0.34.0) #4202 archive now fails in NewReader with file data at 45+18446744073709551615 runs past the start of the central directory. This PR adds:
  • the exact sdk: TDF reader panics and permissive validation on malformed archives (v0.34.0) #4202 shape (a lone 0.manifest.json entry, 32-bit compressed size 0xFFFFFFFF, ZIP64 extra field 0xFFFFFFFFFFFFFFFF) as a regression test at three levels: zipstream unit, IsValidTdf, and LoadTDF with Skip/Lax/Strict. It is also a seed for both fuzz targets.
  • defence-in-depth checks: ReadAllFileData and readBytes now return errZipFileSizeError for a negative length or index instead of reaching make([]byte, n). (ReadFileData already rejected these.)
• Statement.UnmarshalJSON nil dereference (sdk: TDF reader panics and permissive validation on malformed archives (v0.34.0) #4202 part 2). encoding/json matches keys case-insensitively, so "Statement": null sends null to the custom decoder. A JSON null is now a no-op, following the json.Unmarshaler convention. The alias struct is also decoded through aux instead of &aux, so a null can never set it to nil. There are no other custom UnmarshalJSON methods in sdk/, lib/ or otdfctl/.
• payload.isEncrypted: false (sdk: TDF reader panics and permissive validation on malformed archives (v0.34.0) #4202 part 3). We don't support TDFs flagged as cleartext.
  • manifest.schema.json and manifest-lax.schema.json now require "const": true on isEncrypted. Only that property changed, to avoid conflicts with chore(sdk): give the bundled manifest schemas real $id values #4070.
  • So IsValidTdf (which uses Lax), and LoadTDF with WithSchemaValidation(Lax|Strict), now reject false with ErrInvalidPerSchema.
  • Writers already always emit true: CreateTDF and the experimental writer both go through chunkedWriter.buildManifest. A new test pins this and checks that the written manifest passes both schemas.
  • Default-config LoadTDF does not reject isEncrypted:false. The default schema validation level is Skip (the zero value of SchemaValidationIntensity), and I didn't add a hard reject on that path. The Go reader never reads the flag. It always verifies integrity and decrypts, so a false flag can't turn a read into a cleartext read. A test pins this behaviour.
• FuzzLoadTDF was not reaching the parser. Every input failed early at the KAS allowlist lookup ("no KAS allowlist provided and no KeyAccessServerRegistry available"). The fuzz body now passes WithIgnoreAllowlist(true), and the SDK has a stub conn. It also gets seeds for all three DSPX-5102 shapes. With the old Statement.UnmarshalJSON, the null-statement seed panics. With the fix, all seeds pass.
• The manifest size limit is now opt-in (separate commit, fix(sdk): make the manifest size limit opt-in). I removed the 10 MB default cap: defaultMaxManifestSize in sdk/tdf_config.go, and the matching manifestMaxSize default in zipstream/tdf3_reader.go, which IsValidTdf used. There is deliberately no default manifest size cap, because of the 50 TiB payload goal (DSPX-4502): manifests for those payloads exceed 1 GiB. Memory scaling for large manifests is tracked in DSPX-4905.
  • The defence against hostile declared ZIP sizes is the archive bounds check (fix(sdk): bound zip64 reader offsets against the archive (DSPX-4590) #4043) plus the negative-length guards above. A manifest entry that declares more bytes than the archive holds is still rejected before any allocation, whether or not a cap is set.
  • WithMaxManifestSize stays as an opt-in limit and still requires a size greater than 0. Its doc comment now says the default is unlimited.
  • Inside zipstream, a zero or negative manifest limit (the new default) means no limit, and TDFReader.Manifest passes math.MaxInt64 to ReadAllFileData. ReadAllFileData itself is unchanged.
  • Behaviour change: callers that relied on the implicit 10 MB cap to bound memory now need to pass WithMaxManifestSize. I didn't mark it !. The change only relaxes a limit, no API is removed or changed, and nothing that loaded before now fails.
  • New tests:
    • default LoadTDF, IsValidTdf and zipstream Manifest() accept an 11 MB manifest
    • an explicit limit is still enforced, with an inclusive boundary
    • WithMaxManifestSize(0) is still an error
    • with no cap, a declared manifest size past the end of the archive (1<<40, 1<<63, MaxUint64) is still rejected by the bounds check
  • The only other mention of the 10 MB limit in the repo is a historical sdk/CHANGELOG.md entry, which I left as is.
• spec/DSPX-5102.md is filled in.

Behaviour note / possible follow-up: the null-statement manifest is schema-valid. IsValidTdf still returns true for it, but it no longer panics anywhere. LoadTDF now decodes it, and the first read fails the integrity check. Rejecting duplicate or case-variant manifest keys in general would close the parser-differential angle. I left it out because assertion statement values can be arbitrary JSON objects, so a blanket rule needs care. Also out of scope: the xtest container-mutation case (DSPX-4923).

Checklist

• I have added or updated unit tests
• I have added or updated integration tests (if appropriate). Not added: the xtest mutation case is tracked as DSPX-4923.
• I have added or updated documentation (schema descriptions, spec)

Testing Instructions

What I ran locally (Go 1.27.2, golangci-lint 2.14.0; CI pins 2.13.2):

• make fmt: clean. I re-ran it after the manifest-cap commit, along with sdk go test -race ./... (all ok), TestREADMECodeBlocks (ok), sdk golangci-lint (0 issues in changed files; same 10 pre-existing ones), go build ./otdfctl/... ./service/... (ok), and fuzzing (FuzzLoadTDF 30s, FuzzReader 20s, no crashers).
• cd sdk && go test -race -count=1 ./...: all packages ok.
• cd sdk && go test -run TestREADMECodeBlocks: ok.
• I checked that the new tests fail without the fix. With the old assertion.go, TestNullStatementManifestDoesNotPanic, TestStatementUnmarshalJSONNull and the FuzzLoadTDF seed panic. With the old lax schema, TestIsEncryptedFalseRejected fails.
• Short fuzz runs, no crashers: go test -fuzz FuzzLoadTDF -fuzztime 45s ./sdk (about 930k execs) and go test -fuzz FuzzReader -fuzztime 20s ./sdk/internal/zipstream.
• make lint: exits non-zero, all on pre-existing issues in untouched files. It reports 3 in lib/ocrypto (gosec), 10 in sdk (canonicalheader, staticcheck), 60 in service and 1 in otdfctl. Most likely the cause is the local golangci-lint 2.14.0 vs CI's 2.13.2. It reports 0 issues in files changed by this PR; the one it found in the new test was fixed. buf lint passed.
• make govulncheck: could not run. The local govulncheck binary was built with go1.26 and can't load go1.27 dependencies.
• make test: stops at a failure in lib/fixtures (TestTokenManager_*, expects a 120s token buffer and gets 30s). That module is unrelated and doesn't import sdk. I then ran the remaining modules by hand:
  • lib/flattening, lib/identifier and otdfctl: pass.
  • service: everything passes except environment-dependent suites: service/integration and service/entityresolution/integration (need Docker/testcontainers), service/rttests (hits a live local platform; already_exists), and service/pkg/server TestStartTestSuite ("without sdk config" cases).
  • None of the failures touch the changed code.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Benchmark results, click to expand

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	411.780973ms
Throughput	242.85 requests/second

TDF3 Benchmark Results

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	45.110864636s
Average Latency	450.218929ms
Throughput	110.84 requests/second

[github-actions]

X-Test Results

✅ js@main-v0.9.0
bats-test-results
✅ go@pull-4207-pull-4207
✅ java@main-main
✅ go@main-v0.9.0
✅ go@pull-4207-main
✅ go@pull-4207-v0.9.0
✅ js@main-main
✅ java@main-v0.9.0
✅ go@main-main
✅ js@main-pull-4207
✅ java@main-pull-4207
cukes-report
govulncheck-failure-2
govulncheck-failure-1
✅ go@main-pull-4207
govulncheck-failure-7
govulncheck-failure-0
opentdfplatformNG2WLJ.dockerbuild
govulncheck-failure-4

[github-actions]

Benchmark results, click to expand

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	423.214385ms
Throughput	236.29 requests/second

TDF3 Benchmark Results

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	45.569539157s
Average Latency	454.799529ms
Throughput	109.72 requests/second

[github-actions]

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

• otdfctl
• sdk
• service
• lib/fixtures
• tests-bdd

See the workflow run for details.

[github-actions]

X-Test Results

bats-test-results
✅ go@pull-4207-v0.9.0
govulncheck-failure-7
✅ js@main-v0.9.0
govulncheck-failure-0
✅ go@main-main
✅ go@main-v0.9.0
✅ go@pull-4207-pull-4207
✅ go@main-pull-4207
✅ java@main-v0.9.0
govulncheck-failure-1
govulncheck-failure-2
✅ go@pull-4207-main
cukes-report
opentdfplatformSBF5MW.dockerbuild
govulncheck-failure-4
✅ java@main-main
✅ js@main-main
✅ js@main-pull-4207
✅ java@main-pull-4207