fix(sdk): read the TDF manifest under the spec name manifest.json

[pflynn-virtru]

Refs #3513. Read half only — carved out of #4049, which also renames what the writer emits.

Problem

The OpenTDF spec names the manifest entry in a .tdf archive manifest.json. This SDK writes and reads 0.manifest.json. On the read side it is an exact map lookup with no fallback, so a TDF produced by any implementation written against the published spec is rejected outright.

TDFReader.Manifest() is the single choke point for both LoadTDF and IsValidTdf, so validation fails before any schema check runs.

Change

TDFReader.Manifest() accepts either name, preferring manifest.json when an archive carries both. The fallback fires only on a missing entry — an oversized manifest under the spec name stays a size failure, rather than silently returning the superseded non-aligned entry.

The dual-name handling lives in tdf3_reader.go, not reader.go: Reader is a generic ZIP accessor whose other two lookups are payload paths that must stay exact.

The writer is unchanged and still emits 0.manifest.json. That rename is a breaking file-format change that needs java-sdk, web-sdk, client-cpp and xtest to accept the spec name first; #4049 carries it. This PR is the read half, which is backwards compatible on its own and is the prerequisite for any of that work.

Testing

Each behavioral test was verified to fail without the change by reverting tdf3_reader.go and re-running:

Test	Failure without the change
TestManifest_ReadsSpecName	zip: file not found
TestManifest_PrefersSpecNameOverOffspec	read text/plain where application/octet-stream expected
TestManifest_OversizedSpecNameDoesNotFallBack	An error is expected but got nil
TestIsValidTdf_AcceptsSpecManifestName	tdfReader.Manifest failed: zip: file not found

TestManifest_PrefersSpecNameOverOffspec gives the two entries different content so it cannot pass vacuously. TestManifest_ReadsOffspecName and TestManifest_MissingUnderEitherName guard behavior that must not change, and pass either way.

The specNamedManifestTdf fixture in sdk_test.go was generated from the existing non-aligned blob by re-zipping it under the spec entry name, so the two hold the same manifest and payload and differ only in how the manifest entry is named. The existing blob is untouched, which keeps a spec-named and a non-aligned-named archive both under test. The zipstream fixtures use deliberately incomplete manifest JSON because that package treats entry contents as opaque bytes; they verify ZIP name resolution, not schema validation.

$ go test ./... -race          # sdk module — all packages ok
$ go test -run TestREADMECodeBlocks .
ok  github.com/opentdf/platform/sdk  1.220s

golangci-lint on the sdk module: 3 issues, all pre-existing in files this PR does not touch (segment_writer.go, kas_client_test.go) — zero new.

xtest is unaffected: it reads 0.manifest.json, which this SDK still writes.

🤖 Generated with Claude Code

Known limitations

Surfaced by review of this branch and deliberately left out of scope. None is a regression from main except where noted.

• The normative payload.url field is ignored (sdk: payload entry name is hardcoded to 0.payload; the spec's payload.url is never read #4065). ReadPayload/PayloadSize hardcode 0.payload, and payload.url has no read sites in the SDK — it is written at sdk/tdf.go:325 and experimental/tdf/writer.go:468 and never read back. The spec is explicit that the payload filename is whatever url says, and that 0.payload is only a convention: schema/OpenTDF/README.md says "the filename within the archive is referenced by the manifest.json (commonly 0.payload)", payload.md calls it "typically", and the normative JSON schema types url as an unconstrained string with no pattern, enum or const. A spec-valid archive may therefore name its payload anything, and this SDK fails on the first Read — after a KAS rewrap has already been spent. This is exercised in the wild: the community Rust SDK arkavo-org/opentdf-rs writes {index}.payload per entry and sets payload.url to match, so its multi-entry archives carry payloads named 1.payload, 2.payload and so on. See sdk: payload entry name is hardcoded to 0.payload; the spec's payload.url is never read #4065 for the full survey.
• An empty manifest.json shadows a valid 0.manifest.json. The fallback keys on errZipFileNotFound, and a zero-length entry is found, so Manifest() returns an empty string with a nil error and the failure surfaces later as a JSON parse error. New with this PR, but it requires a malformed archive; preferring the spec entry unconditionally is the same policy as fix(sdk)!: name the TDF manifest entry manifest.json per spec #4049 and keeps the two PRs aligned.
• Cross-SDK alignment depends on companion PRs. java-sdk#406 and web-sdk#1042 now make the same spec-name preference. Until they land, released SDKs still read only the non-aligned name.
• Pre-existing reader.go gaps that the new name makes easier to reach. ReadAllFileData does not check CompressionMethod, so a DEFLATE-compressed manifest returns raw compressed bytes with a nil error — reachable via exactly the "re-zip my TDF under the spec name" workflow this PR enables, since mainstream zip tools deflate by default. Separately, local file header names are never compared against the central directory, and duplicate entry names collapse last-wins. All three predate this PR and affect 0.manifest.json identically.

One review finding did not survive checking: IsValidTdf was reported to accept any zip containing a manifest.json. It passes Lax, not Skip (sdk.go:494), and a browser-extension zip is rejected with could not validate manifest.json.

Summary by CodeRabbit

• Bug Fixes
  • TDF archives using the specification-standard manifest.json entry are now recognized and validated.
  • Existing archives using 0.manifest.json remain supported, preserving compatibility with previously created files.
  • When both supported manifest entries are present, the standard manifest.json entry takes precedence.
  • Oversized manifests are reported as size failures without retrying the alternate entry.
  • Missing manifests produce an error that identifies both supported entry names.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 6f3532a6-144a-49b9-afbe-7f8f630e6413

📥 Commits

Reviewing files that changed from the base of the PR and between 815295b and 29498d2.

📒 Files selected for processing (2)

• sdk/internal/zipstream/tdf3_reader_test.go
• sdk/sdk_test.go

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 58df2866-1ad3-4a0a-a4d3-a141a98fc48e

📥 Commits

Reviewing files that changed from the base of the PR and between 61e0aa4 and 815295b.

📒 Files selected for processing (1)

• sdk/internal/zipstream/tdf3_reader_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The TDF3 reader now checks manifest.json first and falls back to 0.manifest.json only when the specification-named entry is missing. Tests cover entry precedence, missing entries, size errors, and repeated SDK validation.

Changes

Manifest name compatibility

Layer / File(s)	Summary
Manifest lookup and fallback sdk/internal/zipstream/zip_headers.go, sdk/internal/zipstream/tdf3_reader.go, sdk/internal/zipstream/tdf3_reader_test.go	Adds the specification manifest constant. Manifest() prefers manifest.json and falls back to 0.manifest.json only when the first entry is missing. Tests cover precedence, missing entries, and size-limit behavior.
SDK validation coverage sdk/sdk_test.go	Verifies that sdk.IsValidTdf accepts manifest.json and succeeds on repeated reads of the same reader.

Priority: ⬆️ High

Estimated code review effort: 2 (Simple) | ~15 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant TDFReader
  participant ZIPArchive
  Caller->>TDFReader: Request Manifest()
  TDFReader->>ZIPArchive: Read manifest.json
  ZIPArchive-->>TDFReader: Manifest or missing-entry error
  TDFReader->>ZIPArchive: Read 0.manifest.json when manifest.json is missing
  ZIPArchive-->>TDFReader: Manifest or error
  TDFReader-->>Caller: Return manifest or error

Loading

Suggested reviewers: elizabethhealy

Merge Risk: ⚪ Minimal · up to 81529

The reader now accepts specification-named manifests while retaining compatibility with existing files. No concrete merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 45.45% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: SDK manifest reading now supports the specification-defined entry name manifest.json.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit checks the archive with care,
Finds the manifest tucked in there.
First the spec name, then the old,
Size errors keep their shape and hold.
Tests hop through the paths and cheer.}

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	168.485939ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	95.53449ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	347.397556ms
Throughput	287.85 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	44.127046708s
Average Latency	440.281557ms
Throughput	113.31 requests/second

[github-actions]

X-Test Results

✅ js@main-v0.9.0
cukes-report
✅ java@main-main
govulncheck-failure-3
✅ java@main-v0.9.0
✅ go@pull-4063-main
bats-test-results
✅ go@pull-4063-pull-4063
✅ go@pull-4063-v0.9.0
govulncheck-failure-8
✅ go@main-pull-4063
✅ java@main-pull-4063
govulncheck-failure-1
✅ go@main-main
✅ go@main-v0.9.0
opentdfplatformQOP4PA.dockerbuild
✅ js@main-pull-4063
✅ js@main-main

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	222.607219ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	125.170854ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	437.43072ms
Throughput	228.61 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	56.076146884s
Average Latency	559.285372ms
Throughput	89.16 requests/second

[github-actions]

X-Test Results

✅ go@main-v0.9.0
✅ go@main-pull-4063
✅ go@pull-4063-main
✅ java@main-v0.9.0
cukes-report
opentdfplatformL7QAGS.dockerbuild
✅ js@main-v0.9.0
✅ java@main-main
✅ go@main-main
✅ go@pull-4063-v0.9.0
govulncheck-failure-1
govulncheck-failure-3
✅ js@main-main
✅ java@main-pull-4063
✅ go@pull-4063-pull-4063
✅ js@main-pull-4063
govulncheck-failure-8

[arkavo-com]

Concrete downstream impact, in case it helps prioritise review:

OpenTDFKit (the Swift SDK) moved its writer to the spec name manifest.json in arkavo-org/OpenTDFKit#53. Since then its cross-SDK X-Test job, which runs against released otdfctl/v0.37.0, fails the Swift-encrypt → Go-decrypt cell with:

Failed to decrypt file: tdfReader.Manifest failed: zip: file not found

Go-encrypt → Swift-decrypt passes, because the Swift reader already accepts both names. So the read-side fallback in this PR is the only thing between a spec-compliant writer and every released Go consumer. It is non-breaking and independent of the writer rename in #4049.

@opentdf/go-sdk — could someone take a look? Happy to add a test vector (a manifest.json-named TDF produced by the Swift SDK) if that would help.

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	181.203568ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	104.48456ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	354.441896ms
Throughput	282.13 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	47.915994617s
Average Latency	478.122905ms
Throughput	104.35 requests/second

[github-actions]

X-Test Failure Report

bats-test-results
✅ go@pull-4063-pull-4063
✅ go@pull-4063-main
✅ java@main-pull-4063
✅ js@main-pull-4063
✅ go@main-main
✅ java@main-main
cukes-report
✅ java@main-v0.9.0
✅ js@main-v0.9.0
✅ js@main-main
govulncheck-failure-1
opentdfplatform817E3Y.dockerbuild
✅ go@pull-4063-v0.9.0
govulncheck-failure-8
✅ go@main-pull-4063
govulncheck-failure-3

[github-actions]

X-Test Results

✅ go@main-v0.9.0
bats-test-results
✅ go@pull-4063-pull-4063
✅ go@pull-4063-main
✅ java@main-pull-4063
✅ js@main-pull-4063
✅ go@main-main
✅ java@main-main
cukes-report
✅ java@main-v0.9.0
✅ js@main-v0.9.0
✅ js@main-main
govulncheck-failure-1
opentdfplatform817E3Y.dockerbuild
✅ go@pull-4063-v0.9.0
govulncheck-failure-8
✅ go@main-pull-4063
govulncheck-failure-3

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	230.640688ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	133.779693ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	420.460553ms
Throughput	237.83 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	58.699647334s
Average Latency	585.498175ms
Throughput	85.18 requests/second

[github-actions]

X-Test Failure Report

✅ go@main-pull-4063
bats-test-results
server-logs-js@main-pull-4063
server-logs-go@pull-4063-main
✅ go@pull-4063-v0.9.0
govulncheck-failure-3
✅ go@pull-4063-pull-4063
✅ java@main-pull-4063
✅ go@main-main
✅ js@main-v0.9.0
cukes-report
✅ java@main-v0.9.0
govulncheck-failure-8
✅ js@main-main
✅ go@main-v0.9.0
✅ java@main-main
opentdfplatformVN5GOO.dockerbuild
govulncheck-failure-1

[github-actions]

X-Test Results

✅ go@main-pull-4063
✅ js@main-pull-4063
✅ go@pull-4063-main
bats-test-results
server-logs-js@main-pull-4063
server-logs-go@pull-4063-main
✅ go@pull-4063-v0.9.0
govulncheck-failure-3
✅ go@pull-4063-pull-4063
✅ java@main-pull-4063
✅ go@main-main
✅ js@main-v0.9.0
cukes-report
✅ java@main-v0.9.0
govulncheck-failure-8
✅ js@main-main
✅ go@main-v0.9.0
✅ java@main-main
opentdfplatformVN5GOO.dockerbuild
govulncheck-failure-1

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	130.034807ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	68.648658ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	245.483255ms
Throughput	407.36 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	35.037628181s
Average Latency	349.4937ms
Throughput	142.70 requests/second

[github-actions]

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

• otdfctl
• service
• tests-bdd

See the workflow run for details.

[github-actions]

X-Test Results

✅ js@main-pull-4063
✅ java@main-main
✅ go@main-pull-4063
✅ js@main-main
cukes-report
✅ go@main-main
✅ java@main-v0.9.0
opentdfplatformYIS8VU.dockerbuild
✅ go@pull-4063-pull-4063
govulncheck-failure-8
✅ java@main-pull-4063
✅ js@main-v0.9.0
✅ go@main-v0.9.0
✅ go@pull-4063-v0.9.0
✅ go@pull-4063-main
govulncheck-failure-3
govulncheck-failure-1