Skip to content

packaging: add portable content bundles - #574

Merged
freezy merged 3 commits into
masterfrom
feat/packaged-content
Sep 2, 2026
Merged

packaging: add portable content bundles#574
freezy merged 3 commits into
masterfrom
feat/packaged-content

Conversation

@freezy

@freezy freezy commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

Adds hash-addressed, inert content bundles to the .vpe package format, stored under table/content/<content-id>/. This is the packaging foundation that consumer components (e.g. a web-show host or an MPF machine directory) build on to ship a directory tree inside a table and resolve it back out at runtime, without VPE ever executing the payload.

This is the packaged-content subsystem in isolation. It was originally developed together with the Light Show Studio work; this branch carries only the portable-content half so it can be reviewed and merged on its own.

What it adds

Runtime (VisualPinball.Unity/Packaging/Content/)

  • PackagedContent.csPackagedContentRef and the split-role IPackagedContentResolver, plus IPackagedContentSource (contributes a directory at write time) and IPackagedContentConsumer (receives a read-only resolver after restore).
  • PackagedContentWriter.cs — deterministic directory ingestion: canonical ordering, per-file SHA-256, case-collision and unsafe-path rejection, overflow-checked totals, verify-on-write.
  • PackagedContentResolver.cs — streaming extraction into persistentDataPath/ContentCache/<hash>/, per-file size + hash verification, atomic publish via temp-dir rename with a .complete marker, and an LRU cache cap (4 GiB default).
  • PackagedContentValidator.cs — normative ValidatePackage() integrity gate plus authoring LintManifest() (entry-point policy, forbidden-executable policy, duplicate-payload warnings).
  • PackagedContentPath.cs — path-traversal-safe relative-path validation and containment checks.
  • PackagedContentConsent.cs — per-content-hash user consent, keyed so changed content re-prompts.

Editor / packaging wiring

  • PackageWriter implements IPackagedContentResolver, drives IPackagedContentSource components, and writes bundles.
  • PackageReader / RuntimePackageReader expose a read-only ContentResolver and inject it into IPackagedContentConsumer components after a table is restored.
  • PackageApi gains the content folder constant; PackagedContentRefDrawer renders the ref in the inspector.

Docs & tests

  • FORMAT.md / README.md specify the normative vpe-content v1 layout, canonical hashing, extraction rules, and an author checklist.
  • PackagedContentTests cover round-tripping through both the editor and runtime resolvers, malformed manifests, path traversal, and package limits.

Format

Bundles are byte-deterministic: <content-id> is the first 16 hex chars of the canonical SHA-256 over each file's ordinal path + NUL + raw file hash + LF, so identical trees deduplicate independent of source path, timestamps, or platform. Readers accept exactly vpe-content version 1 and reject unknown versions before extraction.

Notes for review

  • No behavior change to existing tables: the content/ folder is only written when a component implements IPackagedContentSource, and only read when one implements IPackagedContentConsumer.
  • The PackageWriter.cs hunks were 3-way merged onto current master (the file has diverged since the original commit); all other files applied cleanly.

Add hash-addressed, inert content bundles stored under table/content/,
with secure extraction, validation, consent, and LRU cache management.

Bundles are deterministic: files are canonically ordered and hashed so
identical trees deduplicate regardless of source location, timestamps,
or platform. Writers reject absolute/unsafe paths and case collisions;
readers verify every file's declared size and SHA-256 while streaming
into an atomically published cache directory under persistentDataPath.

Split-role IPackagedContentResolver: PackageWriter ingests directories
via IPackagedContentSource at write time, while PackageReader and
RuntimePackageReader inject a read-only resolver into
IPackagedContentConsumer components after a table is restored.

Cover round-tripping, malformed manifests, path traversal, and package
limits with tests.
@freezy freezy self-assigned this Sep 2, 2026
@greptile-apps

greptile-apps Bot commented Sep 2, 2026

Copy link
Copy Markdown

Greptile Summary

Adds deterministic, hash-addressed content bundles to .vpe packages and read-only extraction through a shared verified cache.

  • Introduces bundle manifests, canonical hashing, path and integrity validation, consent handling, and cache publication/eviction.
  • Wires content producers into package writing and injects resolvers into restored consumers.
  • Adds editor presentation, format documentation, and packaging tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
VisualPinball.Unity/VisualPinball.Unity/Packaging/Content/PackagedContentResolver.cs Implements verified streaming extraction, heartbeat-protected temporary publication, and bounded cache cleanup; the previously reported live-extraction cleanup path is addressed.
VisualPinball.Unity/VisualPinball.Unity/Packaging/Content/PackagedContentWriter.cs Implements deterministic directory preparation and verified package writing, with the remaining portable-filesystem TOCTOU limitation explicitly accepted by the author.
VisualPinball.Unity/VisualPinball.Unity/Packaging/Content/PackagedContentValidator.cs Defines reference, manifest, package-integrity, and authoring-policy validation for version-one content bundles.
VisualPinball.Unity/VisualPinball.Unity.Editor/Packaging/PackageWriter.cs Collects content sources before component serialization and writes prepared bundles in stable content-ID order.
VisualPinball.Unity/VisualPinball.Unity/Packaging/RuntimePackageReader.cs Exposes and injects the read-only packaged-content resolver during runtime restoration.
VisualPinball.Unity/VisualPinball.Unity.Test/Packaging/PackagedContentTests.cs Covers deterministic round trips, validation failures, path safety, cancellation, cleanup, and cache eviction.

Sequence Diagram

sequenceDiagram
    participant Source as Content Source
    participant Writer as Package Writer
    participant Package as .vpe Package
    participant Reader as Package Reader
    participant Resolver as Content Resolver
    participant Cache as Verified Cache
    Source->>Writer: PreparePackagedContent
    Writer->>Writer: Hash and validate directory
    Writer->>Package: Write manifest and payload
    Reader->>Resolver: Inject read-only resolver
    Resolver->>Package: Read manifest and files
    Resolver->>Resolver: Verify paths, sizes, and hashes
    Resolver->>Cache: Atomically publish completed bundle
    Resolver-->>Reader: Return extracted directory
Loading

Reviews (3): Last reviewed commit: "packaging: heartbeat live extractions, b..." | Re-trigger Greptile

Address review feedback on the content bundle subsystem.

Startup temp-directory cleanup only reaps orphans older than one hour.
Resolvers share the default cache root, so a blanket wipe of *.tmp-*
could delete the extraction another concurrently constructed resolver
is still writing. An in-progress extraction keeps its directory young,
so an age guard reaps crashed-run orphans without touching live work.

Re-check for a reparse point before measuring and hashing each source
file. Enumeration skips links, but a source tree mutated mid-export
could swap a regular file for a symlink pointing outside sourceRoot
before it is hashed; the captured hash then also guards the later write.
// before it is measured and hashed, which would otherwise package external bytes. The
// hash captured here also guards the later write: bytes that change afterwards fail the
// verification in Write().
if (IsLink(file.FullPath)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security Link check leaves containment race

If another process can mutate the selected source tree, replacing an enumerated file with a symbolic link after IsLink returns but before hashing causes the writer to hash and later package bytes outside sourceRoot; leaving the link stable makes write-time hash verification accept the external payload. How this was verified: the pathname is independently reopened for hashing and writing after the sole reparse-point check, with no no-follow handle or stable identity check.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardened in 0cc1092. The measure-and-hash read is now bracketed by reparse-point checks on both sides (no longer a sole check), and size + hash come from a single handle so the path is resolved once. A link seen at either boundary is rejected before any bytes are packaged. The only residual is the sub-syscall window inside the read itself, which can't be closed without no-follow open semantics that netstandard2.1 doesn't expose; the writer only ever ingests the author's own local tree at editor export time, so this is outside the design's trust boundary (extraction from untrusted packages is the defended side).

Follow-up to review on the content bundle subsystem.

An extraction that spends a long time writing a single large file never
creates another directory entry, so its temp directory's write time
would go stale and a concurrently constructed resolver could reap it
mid-write. The extraction now refreshes its temp directory's write time
every few minutes, keeping it well under the one-hour stale cutoff.

Bracket the source-file measure-and-hash with reparse-point checks on
both sides instead of one, and read size and hash from a single handle
so the path is resolved once. A link present at either boundary is
rejected before its bytes are packaged. The residual is the sub-syscall
window inside the read, which cannot be closed without no-follow open
semantics that netstandard2.1 does not expose; the writer operates on
the author's own local tree at export time.
@freezy
freezy merged commit c5aa041 into master Sep 2, 2026
15 checks passed
@freezy
freezy deleted the feat/packaged-content branch September 2, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant