diff --git a/CHANGELOG.md b/CHANGELOG.md index 477da35..9a0f1b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **transform**: new module **`vtx_transform`** (`VTX_BUILD_TRANSFORM`, on by default) -- the home for every tool that modifies an existing replay (repair, cut, entity filter). Cut and filter read the source through `vtx_reader` and re-serialize through `vtx_writer`'s formatter policies, so it is the one module that links both; `vtx_reader`, `vtx_writer` and `vtx_differ` stay independent siblings on `vtx_common` (the writer does **not** link the reader). Skipped automatically when reader or writer is off; `vtx_inspector` requires it. Public headers under `vtx/transform/`, exported as `VTX::vtx_transform` +- **transform**: **`FilterReplayFile`** (`vtx/transform/vtx_replay_filter.h`) -- writes a copy of a replay with entities dropped (blacklist) or exclusively kept (whitelist) by **bucket** (schema bucket name glob or bucket index; a dropped bucket is emptied in every frame and keeps its positional slot), **unique id glob**, **struct name glob** or **scalar property value** (`struct.field` resolved through the file's schema; empty struct = any struct with that field); rules combine as a union. Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums), and the footer is rebuilt with the same `total_frames`, duration and per-frame time table -- only the seek table changes. Timeline events are carried over and (by default) pruned to entities that still exist. Progress callback with cancel (the partial destination is removed); `ReplayFilterResult` reports kept/dropped counts, per-struct and per-bucket drops, event counts and sizes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for previews on in-memory frames. Both backends, output keeps the source format. Covered by `ReplayFilterTest.*` (both backends: bucket / struct / unique id / property / numeric / case-insensitive rules, keep-nothing, events carried + pruned, error paths, cancel) plus `ReplayFilterGlob` / `ReplayFilterMatcher` unit tests +- **transform**: **`CutReplayFile`** / **`PlanCutFrames`** / **`PlanCutChunks`** (`vtx/transform/vtx_replay_cut.h`) -- the inspector's `ReplayCutService` promoted to an SDK entry point: header copied verbatim, whole chunks copied verbatim with rebased seek entries, edge chunks re-serialized with only the kept frames, footer rebuilt (frames renumbered from 0, time table sliced with absolute ticks, gaps/segments rebased, duration recomputed). Now **both backends** (the inspector cut was FlatBuffers-only) and the per-chunk zstd decision is mirrored from the source instead of forced on. `ReplayCutResult` reports frames, chunks written / rewritten and size; the destination is removed on failure. First tests for the cut -- it was previously untested and never compiled by CI (GUI tools are off there): `ReplayCutTest.*` (both backends: whole-chunk verbatim copy, exact-frame edge rewrite, bound clamping, error paths) and `ReplayCutPlan.*` +- **common/framing**: **`vtx/common/vtx_replay_framing.h`** -- the `.vtx` on-disk framing defined once: size prefixes, header block, footer trailer, the zstd-if-beneficial rule (`ShouldCompress` / `Compress` / `CompressIfBeneficial`), xxHash64 chunk checksums, the zstd sniff, and a layout probe (`ProbeHeader` / `ProbeFooterTrailer` / `ProbeLayout`) plus `CopyRange` for rewriters. `ChunkedFileSink`, `ChunkedNetworkSink`, `RepairReplayFile`, `CutReplayFile`, `FilterReplayFile` and the reader's zstd sniff all go through it -- four independent copies of the framing collapsed into one, so their outputs are byte-identical by construction (the crash-recovery byte-identity suite is the regression net). Covered by `ReplayFraming.*` (the byte layout of every helper, probes on synthetic files) and `ReplayFramingOnDisk.*` (a file recorded by the real sink re-derives its entire seek table through the shared helpers, both backends) +- **writer/serialization**: `SessionFooter::events` -- both footer serializers now write timeline events when the pointer is set. The writer facade still writes none; the replay filter uses it to carry a source file's events into the rewritten footer +- **writer/sinks**: `ChunkedNetworkSink` now fills `checksum` in the seek table it streams (was always 0), the same contract as the file sink -- a receiver that concatenates the stream gets a footer whose chunks verify +- **tools/inspector**: **File > Filter Entities** -- rewrite the loaded replay into a new `.vtx` without the selected entities (or with only them). Bucket and struct checklists with per-item counts for the current frame, unique id globs (one per line), property rules (struct / field / value glob), case-insensitive and event-pruning toggles; a live preview shows how many entities of the current frame the rules keep per bucket and reports unresolvable rules before anything runs. The rewrite runs on a worker thread (`VTX::FilterReplayFile`) with a chunk progress bar and Cancel, then reports frames, kept/dropped entities, per-struct drops, event counts and sizes +- **tools/inspector**: **File > Cut Replay** now works on Protobuf replays too (it runs through `VTX::CutReplayFile`) + +### Changed + +- **transform**: `RepairReplayFile` / `ReplayNeedsRecovery` / `RecoveryJournalPath` move from `vtx_writer` to `vtx_transform` -- include **`vtx/transform/vtx_replay_recovery.h`** (was `vtx/writer/core/vtx_replay_recovery.h`) and link `VTX::vtx_transform`. The rule is now uniform: the writer only produces new recordings, and every tool that modifies an existing `.vtx` (repair, cut, entity filter) lives in the transform module. `RecoveryJournal` / `DurableFile` stay in the writer (they are part of recording). Behaviour and on-disk output are unchanged; the `CrashRecovery*` byte-identity suite moves with it to `tests/transform/`. `vtx_inspector` links `vtx_transform` (not `vtx_writer`) for **File > Repair Replay**; `vtx_benchmarks` now requires `VTX_BUILD_TRANSFORM` + +### Fixed + +- **build**: Linux shared builds (`VTX_BUILD_SHARED=ON`) exported the IXWebSocket and mbedTLS symbols `libvtx_writer.so` links from static archives. A consumer that also links its own copy of those archives (`vtx_tests` does, for its in-process WebSocket server) ended up with one interposed set of their globals, constructed by both images and destroyed twice -- `free(): double free detected in tcache 2` at exit. Latent on `main` (the test executable happened to bind to the `.so`'s copy); surfaced by the link-order change that adding `vtx_transform` to the test executable caused. `libvtx_writer.so` now hides the symbols of every static archive it links (`--exclude-libs,ALL`), keeping a private copy exactly like the Windows DLL always did +- **docs**: `FILE_FORMAT.md` described the 4 bytes before each chunk as a "marker / padding" and the trailing 4 bytes of the file as a "sentinel"; they are the chunk's size prefix and the repeated format magic +- **reader**: `GetFooter().chunk_index[i].checksum` was always 0 on both backends -- the footer-to-native conversions copied every seek-table field except the xxHash64 checksum (the internal seek table used for chunk loading did carry it). Visible as `checksum: 0` for every chunk in the CLI's `chunks` command, and the inspector's cut copied 0 into the seek-table entries of verbatim chunks. The FlatBuffers footer conversion also dropped each timeline event's `entity_unique_id` and `location`. All three conversions now carry the fields; `ReplayFilterTest.DropByStructName` asserts the footer checksums of a freshly written file match the bytes on disk and `ReplayFilterTest.EventsCarriedAndPruned` reads events back with their entity id and location on both backends + ## [0.5.0] - 2026-07-27 ### Added diff --git a/CMakeLists.txt b/CMakeLists.txt index 92095c2..6658985 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -71,6 +71,7 @@ set(VTX_INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/sdk/include") option(VTX_BUILD_WRITER "Build vtx_writer module" ON) option(VTX_BUILD_READER "Build vtx_reader module" ON) option(VTX_BUILD_DIFFER "Build vtx_differ module" ON) +option(VTX_BUILD_TRANSFORM "Build vtx_transform module (repair / cut / entity-filter of existing replays; needs reader + writer)" ON) option(BUILD_VTX_TOOL "Build tool suite (tools/)" ON) option(BUILD_VTX_SAMPLES "Build sample programs (samples/)" ON) option(VTX_BUILD_TESTS "Build unit test suite (tests/)" ON) @@ -118,6 +119,9 @@ endif() if(VTX_BUILD_DIFFER) add_compile_definitions(VTX_HAS_DIFFER) endif() +if(VTX_BUILD_TRANSFORM AND VTX_BUILD_READER AND VTX_BUILD_WRITER) + add_compile_definitions(VTX_HAS_TRANSFORM) +endif() # --- Tools --- if(BUILD_VTX_TOOL AND VTX_BUILD_READER AND VTX_BUILD_WRITER) @@ -145,7 +149,7 @@ else() endif() # --- Benchmarks --- -if(VTX_BUILD_BENCHMARKS AND VTX_BUILD_READER AND VTX_BUILD_WRITER AND VTX_BUILD_DIFFER) +if(VTX_BUILD_BENCHMARKS AND VTX_BUILD_READER AND VTX_BUILD_WRITER AND VTX_BUILD_DIFFER AND VTX_BUILD_TRANSFORM) message(STATUS "VTX: Building benchmarks...") add_subdirectory(benchmarks) else() diff --git a/README.md b/README.md index 69dd5f8..2496acf 100644 --- a/README.md +++ b/README.md @@ -140,16 +140,18 @@ The reader's chunk cache is the single sharpest performance edge in the SDK. A w ## Architecture ``` -vtx_common Core types, schemas, serialization adapters, compression (zstd) +vtx_common Core types, schemas, serialization adapters, compression (zstd), on-disk framing | - +--- vtx_writer Record live frame data into .vtx replay files + +--- vtx_writer Record live frame data into .vtx replay files | - +--- vtx_reader Stream and random-access .vtx replay files with chunk-based caching + +--- vtx_reader Stream and random-access .vtx replay files with chunk-based caching | - +--- vtx_differ Compute structural diffs between frames or replay trees + +--- vtx_differ Compute structural diffs between frames or replay trees + | + +--- vtx_transform Modify existing replays: repair a crashed one, cut a frame range, filter entities (uses reader + writer) ``` -Four static libraries. Enable or disable each with a CMake option. Full module breakdown and extension model in [SDK Architecture](https://github.com/ZenosInteractive/VTX/wiki/SDK-Architecture). +Five static libraries. Enable or disable each with a CMake option. Full module breakdown and extension model in [SDK Architecture](https://github.com/ZenosInteractive/VTX/wiki/SDK-Architecture). ## Tools diff --git a/benchmarks/CMakeLists.txt b/benchmarks/CMakeLists.txt index 1d03045..e89c3ee 100644 --- a/benchmarks/CMakeLists.txt +++ b/benchmarks/CMakeLists.txt @@ -74,6 +74,7 @@ target_link_libraries(vtx_benchmarks PRIVATE vtx_reader vtx_writer vtx_differ + vtx_transform # bench_crash_recovery.cpp: VTX::RepairReplayFile ) target_include_directories(vtx_benchmarks PRIVATE diff --git a/benchmarks/bench_crash_recovery.cpp b/benchmarks/bench_crash_recovery.cpp index 087bcc9..3dff19a 100644 --- a/benchmarks/bench_crash_recovery.cpp +++ b/benchmarks/bench_crash_recovery.cpp @@ -18,7 +18,7 @@ #include "vtx/common/vtx_logger.h" #include "vtx/common/vtx_types.h" #include "vtx/reader/core/vtx_reader_facade.h" -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" #include "vtx/writer/core/vtx_writer_facade.h" // brings SchemaRegistry/SchemaSanitizer for writer.h #include "vtx/writer/core/writer.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ba62576..ffd5420 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -3,16 +3,19 @@ ## Module Dependency Graph ``` -vtx_common Core types, compression, schema, serialization adapters +vtx_common Core types, compression, on-disk framing, schema, serialization adapters | - +--- vtx_writer Writes .vtx replay files (frame recording, chunking, flushing) + +--- vtx_writer Writes .vtx replay files (frame recording, chunking, flushing) | - +--- vtx_reader Reads .vtx replay files (chunk-based caching, random access) + +--- vtx_reader Reads .vtx replay files (chunk-based caching, random access) | - +--- vtx_differ Structural diff between frames (binary tree-diff engine) + +--- vtx_differ Structural diff between frames (binary tree-diff engine) + | + +--- vtx_transform Modifies existing replays (repair a crashed one, cut a frame range, filter entities) + -- the one module that links both vtx_reader and vtx_writer ``` -All four modules are built as static libraries. Each can be toggled independently via CMake options (`VTX_BUILD_WRITER`, `VTX_BUILD_READER`, `VTX_BUILD_DIFFER`). `vtx_common` is always built as the shared foundation. +All five modules are built as static libraries. Each can be toggled via CMake options (`VTX_BUILD_WRITER`, `VTX_BUILD_READER`, `VTX_BUILD_DIFFER`, `VTX_BUILD_TRANSFORM`); writer, reader and differ are independent siblings on `vtx_common`, while `vtx_transform` requires reader and writer. `vtx_common` is always built as the shared foundation. ## Module Overview @@ -30,6 +33,7 @@ Provides the shared type system, serialization infrastructure, and utilities use | Diagnostics | `vtx_diagnostics.h` | One SDK-wide error model: `Severity`, `VtxErrorCode`, `VtxDiagnostic` (`VtxError`/`VtxWarning`), `VtxResult`, `ValidationReport` | | Validation | `vtx_validation.h`, `vtx_frame_accessor.h` | `ValidateSchema` / `ValidateEntity` / `ValidateFrame`; strict accessors `FrameAccessor::TryResolve`, `EntityView::TryGet`, `EntityMutator::TrySet` | | Compression | `vtx_deserializer_service.h` | zstd-based chunk compression/decompression | +| On-disk framing | `vtx_replay_framing.h` | Single definition of the `.vtx` block layout: size prefixes, footer trailer, the zstd-if-beneficial rule, xxHash64 chunk checksums, and a layout probe for rewriters. Shared by both sinks and the `vtx_transform` tools (`RepairReplayFile`, `CutReplayFile`, `FilterReplayFile`), so their output is byte-identical by construction | | Logger | `vtx_logger.h` | Thread-safe singleton logger with `VTX_INFO`, `VTX_WARN`, `VTX_ERROR`, `VTX_DEBUG` macros using `std::format` syntax | | Hashing | `vtx_types_helpers.h` | xxHash64 content hashing for fast entity comparison | | Generated code | `src/generated/` | Protobuf (`.pb.h/.cc`) and FlatBuffers (`_generated.h`) schemas, auto-generated from `schemas/` | @@ -45,7 +49,7 @@ Records live frame data into `.vtx` replay files (or streams the same bytes over - **Config**: `WriterFacadeConfig` (file path) or `NetworkWriterFacadeConfig` (host + port) — chunk size, compression, and the schema supplied as a JSON path, an in-memory JSON string (`schema_json_content`), or a pre-built `SchemaRegistry` (`schema_registry`); `create_output_dirs` (default on) auto-creates missing parent directories of the output file - **Policy**: Template-parameterized writer policies select FlatBuffers or Protobuf serialization at compile time - **Sinks**: `ChunkedFileSink` (file), `ChunkedNetworkSink` (TCP stream) — both produce the **same `.vtx` byte sequence**, so a socket receiver only has to concatenate incoming bytes into a file to get a valid replay -- **Crash durability & recovery** (file sink only): `DurableFile` (FILE* wrapper with real `fsync`; `durable_writes` on by default) makes every chunk power-loss durable, each chunk carries an xxHash64 checksum in the seek table, and `RecoveryJournal` maintains an append-only `.vtx.recovery` write-ahead sidecar (chunk commits, exact per-frame times, in-flight frames) that is deleted on clean `Stop()`. `RepairReplayFile()` (`vtx_replay_recovery.h`) reconstructs a valid file from a crashed recording — footer, seek table, and derived time data byte-exact — with `ReplayNeedsRecovery()` / `RecoveryJournalPath()` as the user-driven detection helpers (never automatic on open). `IFileSinkPerfObserver` (`Config::perf_observer` / `WriterFacadeConfig::perf_observer`) exposes per-stage sink timings (serialize / compress / disk write incl. the durability flush) +- **Crash durability & recovery** (file sink only): `DurableFile` (FILE* wrapper with real `fsync`; `durable_writes` on by default) makes every chunk power-loss durable, each chunk carries an xxHash64 checksum in the seek table, and `RecoveryJournal` maintains an append-only `.vtx.recovery` write-ahead sidecar (chunk commits, exact per-frame times, in-flight frames) that is deleted on clean `Stop()`. Reconstructing a crashed recording from that sidecar is `RepairReplayFile()` in `vtx_transform` (below) — the writer only produces new recordings. `IFileSinkPerfObserver` (`Config::perf_observer` / `WriterFacadeConfig::perf_observer`) exposes per-stage sink timings (serialize / compress / disk write incl. the durability flush) - **Data-source interface**: `IFrameDataSource` (`Initialize()` / `GetNextFrame()` / `GetExpectedTotalFrames()`). Concrete implementations: `samples/advance_write.cpp` (JSON / Protobuf / FlatBuffers from disk); `PipeFrameDataSource` (stdin, Windows named pipes, POSIX FIFOs — including a **server mode** that creates the pipe and waits, for independent game-injector-style external producers); `WebSocketFrameDataSource` (`ws://` + `wss://` with TLS). Both streaming sources share the `IFramePayloadAdapter` concept, so a single adapter plugs into either transport. A `SharedMemoryFrameDataSource` (zero-copy SPSC ring, pluggable `ISharedMemoryTransport`) is present in-tree but **experimental / WIP and not yet functional** — landed unintentionally, not a supported input path - **Dependencies**: protobuf + flatbuffers (serialization), zstd (chunk compression), xxHash/xxh3 (chunk + journal-record checksums), IXWebSocket + mbedTLS (WebSocket transport; hidden behind a PIMPL boundary in `websocket_client.cpp` — never leak into the public SDK headers) @@ -72,6 +76,15 @@ Computes structural diffs between two serialized frames. - **Adapters**: `FlatbufferViewAdapter`, `FProtobufViewAdapter` — zero-copy binary node views into serialized buffers - **Output**: `PatchIndex` — list of `DiffIndexOp` operations (Add, Remove, Replace, ReplaceRange) with binary paths +### vtx_transform + +Every tool that modifies an existing `.vtx` lives here — the writer only produces new recordings, the reader only consumes them. It sits above `vtx_reader` and `vtx_writer`: cut and filter read the source through the reader facade and re-serialize chunks through the writer's formatter policies, repair rebuilds a crashed file from its journal through the writer's `RecoveryJournal`, and every byte is framed via `vtx_replay_framing.h` — which is what lets reader and writer stay independent siblings. Requires both (`VTX_BUILD_TRANSFORM`, on by default; skipped automatically when either is off). + +- **Repair** (`vtx_replay_recovery.h`): `RepairReplayFile` reconstructs a recording that died before `Stop()` from its `.vtx.recovery` sidecar — drops a torn tail chunk, verifies each surviving chunk's checksum, re-appends the in-flight frames and synthesizes the footer with the exact per-frame times (at a chunk boundary the result is byte-identical to a clean `Stop()`). `ReplayNeedsRecovery` / `RecoveryJournalPath` are the user-driven detection helpers — never automatic on open. Needs no reader: it works from the journal and the raw bytes +- **Cut** (`vtx_replay_cut.h`): `PlanCutFrames` / `PlanCutChunks` resolve a frame or chunk range against a footer (cheap, for previews); `CutReplayFile` executes the plan — header copied verbatim, whole chunks copied verbatim with rebased seek entries, edge chunks re-serialized with only the kept frames, footer rebuilt with frames renumbered from 0 and the time table sliced (tick values stay absolute). Both backends; the output keeps the source format +- **Entity filter** (`vtx_replay_filter.h`): `FilterReplayFile` rewrites every chunk with entities dropped (blacklist) or exclusively kept (whitelist) by bucket, unique-id glob, struct-name glob or scalar property value. Every frame survives; header, frame count and time table are preserved and only the seek table changes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for in-memory previews +- These are the SDK entry points behind the inspector's **File > Repair Replay**, **File > Cut Replay** and **File > Filter Entities** (`vtx_inspector` links `vtx_reader` + `vtx_transform`, not `vtx_writer`) + ## Design Patterns ### Facade Pattern @@ -107,7 +120,8 @@ Each module exposes a single abstract interface (`IVtxReaderFacade`, `IVtxWriter | Namespace | Module | Contents | |---|---|---| -| `VTX` | vtx_common, vtx_reader, vtx_writer | Core types, reader/writer facades, format detection, integration primitives (`JsonMapping`, `ProtoBinding`, `FlatBufferBinding`, `IFrameDataSource`) | +| `VTX` | vtx_common, vtx_reader, vtx_writer, vtx_transform | Core types, reader/writer facades, format detection, replay tools (`RepairReplayFile`, `CutReplayFile`, `FilterReplayFile`), integration primitives (`JsonMapping`, `ProtoBinding`, `FlatBufferBinding`, `IFrameDataSource`) | +| `VTX::Framing` | vtx_common | The `.vtx` on-disk framing helpers (`vtx_replay_framing.h`) | | `VtxDiff` | vtx_differ | Diff engine, patch types, binary view adapters | | `VtxDiff::Flatbuffers` | vtx_differ | FlatBuffers binary view adapter | | `VtxDiff::Protobuf` | vtx_differ | Protobuf binary view adapter | diff --git a/docs/BUILD.md b/docs/BUILD.md index f453ca9..488139a 100644 --- a/docs/BUILD.md +++ b/docs/BUILD.md @@ -204,6 +204,7 @@ cmake --build --preset windows-release | `VTX_BUILD_WRITER` | `ON` | Build the writer module | | `VTX_BUILD_READER` | `ON` | Build the reader module | | `VTX_BUILD_DIFFER` | `ON` | Build the differ module | +| `VTX_BUILD_TRANSFORM` | `ON` | Build the transform module (repair / cut / entity filter of existing replays); needs reader + writer, skipped otherwise | | `BUILD_VTX_TOOL` | `ON` | Build tools | | `BUILD_VTX_SAMPLES` | `ON` | Build sample programs | | `VTX_BUILD_TESTS` | `ON` | Build the unit test suite | @@ -242,6 +243,7 @@ Shared-build caveats: 1. Windows still uses `CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS`. 2. If your process also links another protobuf runtime, ABI duplication is your problem. 3. Installed Linux binaries rely on `$ORIGIN/../lib` RPATH. +4. On Linux, `libvtx_writer.so` hides the symbols of the static archives it links (IXWebSocket, mbedTLS, zstd, flatbuffers) with `--exclude-libs,ALL`, exactly as the Windows DLL does implicitly. A process can therefore link its own copy of those libraries without symbol interposition -- two visible copies of a global with a destructor would otherwise be constructed and destroyed twice (a double free at exit). Consumers get zstd/flatbuffers through `vtx_common`'s link interface, never through `vtx_writer`. ## Outputs @@ -255,6 +257,7 @@ dist/ vtx_writer.lib vtx_reader.lib vtx_differ.lib + vtx_transform.lib cmake/VTX/ bin/ vtx_inspector.exe @@ -303,6 +306,7 @@ Imported targets: | `VTX::vtx_writer` | Replay writer | | `VTX::vtx_reader` | Replay reader | | `VTX::vtx_differ` | Frame differ | +| `VTX::vtx_transform` | Tools that modify an existing replay (repair, cut, entity filter) | ## Project Structure diff --git a/docs/FILE_FORMAT.md b/docs/FILE_FORMAT.md index 61f66fa..cc2f96d 100644 --- a/docs/FILE_FORMAT.md +++ b/docs/FILE_FORMAT.md @@ -23,10 +23,12 @@ VTX files store frame-based replay data in a chunked binary format. Each file co +------------------+ | Footer Size (4B) | uint32_t — compressed footer size +------------------+ -| Sentinel (4B) | Padding / alignment +| Magic (4 bytes) | "VTXF" or "VTXP" again — the footer trailer +------------------+ ``` +The framing itself — the size prefixes, the footer trailer, the zstd-if-beneficial rule and the chunk checksums — is defined once in `sdk/include/vtx/common/vtx_replay_framing.h`. Every producer goes through it: the file and network sinks while recording, and the `vtx_transform` tools that modify an existing file (`RepairReplayFile`, `CutReplayFile`, `FilterReplayFile`). Readers locate the blocks the same way (`Framing::ProbeLayout`). + ## Magic Bytes The first 4 bytes identify the serialization format: @@ -66,8 +68,8 @@ The **property schema** embedded in the header defines the layout of `PropertyCo Each chunk contains a batch of serialized frames. Chunks are the unit of I/O and caching. **On disk:** -1. `uint32_t` — chunk marker / padding (4 bytes, skipped on read) -2. Compressed blob — zstd-compressed serialized chunk data +1. `uint32_t` — size in bytes of the payload that follows +2. Payload — serialized chunk data, zstd-compressed when that shrinks it and left raw otherwise (readers sniff the zstd magic rather than trusting a flag) **Chunk contents (after decompression):** - Array of serialized frames, each containing: diff --git a/docs/SDK_API.md b/docs/SDK_API.md index 6b38d62..74d0a60 100644 --- a/docs/SDK_API.md +++ b/docs/SDK_API.md @@ -267,10 +267,10 @@ If the facade is destroyed without an explicit `Stop()`, its destructor finalize A recording that dies before `Stop()` -- process crash or power loss, mid-chunk or mid-frame -- is **recoverable up to the last recorded frame**. While recording, the file sink maintains a write-ahead sidecar **`.vtx.recovery`**: every recorded frame is journaled before it joins the pending batch, and every flushed chunk is committed to the journal *after* its bytes are durable in the `.vtx` (data-before-journal ordering, append-only log, per-record checksums). On a clean `Stop()` the sidecar is deleted -- its presence at open time is the signal of an unclean shutdown. -Recovery is **never automatic**; the flow is user-driven: +Recovery is **never automatic**; the flow is user-driven. The repair entry point lives in `vtx_transform` (link `VTX::vtx_transform`): the writer only produces new recordings, and every tool that modifies an existing `.vtx` sits in that module (see "Rewriting Replays" below): ```cpp -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" if (VTX::ReplayNeedsRecovery(path)) { // cheap: does ".recovery" exist? VTX::RepairResult r = VTX::RepairReplayFile(path); @@ -370,6 +370,63 @@ The writer behaves identically behind the `IVtxWriterFacade` abstraction -- the --- +## Rewriting Replays (`vtx_transform`) + +`vtx_transform` is where every tool that modifies an existing `.vtx` lives -- the writer only produces new recordings. Three entry points: **repair** a crashed recording (`RepairReplayFile`, documented above under *Crash recovery* next to the journal that feeds it), **cut** a frame range and **filter** entities. Cut and filter read the source through the reader and re-serialize through the writer's formatter policies, so it is the one module that links both (`VTX_BUILD_TRANSFORM`, on by default; skipped automatically when reader or writer is off). Both rewrites copy the header block byte for byte, frame every output block through the shared `vtx/common/vtx_replay_framing.h`, support both backends and keep the source format. Neither touches the source -- `dest_path` must be a different file, and a failed or cancelled rewrite removes its partial destination. + +### Cutting a frame range + +`CutReplayFile` writes the frames of a range into a new replay. Plan first -- the plan is cheap, computed from the footer, and is what a UI previews -- then execute it against the same footer: + +```cpp +#include "vtx/transform/vtx_replay_cut.h" + +const VTX::FileFooter footer = reader->GetFooter(); + +VTX::ReplayCutPlan plan = VTX::PlanCutFrames(footer, 1200, 1799); // exact frames, inclusive +// or: VTX::PlanCutChunks(footer, 3, 5); // whole chunks, copied verbatim +if (!plan.valid) { /* plan.error */ } +// plan.first_frame / last_frame, first_chunk / last_chunk, trims_head / trims_tail, +// chunk_bytes (source bytes involved), FrameCount() + +VTX::ReplayCutResult r = VTX::CutReplayFile("match.vtx", footer, plan, "match_round2.vtx"); +if (r.ok()) { + // r.total_frames, r.chunks_written, r.chunks_rewritten (edge chunks re-serialized: 0, 1 or 2), r.output_bytes +} else { + // r.error +} +``` + +Whole chunks inside the range are copied verbatim (same bytes, same checksum; only their seek entry is rebased). When a bound falls inside a chunk, that edge chunk is re-serialized with only the kept frames -- the per-chunk zstd decision is mirrored from the source and the entry gets a fresh xxHash64 -- so exact cuts never decode more than the two edge chunks and whole-chunk cuts never decode a frame. The footer is rebuilt: frames renumbered from 0, the per-frame time table sliced with **absolute** tick values, gaps and segments filtered and rebased, duration recomputed from the kept stamps. Timeline events are not carried over. Bounds are clamped to the replay; a start that falls between chunks snaps to the next covered frame. + +### Filtering entities + +`FilterReplayFile` writes a copy of a replay with some entities removed -- a blacklist (`Drop`: entities matching any rule go) or a whitelist (`Keep`: only matching entities stay). Rules select **whole buckets** (schema bucket name glob, or the bucket index as digits), **unique id globs**, **schema struct name globs**, or the **string form of one scalar property**; several rules combine as a union. Every frame survives: each chunk is re-serialized with the surviving entities (same frame range, same per-chunk compression, fresh xxHash64 checksums), and the footer keeps `total_frames`, `duration_seconds` and the per-frame time table -- only the seek table is rebuilt. Timeline events ride along and are, by default, pruned to the entities that still appear somewhere in the output. + +```cpp +#include "vtx/transform/vtx_replay_filter.h" + +VTX::ReplayFilterSpec spec; +spec.mode = VTX::ReplayFilterMode::Drop; // or Keep (whitelist) +spec.rules.push_back(VTX::ReplayFilterRule::Bucket("SystemHealth")); // every entity of that bucket +spec.rules.push_back(VTX::ReplayFilterRule::StructName("Vehicle*")); // glob on the struct name +spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); // glob on the unique id +spec.rules.push_back(VTX::ReplayFilterRule::Property("Vehicles", "Team", "2")); // one scalar field, as text + +VTX::ReplayFilterResult r = VTX::FilterReplayFile("capture.vtx", "capture_filtered.vtx", spec, + [](int32_t done, int32_t total) { return true; }); // optional; false cancels +if (r.ok()) { + // r.total_frames, r.chunks_rewritten, r.entities_kept / r.entities_dropped, + // r.dropped_by_struct / r.dropped_by_bucket (name -> count), r.events_kept / r.events_dropped, r.output_bytes +} else { + // r.error -- the destination is removed on any failure or cancel +} +``` + +A `Property` rule with an empty struct name means "every struct declaring that field". Matchable field types are Bool (`true`/`false`), Int32, Int64, Float, Double (shortest round-trip decimal) and String; arrays, maps, nested structs and the compound value types cannot be matched, and unresolvable rules (unknown struct or field, empty pattern) fail before anything is written. A `Bucket` rule that matches no schema bucket is an error too, and it lists the available names. Only top-level bucket entities are filtered -- nested containers inside a kept entity are never touched -- and a bucket that ends up empty (including a dropped bucket) stays in place, since bucket index is positional: readers still see the bucket, with no entities. The schema is never pruned: dropping every instance of a struct leaves the struct declared. `ReplayFilterMatcher` exposes the same rule resolution for previews on in-memory frames (`Keeps(bucket_index, bucket, entity_index)`, `Apply(bucket_index, bucket)`, `BucketMatches(bucket_index)`), and `GlobMatch` is the pattern primitive (`*` any run, `?` one character). + +--- + ## Frame Post-Processor A writer-side hook that runs on every `RecordFrame()` call, **after** timer validation and **before** the serializer touches the native `Frame`. Whatever the processor mutates lands on disk -- callers who re-read the `.vtx` see the post-processed values. diff --git a/sdk/include/vtx/common/vtx_replay_framing.h b/sdk/include/vtx/common/vtx_replay_framing.h new file mode 100644 index 0000000..8fb951c --- /dev/null +++ b/sdk/include/vtx/common/vtx_replay_framing.h @@ -0,0 +1,300 @@ +/** + * @file vtx_replay_framing.h + * @brief The .vtx on-disk framing, defined once for every producer and consumer. + * + * @details A .vtx file is laid out as + * + * [magic][u32 header_size][header] [u32 chunk_size][chunk]* [footer][u32 footer_size][magic] + * + * Every blob (header, chunk, footer) is zstd-compressed when that is beneficial and + * left raw otherwise; readers sniff the zstd magic instead of trusting a flag. A + * chunk's seek-table checksum is xxHash64 over its on-disk payload (the bytes after + * the size prefix). + * + * This header is the single definition of those rules. ChunkedFileSink and + * ChunkedNetworkSink (recording) and the vtx_transform tools that modify an existing + * file (RepairReplayFile, CutReplayFile, FilterReplayFile) all frame bytes through it, + * so a format change is made in one place and their outputs stay byte-identical to + * each other by construction. The reader keeps its own header/footer parser (it owns + * the corrupt-file diagnostics) but shares the constants and the zstd sniff. + * + * Byte order: sizes are stored in host byte order, which is little-endian on every + * supported target -- exactly the bytes the format has always shipped. + * + * @author Zenos Interactive + */ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace VTX { + namespace Framing { + + inline constexpr size_t kMagicSize = 4; ///< "VTXF" / "VTXP" + inline constexpr size_t kSizePrefixSize = sizeof(uint32_t); ///< u32 before header / chunk payloads + inline constexpr size_t kFooterTrailerSize = kSizePrefixSize + kMagicSize; ///< [u32 footer_size][magic] + inline constexpr size_t kCompressMinBytes = 512; ///< Smaller payloads stay raw. + inline constexpr int8_t kDefaultCompressionLevel = 10; ///< zstd level of the writer defaults. + inline constexpr unsigned char kZstdMagic[4] = {0x28, 0xB5, 0x2F, 0xFD}; ///< zstd frame magic, as stored. + + // ------------------------------------------------------------------ + // Compression + // ------------------------------------------------------------------ + + /// True if @p data starts with the zstd frame magic. + inline bool LooksLikeZstd(const void* data, size_t size) { + return data != nullptr && size >= sizeof(kZstdMagic) && + std::memcmp(data, kZstdMagic, sizeof(kZstdMagic)) == 0; + } + + inline bool LooksLikeZstd(std::string_view bytes) { + return LooksLikeZstd(bytes.data(), bytes.size()); + } + + /// The writer's rule for attempting compression at all: enabled and not tiny. + inline bool ShouldCompress(size_t payload_size, bool enabled) { + return enabled && payload_size >= kCompressMinBytes; + } + + /// zstd at @p level; returns @p payload unchanged when zstd fails or does not + /// shrink it (readers accept either form). Defined in vtx_common + /// (vtx_replay_framing.cpp) so this header carries no zstd include. + std::string Compress(std::string payload, int8_t level); + + /// ShouldCompress + Compress: the complete zstd-if-beneficial rule. + inline std::string CompressIfBeneficial(std::string payload, bool enabled = true, + int8_t level = kDefaultCompressionLevel) { + if (!ShouldCompress(payload.size(), enabled)) { + return payload; + } + return Compress(std::move(payload), level); + } + + // ------------------------------------------------------------------ + // Checksums + // ------------------------------------------------------------------ + + /// Seek-table checksum of a chunk: xxHash64 over the on-disk payload (after the size prefix). + inline uint64_t PayloadChecksum(const void* data, size_t size) { + return XXH3_64bits(data, size); + } + + inline uint64_t PayloadChecksum(std::string_view payload) { + return PayloadChecksum(payload.data(), payload.size()); + } + + // ------------------------------------------------------------------ + // Size prefixes and framed blocks + // ------------------------------------------------------------------ + + using SizePrefix = std::array; + + /// The u32 prefix exactly as it is written (host byte order). + inline SizePrefix EncodeSizePrefix(uint32_t size) { + SizePrefix out {}; + std::memcpy(out.data(), &size, sizeof(size)); + return out; + } + + inline uint32_t DecodeSizePrefix(const char* bytes) { + uint32_t size = 0; + std::memcpy(&size, bytes, sizeof(size)); + return size; + } + + /// Bytes a chunk occupies on disk: prefix + payload (what ChunkIndexEntry::chunk_size_bytes stores). + inline uint32_t ChunkSizeOnDisk(size_t payload_size) { + return static_cast(payload_size + kSizePrefixSize); + } + + /// [u32 size][payload] + inline std::string ChunkBlock(std::string_view payload) { + std::string block; + block.reserve(kSizePrefixSize + payload.size()); + const SizePrefix prefix = EncodeSizePrefix(static_cast(payload.size())); + block.append(prefix.data(), prefix.size()); + block.append(payload.data(), payload.size()); + return block; + } + + /// [magic][u32 size][payload] + inline std::string HeaderBlock(std::string_view magic, std::string_view payload) { + std::string block; + block.reserve(magic.size() + kSizePrefixSize + payload.size()); + block.append(magic.data(), magic.size()); + const SizePrefix prefix = EncodeSizePrefix(static_cast(payload.size())); + block.append(prefix.data(), prefix.size()); + block.append(payload.data(), payload.size()); + return block; + } + + /// [u32 footer_size][magic] -- the bytes that follow the footer payload. + inline std::string FooterTrailer(uint32_t payload_size, std::string_view magic) { + std::string trailer; + trailer.reserve(kSizePrefixSize + magic.size()); + const SizePrefix prefix = EncodeSizePrefix(payload_size); + trailer.append(prefix.data(), prefix.size()); + trailer.append(magic.data(), magic.size()); + return trailer; + } + + /// [payload][u32 size][magic] + inline std::string FooterBlock(std::string_view payload, std::string_view magic) { + std::string block; + block.reserve(payload.size() + kSizePrefixSize + magic.size()); + block.append(payload.data(), payload.size()); + block += FooterTrailer(static_cast(payload.size()), magic); + return block; + } + + // ------------------------------------------------------------------ + // Locating the blocks of a finished file (rewriters copy bytes by range) + // ------------------------------------------------------------------ + + struct FileLayout { + uint64_t file_size = 0; + uint64_t header_end = 0; ///< First byte after [magic][u32][header]; chunks start here. + uint64_t footer_offset = 0; ///< First byte of the footer payload. + uint32_t footer_size = 0; ///< Footer payload bytes (before the trailer). + bool footer_compressed = false; + }; + + /// Size of the stream in bytes; restores the read position. + inline uint64_t StreamSize(std::istream& in) { + in.clear(); + const std::streampos prev = in.tellg(); + in.seekg(0, std::ios::end); + const std::streamoff end = in.tellg(); + in.clear(); + in.seekg(prev); + return end < 0 ? 0 : static_cast(end); + } + + inline bool ReadAt(std::istream& in, uint64_t offset, void* dst, size_t size) { + in.clear(); + in.seekg(static_cast(offset)); + return static_cast(in.read(static_cast(dst), static_cast(size))); + } + + /// True if the @p available bytes at @p offset start with the zstd magic. + inline bool IsZstdAt(std::istream& in, uint64_t offset, uint64_t available) { + unsigned char magic[sizeof(kZstdMagic)]; + if (available < sizeof(magic) || !ReadAt(in, offset, magic, sizeof(magic))) { + return false; + } + return LooksLikeZstd(magic, sizeof(magic)); + } + + /// Validates [magic][u32 header_size] at offset 0 and yields where the chunks begin. + inline bool ProbeHeader(std::istream& in, std::string_view expected_magic, uint64_t file_size, + uint64_t& header_end, std::string& error) { + if (file_size < expected_magic.size() + kSizePrefixSize) { + error = "The replay is too small to hold a header."; + return false; + } + std::string magic(expected_magic.size(), '\0'); + if (!ReadAt(in, 0, magic.data(), magic.size()) || magic != expected_magic) { + error = "The replay does not start with the expected magic bytes."; + return false; + } + char prefix[kSizePrefixSize]; + if (!ReadAt(in, expected_magic.size(), prefix, sizeof(prefix))) { + error = "Could not read the header size of the replay."; + return false; + } + const uint32_t header_size = DecodeSizePrefix(prefix); + const uint64_t end = expected_magic.size() + kSizePrefixSize + header_size; + if (header_size == 0 || end > file_size) { + error = "The replay header size is implausible; the file is corrupt or truncated."; + return false; + } + header_end = end; + return true; + } + + /// Validates the trailing [u32 footer_size][magic] and yields where the footer payload starts. + /// A footerless (crash-truncated) file fails here. + inline bool ProbeFooterTrailer(std::istream& in, std::string_view expected_magic, uint64_t file_size, + uint64_t& footer_offset, uint32_t& footer_size, std::string& error) { + const size_t trailer_size = kSizePrefixSize + expected_magic.size(); + if (file_size < trailer_size) { + error = "The replay is too small to hold a footer."; + return false; + } + std::string trailer(trailer_size, '\0'); + if (!ReadAt(in, file_size - trailer_size, trailer.data(), trailer.size())) { + error = "Could not read the footer trailer of the replay."; + return false; + } + if (std::string_view(trailer).substr(kSizePrefixSize) != expected_magic) { + error = "The replay has no footer (incomplete recording?). Repair it first."; + return false; + } + const uint32_t size = DecodeSizePrefix(trailer.data()); + if (size == 0 || static_cast(size) + trailer_size > file_size) { + error = "The replay footer size is implausible; the file is corrupt or truncated."; + return false; + } + footer_size = size; + footer_offset = file_size - trailer_size - size; + return true; + } + + /// Header + footer probe of a complete file, plus the footer's compression state. + inline bool ProbeLayout(std::istream& in, std::string_view expected_magic, FileLayout& out, + std::string& error) { + FileLayout layout; + layout.file_size = StreamSize(in); + if (!ProbeHeader(in, expected_magic, layout.file_size, layout.header_end, error)) { + return false; + } + if (!ProbeFooterTrailer(in, expected_magic, layout.file_size, layout.footer_offset, layout.footer_size, + error)) { + return false; + } + if (layout.footer_offset < layout.header_end) { + error = "The replay header and footer overlap; the file is corrupt."; + return false; + } + layout.footer_compressed = IsZstdAt(in, layout.footer_offset, layout.footer_size); + out = layout; + return true; + } + + /// Streams [offset, offset + bytes) from @p in to @p out through a bounded buffer. + inline bool CopyRange(std::istream& in, std::ostream& out, uint64_t offset, uint64_t bytes, + std::string& error) { + constexpr size_t kBufferSize = 4 * 1024 * 1024; + std::vector buffer( + static_cast(std::min(std::max(bytes, 1), kBufferSize))); + in.clear(); + in.seekg(static_cast(offset)); + uint64_t remaining = bytes; + while (remaining > 0) { + const size_t step = static_cast(std::min(remaining, buffer.size())); + if (!in.read(buffer.data(), static_cast(step))) { + error = "Read failed at offset " + std::to_string(offset + (bytes - remaining)) + "."; + return false; + } + if (!out.write(buffer.data(), static_cast(step))) { + error = "Write failed after " + std::to_string(bytes - remaining) + " bytes."; + return false; + } + remaining -= step; + } + return true; + } + + } // namespace Framing +} // namespace VTX diff --git a/sdk/include/vtx/common/vtx_types.h b/sdk/include/vtx/common/vtx_types.h index 128fa70..040ce0e 100644 --- a/sdk/include/vtx/common/vtx_types.h +++ b/sdk/include/vtx/common/vtx_types.h @@ -1216,6 +1216,7 @@ namespace VTX { const std::vector* created_utc = nullptr; const std::vector* gaps = nullptr; const std::vector* segments = nullptr; + const std::vector* events = nullptr; ///< Optional timeline events; null or empty writes none. }; struct ChunkIndexData { diff --git a/sdk/include/vtx/reader/core/vtx_reader.h b/sdk/include/vtx/reader/core/vtx_reader.h index fa3c825..071b508 100644 --- a/sdk/include/vtx/reader/core/vtx_reader.h +++ b/sdk/include/vtx/reader/core/vtx_reader.h @@ -19,6 +19,7 @@ #include "vtx_deserializer_service.h" #include "vtx/common/vtx_diagnostics.h" +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/common/vtx_types_helpers.h" #include "vtx/common/vtx_concepts.h" @@ -760,9 +761,9 @@ namespace VTX { return {}; } - if (raw_buffer.size() <= 4) + if (raw_buffer.size() <= Framing::kSizePrefixSize) return {}; - compressed_blob = raw_buffer.substr(4); + compressed_blob = raw_buffer.substr(Framing::kSizePrefixSize); } if (stop_token.stop_requested()) diff --git a/sdk/include/vtx/transform/vtx_replay_cut.h b/sdk/include/vtx/transform/vtx_replay_cut.h new file mode 100644 index 0000000..36763c7 --- /dev/null +++ b/sdk/include/vtx/transform/vtx_replay_cut.h @@ -0,0 +1,88 @@ +/** + * @file vtx_replay_cut.h + * @brief Write a copy of a .vtx replay holding only a sub-range of its frames. + * + * @details CutReplayFile() slices a finished replay in time: the header block is + * copied byte for byte, chunks fully inside the range are copied verbatim, the two + * edge chunks are re-serialized with only the kept frames when the range starts or + * ends inside them, and the footer is rebuilt -- frame numbering rebased to 0, the + * per-frame time table sliced (tick values stay absolute), gaps and segments + * filtered and rebased, duration recomputed from the kept stamps. Timeline events + * are not carried over. + * + * const VTX::ReplayCutPlan plan = VTX::PlanCutFrames(footer, 1200, 1799); + * if (plan.valid) { + * const auto r = VTX::CutReplayFile("match.vtx", footer, plan, "match_round2.vtx"); + * if (!r.ok()) { ... r.error ... } + * } + * + * The plan is computed from the source footer so a UI can preview the resolved + * chunks/frames before anything is written; CutReplayFile() executes that plan + * against the same footer. Both FlatBuffers and Protobuf files are supported and + * the output keeps the source format. Its sibling FilterReplayFile() slices *what* + * is in every frame instead. + * + * @author Zenos Interactive + */ +#pragma once + +#include +#include + +#include "vtx/common/vtx_types.h" + +namespace VTX { + + /** + * @brief Resolved cut request. + * @details Interior chunks are copied verbatim; when the range starts or ends + * inside a chunk (trims_head / trims_tail) that edge chunk is + * re-serialized with only the kept frames and gets a fresh seek-table + * entry (size, checksum, frame range). + */ + struct ReplayCutPlan { + bool valid = false; + std::string error; ///< Set when !valid. + + int32_t first_chunk = 0; ///< Index into footer.chunk_index. + int32_t last_chunk = 0; + int32_t first_frame = 0; ///< Exact kept range, source frame numbering. + int32_t last_frame = 0; + bool trims_head = false; ///< first_chunk starts before first_frame. + bool trims_tail = false; ///< last_chunk ends after last_frame. + uint64_t chunk_bytes = 0; ///< On-disk bytes of the involved source chunks. + + int32_t FrameCount() const { return valid ? last_frame - first_frame + 1 : 0; } + }; + + /// Exact cut: keeps [start_frame, end_frame] inclusive, trimming inside the edge + /// chunks when the bounds do not fall on chunk boundaries. Bounds are clamped to + /// the replay; a start that falls in a gap between chunks snaps to the next frame. + ReplayCutPlan PlanCutFrames(const FileFooter& footer, int32_t start_frame, int32_t end_frame); + + /// Whole-chunk cut: keeps chunks [first_chunk, last_chunk] untouched. + ReplayCutPlan PlanCutChunks(const FileFooter& footer, int32_t first_chunk, int32_t last_chunk); + + struct ReplayCutResult { + std::string error; ///< Non-empty on failure (the destination is removed). + + int32_t total_frames = 0; ///< Frames in the output. + int32_t chunks_written = 0; ///< Chunks in the output. + int32_t chunks_rewritten = 0; ///< Edge chunks re-serialized (0, 1 or 2). + uint64_t output_bytes = 0; + + bool ok() const { return error.empty(); } + }; + + /** + * @brief Writes @p dest_path as the cut of @p source_path described by @p plan. + * @param footer The source footer the plan was computed from. + * @details The format is taken from the source's magic bytes. When the plan trims + * an edge chunk the source is reopened with the standard reader to + * re-serialize it; whole-chunk cuts never decode a frame. dest_path must + * differ from source_path. + */ + ReplayCutResult CutReplayFile(const std::string& source_path, const FileFooter& footer, const ReplayCutPlan& plan, + const std::string& dest_path); + +} // namespace VTX diff --git a/sdk/include/vtx/transform/vtx_replay_filter.h b/sdk/include/vtx/transform/vtx_replay_filter.h new file mode 100644 index 0000000..bd7983b --- /dev/null +++ b/sdk/include/vtx/transform/vtx_replay_filter.h @@ -0,0 +1,203 @@ +/** + * @file vtx_replay_filter.h + * @brief Write a copy of a .vtx replay with some entities removed. + * + * @details FilterReplayFile() rewrites an existing replay into a new file, keeping + * every frame but dropping (or keeping only) the entities selected by a small rule + * set: whole buckets, unique id globs, schema struct names, or the value of one + * scalar property. The header block (schema, uuid, name, recording timestamp, + * metadata) is copied byte for byte, every chunk is re-serialized with the + * surviving entities, and the footer is rebuilt with the same frame count, duration + * and per-frame time table; only the seek table (offsets, sizes, checksums) + * changes. Timeline events are carried over, optionally pruned to the entities + * that still exist. + * + * VTX::ReplayFilterSpec spec; + * spec.mode = VTX::ReplayFilterMode::Drop; + * spec.rules.push_back(VTX::ReplayFilterRule::Bucket("SystemHealth")); + * spec.rules.push_back(VTX::ReplayFilterRule::StructName("Vehicle*")); + * spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); + * const auto r = VTX::FilterReplayFile("in.vtx", "out.vtx", spec); + * if (!r.ok()) { ... r.error ... } + * + * Buckets are positional (the schema names them by index), so a dropped bucket is + * emptied in every frame rather than removed: readers still see it, with no + * entities. + * + * Part of vtx_transform (VTX_BUILD_TRANSFORM): the source is read through the + * standard reader and re-serialized through the writer's formatter policies, so the + * module needs both vtx_reader and vtx_writer. Its sibling CutReplayFile() slices + * *when* (a frame range) instead of *what*. + * + * @author Zenos Interactive + */ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vtx/common/vtx_property_cache.h" +#include "vtx/common/vtx_types.h" + +namespace VTX { + + /// How the rule set selects the entities that survive. + enum class ReplayFilterMode : uint8_t { + Drop, ///< Blacklist: an entity matching any rule is removed. + Keep, ///< Whitelist: only entities matching at least one rule are kept. + }; + + enum class ReplayFilterRuleKind : uint8_t { + UniqueId, ///< Glob against the entity's unique id (Bucket::unique_ids). + StructName, ///< Glob against the schema struct name of the entity's type id. + Property, ///< Glob against the string form of one scalar property value. + Bucket, ///< Every entity of the buckets whose schema name matches the glob (or whose index equals it). + }; + + /** + * @brief One selection rule. Patterns are globs: '*' matches any run of + * characters, '?' exactly one; everything else is literal. + * @details Property rules resolve `struct_name.field_name` through the file's + * schema; an empty struct_name means "every struct declaring that + * field". Supported field types: Bool ("true"/"false"), Int32, Int64, + * Float, Double (shortest round-trip decimal) and String. Arrays, maps, + * nested structs and the compound value types are not matchable. + * Bucket rules resolve against the schema's bucket names; a pattern made + * only of digits selects the bucket at that index instead (also for + * buckets the schema does not name). A bucket rule matching no bucket + * is an error. + */ + struct ReplayFilterRule { + ReplayFilterRuleKind kind = ReplayFilterRuleKind::UniqueId; + std::string pattern; ///< Glob matched against the id / struct name / value / bucket name. + std::string struct_name; ///< Property only. Empty = any struct with that field. + std::string field_name; ///< Property only. + bool case_insensitive = false; ///< ASCII case-insensitive match. + + static ReplayFilterRule UniqueId(std::string pattern, bool case_insensitive = false); + static ReplayFilterRule StructName(std::string pattern, bool case_insensitive = false); + static ReplayFilterRule Property(std::string struct_name, std::string field_name, std::string value_pattern, + bool case_insensitive = false); + static ReplayFilterRule Bucket(std::string pattern, bool case_insensitive = false); + }; + + struct ReplayFilterSpec { + ReplayFilterMode mode = ReplayFilterMode::Drop; + std::vector rules; ///< Combined as a union: an entity matches if any rule matches. + bool prune_events = true; ///< Drop timeline events whose entity_unique_id no longer appears in any frame. + }; + + struct ReplayFilterResult { + std::string error; ///< Non-empty on failure (the destination is removed). + + int32_t total_frames = 0; ///< Frames in the output (same as the source). + int32_t chunks_rewritten = 0; ///< Chunks re-serialized (all of them). + uint64_t entities_seen = 0; ///< Top-level bucket entities visited, summed over all frames. + uint64_t entities_kept = 0; + uint64_t entities_dropped = 0; + uint64_t events_kept = 0; + uint64_t events_dropped = 0; + uint64_t source_bytes = 0; + uint64_t output_bytes = 0; + double elapsed_seconds = 0.0; + /// Dropped entity count per struct name (entities of an unknown type id are + /// listed as "type#"). + std::map dropped_by_struct; + /// Dropped entity count per bucket (schema name, or "bucket#" when the + /// schema does not name it). + std::map dropped_by_bucket; + + bool ok() const { return error.empty(); } + }; + + /// Called before the first chunk (0, total) and after every rewritten chunk; + /// return false to cancel (the partial destination is removed and the result + /// carries an error). + using ReplayFilterProgress = std::function; + + /// Glob matcher used by the rules ('*' any run, '?' one character). + bool GlobMatch(std::string_view pattern, std::string_view text, bool case_insensitive = false); + + /** + * @brief Resolves a rule set against a file's schema and evaluates entities. + * @details Built once per file; used by FilterReplayFile() and available to + * callers that want to preview a spec on in-memory frames. Every query + * takes the bucket's index within the frame, which is what bucket rules + * and the schema's bucket names key on. + */ + class ReplayFilterMatcher { + public: + ReplayFilterMatcher(const ReplayFilterSpec& spec, const PropertyAddressCache& cache); + + bool valid() const { return error_.empty(); } + const std::string& error() const { return error_; } + ReplayFilterMode mode() const { return mode_; } + + /// True if entity @p index of @p bucket (the frame's bucket number + /// @p bucket_index) matches at least one rule. + bool Matches(int32_t bucket_index, const Bucket& bucket, size_t index) const; + /// Applies the mode: true if the entity survives the filter. + bool Keeps(int32_t bucket_index, const Bucket& bucket, size_t index) const { + return Matches(bucket_index, bucket, index) == (mode_ == ReplayFilterMode::Keep); + } + /// True if a bucket rule selects the whole bucket @p bucket_index. + bool BucketMatches(int32_t bucket_index) const; + + /// Struct name for a type id ("type#" when the schema does not know it). + std::string TypeName(int32_t type_id) const; + /// Schema name of a bucket ("bucket#" when the schema does not name it). + std::string BucketName(int32_t bucket_index) const; + + /** + * @brief Removes every entity Keeps() rejects, in place. + * @param bucket_index The bucket's number within its frame. + * @param on_drop Optional; called with the bucket and the index of each + * dropped entity before it is removed. + * @return Number of removed entities. When anything was removed the bucket's + * type_ranges are cleared (serialization rebuilds them from + * entity_type_id). + */ + size_t Apply(int32_t bucket_index, Bucket& bucket, + const std::function& on_drop = nullptr) const; + + private: + struct ResolvedRule { + ReplayFilterRuleKind kind = ReplayFilterRuleKind::UniqueId; + std::string pattern; + bool case_insensitive = false; + std::unordered_map slots; ///< Property: type id -> address. + std::vector bucket_indices; ///< Bucket: schema buckets matched by name. + int32_t bucket_ordinal = -1; ///< Bucket: numeric pattern, matched by index. + }; + + bool RuleMatches(const ResolvedRule& rule, int32_t bucket_index, const Bucket& bucket, size_t index) const; + + ReplayFilterMode mode_ = ReplayFilterMode::Drop; + std::vector rules_; + std::unordered_map type_names_; + std::vector bucket_names_; + std::string error_; + }; + + /** + * @brief Writes @p dest_path as a copy of @p source_path with the entities + * selected by @p spec removed (Drop) or exclusively kept (Keep). + * @details The source is opened with the standard reader (both FlatBuffers and + * Protobuf files are supported; the output keeps the source format). + * Header bytes are copied verbatim, chunks keep their frame ranges and + * per-chunk compression, the footer keeps total_frames, duration and the + * per-frame time table. Nested containers inside a kept entity are never + * touched; buckets that end up empty (including dropped buckets) stay in + * place, since bucket index is positional. dest_path must differ from + * source_path. + */ + ReplayFilterResult FilterReplayFile(const std::string& source_path, const std::string& dest_path, + const ReplayFilterSpec& spec, const ReplayFilterProgress& progress = nullptr); + +} // namespace VTX diff --git a/sdk/include/vtx/writer/core/vtx_replay_recovery.h b/sdk/include/vtx/transform/vtx_replay_recovery.h similarity index 88% rename from sdk/include/vtx/writer/core/vtx_replay_recovery.h rename to sdk/include/vtx/transform/vtx_replay_recovery.h index ff8bda4..972fbf8 100644 --- a/sdk/include/vtx/writer/core/vtx_replay_recovery.h +++ b/sdk/include/vtx/transform/vtx_replay_recovery.h @@ -19,6 +19,12 @@ * } * auto ctx = VTX::OpenReplayFile(path); * + * Part of vtx_transform (VTX_BUILD_TRANSFORM), next to CutReplayFile() and + * FilterReplayFile(): every tool that modifies an existing .vtx lives there, while the + * writer only produces new recordings. Repair needs no reader -- it works from the + * journal and the raw bytes through the writer's RecoveryJournal and the shared + * framing. + * * @author Zenos Interactive */ #pragma once diff --git a/sdk/include/vtx/writer/policies/sinks/file_sink.h b/sdk/include/vtx/writer/policies/sinks/file_sink.h index 7d2cb7e..48fa3f1 100644 --- a/sdk/include/vtx/writer/policies/sinks/file_sink.h +++ b/sdk/include/vtx/writer/policies/sinks/file_sink.h @@ -6,8 +6,7 @@ #include #include #include -#include -#include +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/common/vtx_concepts.h" #include "vtx/writer/policies/sinks/durable_file.h" @@ -93,9 +92,10 @@ namespace VTX { std::string header_payload = SerializerPolicy::SerializeHeader(config_.header_config, schema); NotifySerialize(serialize_start); header_payload = CompressIfBeneficial(std::move(header_payload)); - uint32_t final_size = static_cast(header_payload.size()); - TimedWrite(&final_size, sizeof(final_size)); - TimedWrite(header_payload.data(), final_size); + const Framing::SizePrefix header_prefix = + Framing::EncodeSizePrefix(static_cast(header_payload.size())); + TimedWrite(header_prefix.data(), header_prefix.size()); + TimedWrite(header_payload.data(), header_payload.size()); TimedSyncOrFlush(); // Start the crash-recovery journal only once the header is durable. If it @@ -140,10 +140,10 @@ namespace VTX { payload = CompressIfBeneficial(std::move(payload)); uint64_t current_offset = file_.Tell(); - uint32_t final_size = static_cast(payload.size()); + const Framing::SizePrefix prefix = Framing::EncodeSizePrefix(static_cast(payload.size())); - TimedWrite(&final_size, sizeof(final_size)); - TimedWrite(payload.data(), final_size); + TimedWrite(prefix.data(), prefix.size()); + TimedWrite(payload.data(), payload.size()); TimedSyncOrFlush(); ChunkIndexData indexEntry; @@ -151,8 +151,8 @@ namespace VTX { indexEntry.file_offset = current_offset; indexEntry.start_frame = start_frame; indexEntry.end_frame = total_frames - 1; - indexEntry.chunk_size_bytes = final_size + sizeof(uint32_t); - indexEntry.checksum = XXH3_64bits(payload.data(), payload.size()); + indexEntry.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + indexEntry.checksum = Framing::PayloadChecksum(payload); seek_table_.push_back(indexEntry); // Commit the chunk to the journal AFTER its bytes are durable on disk @@ -186,9 +186,8 @@ namespace VTX { footer_payload = CompressIfBeneficial(std::move(footer_payload)); TimedWrite(footer_payload.data(), footer_payload.size()); - uint32_t final_size = static_cast(footer_payload.size()); - TimedWrite(&final_size, sizeof(final_size)); - WriteBlob(SerializerPolicy::GetMagicBytes()); + WriteBlob(Framing::FooterTrailer(static_cast(footer_payload.size()), + SerializerPolicy::GetMagicBytes())); TimedSyncOrFlush(); // Clean shutdown: the footer is durable, so the recovery journal is no @@ -234,29 +233,16 @@ namespace VTX { } void WriteBlob(const std::string& data) { TimedWrite(data.data(), data.size()); } + // The shared zstd-if-beneficial rule (vtx_replay_framing.h); only the actual + // zstd call is timed for the perf observer. std::string CompressIfBeneficial(std::string payload) { - if (!config_.b_use_compression || payload.size() < 512) { + if (!Framing::ShouldCompress(payload.size(), config_.b_use_compression)) { return payload; } - const auto compression_start = std::chrono::steady_clock::now(); - size_t const max_size = ZSTD_compressBound(payload.size()); - std::string compressed_blob(max_size, '\0'); - - size_t const compressed_size = ZSTD_compress(compressed_blob.data(), max_size, payload.data(), - payload.size(), config_.compression_level); + payload = Framing::Compress(std::move(payload), config_.compression_level); NotifyCompress(compression_start); - - if (ZSTD_isError(compressed_size)) { - return payload; - } - - if (compressed_size >= payload.size()) { - return payload; - } - - compressed_blob.resize(compressed_size); - return compressed_blob; + return payload; } Config config_; diff --git a/sdk/include/vtx/writer/policies/sinks/network_sink.h b/sdk/include/vtx/writer/policies/sinks/network_sink.h index c0afd67..9c19f8e 100644 --- a/sdk/include/vtx/writer/policies/sinks/network_sink.h +++ b/sdk/include/vtx/writer/policies/sinks/network_sink.h @@ -40,9 +40,9 @@ using vtx_socklen_t = socklen_t; #include #include #include -#include #include "vtx/common/vtx_concepts.h" +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" namespace VTX { @@ -103,8 +103,9 @@ namespace VTX { std::string header_payload = SerializerPolicy::SerializeHeader(config_.header_config, schema); header_payload = CompressIfBeneficial(std::move(header_payload)); - uint32_t final_size = static_cast(header_payload.size()); - SendAll(reinterpret_cast(&final_size), sizeof(final_size)); + const Framing::SizePrefix header_prefix = + Framing::EncodeSizePrefix(static_cast(header_payload.size())); + SendAll(header_prefix.data(), header_prefix.size()); SendAll(header_payload); } @@ -117,8 +118,8 @@ namespace VTX { payload = CompressIfBeneficial(std::move(payload)); uint64_t current_offset = bytes_sent_; - uint32_t final_size = static_cast(payload.size()); - SendAll(reinterpret_cast(&final_size), sizeof(final_size)); + const Framing::SizePrefix prefix = Framing::EncodeSizePrefix(static_cast(payload.size())); + SendAll(prefix.data(), prefix.size()); SendAll(payload); ChunkIndexData entry; @@ -126,7 +127,8 @@ namespace VTX { entry.file_offset = current_offset; entry.start_frame = start_frame; entry.end_frame = total_frames - 1; - entry.chunk_size_bytes = final_size + static_cast(sizeof(uint32_t)); + entry.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + entry.checksum = Framing::PayloadChecksum(payload); // same seek-table contract as the file sink seek_table_.push_back(entry); } @@ -143,9 +145,8 @@ namespace VTX { std::string footer_payload = SerializerPolicy::SerializeFooter(seek_table_, footerData); footer_payload = CompressIfBeneficial(std::move(footer_payload)); SendAll(footer_payload); - uint32_t final_size = static_cast(footer_payload.size()); - SendAll(reinterpret_cast(&final_size), sizeof(final_size)); - SendAll(SerializerPolicy::GetMagicBytes()); + SendAll(Framing::FooterTrailer(static_cast(footer_payload.size()), + SerializerPolicy::GetMagicBytes())); vtx_close_socket(socket_); socket_ = kVtxInvalidSocket; @@ -166,19 +167,8 @@ namespace VTX { void SendAll(const std::string& data) { SendAll(data.data(), data.size()); } std::string CompressIfBeneficial(std::string payload) { - if (!config_.b_use_compression || payload.size() < 512) - return payload; - - size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), config_.compression_level); - - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) - return payload; - - compressed.resize(compressed_size); - return compressed; + return Framing::CompressIfBeneficial(std::move(payload), config_.b_use_compression, + config_.compression_level); } #ifdef _WIN32 diff --git a/sdk/src/CMakeLists.txt b/sdk/src/CMakeLists.txt index c274dc2..0b020ad 100644 --- a/sdk/src/CMakeLists.txt +++ b/sdk/src/CMakeLists.txt @@ -15,3 +15,11 @@ endif() if(VTX_BUILD_DIFFER) add_subdirectory(vtx_differ) endif() + +# Tools that modify an existing replay (repair, cut, entity filter): they read +# through vtx_reader and re-serialize through vtx_writer's policies, so it needs both. +if(VTX_BUILD_TRANSFORM AND VTX_BUILD_READER AND VTX_BUILD_WRITER) + add_subdirectory(vtx_transform) +elseif(VTX_BUILD_TRANSFORM) + message(STATUS "VTX: vtx_transform skipped (requires VTX_BUILD_READER and VTX_BUILD_WRITER).") +endif() diff --git a/sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp b/sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp new file mode 100644 index 0000000..0540bf3 --- /dev/null +++ b/sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp @@ -0,0 +1,25 @@ +#include "vtx/common/vtx_replay_framing.h" + +#include + +#include + +namespace VTX { + namespace Framing { + + // The one zstd call in the framing; kept out of the header so consumers that + // only include reader/common headers need no zstd headers on their path. + std::string Compress(std::string payload, int8_t level) { + const size_t max_size = ZSTD_compressBound(payload.size()); + std::string compressed(max_size, '\0'); + const size_t compressed_size = + ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), level); + if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) { + return payload; + } + compressed.resize(compressed_size); + return compressed; + } + + } // namespace Framing +} // namespace VTX diff --git a/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp b/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp index fe952e1..b99b1c8 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp @@ -3,20 +3,14 @@ #include #include +#include "vtx/common/vtx_replay_framing.h" #include "vtx_schema.pb.h" using namespace VTX; std::string VTX::ReplayUnpacker::Decompress(const std::string& compressed_data) { - //zstd includes a magic number of 4 bytes, if is smaller than that is not a valid zstd buffer - if (compressed_data.size() < 4) { - return compressed_data; - } - - //get magic number - const uint8_t* bytes = reinterpret_cast(compressed_data.data()); - bool is_zstd = (bytes[0] == 0x28 && bytes[1] == 0xB5 && bytes[2] == 0x2F && bytes[3] == 0xFD); - - if (!is_zstd) { + // Blobs are stored raw when zstd would not help; the zstd magic decides (one + // definition for every producer and consumer in vtx_replay_framing.h). + if (!Framing::LooksLikeZstd(compressed_data)) { return compressed_data; } diff --git a/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp b/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp index 33b9086..fb33b35 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp @@ -179,6 +179,7 @@ VTX::FileFooter VTX::FlatBuffersReaderPolicy::GetVTXFooter(const FooterType& fbs entry.end_frame = chunk->end_frame; entry.file_offset = chunk->file_offset; + entry.checksum = chunk->checksum; out.chunk_index.push_back(entry); } } @@ -190,6 +191,12 @@ VTX::FileFooter VTX::FlatBuffersReaderPolicy::GetVTXFooter(const FooterType& fbs out_ev.game_time = ev->game_time; out_ev.event_type = ev->event_type; out_ev.label = ev->label; + out_ev.entity_unique_id = ev->entity_unique_id; + if (ev->location) { + out_ev.location.x = ev->location->x(); + out_ev.location.y = ev->location->y(); + out_ev.location.z = ev->location->z(); + } out.events.push_back(out_ev); } } diff --git a/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp b/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp index 2473175..c67a2be 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp @@ -151,6 +151,7 @@ VTX::FileFooter VTX::ProtobufReaderPolicy::GetVTXFooter(const FooterType& pb_foo entry.end_frame = pb_chunk.end_frame(); entry.file_offset = pb_chunk.file_offset(); entry.chunk_size_bytes = pb_chunk.chunk_size_bytes(); + entry.checksum = pb_chunk.checksum(); out.chunk_index.push_back(entry); } diff --git a/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp b/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp index 78ca304..9d9abd4 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp @@ -254,6 +254,7 @@ void VTX::Serialization::FromFlat(const fbsvtx::ChunkIndexEntry* src, VTX::Chunk dst.end_frame = src->end_frame(); dst.file_offset = src->file_offset(); dst.chunk_size_bytes = src->chunk_size_bytes(); + dst.checksum = src->checksum(); } void VTX::Serialization::FromFlat(const fbsvtx::ReplayTimeData* src, VTX::ReplayTimeData& dst) { diff --git a/sdk/src/vtx_transform/CMakeLists.txt b/sdk/src/vtx_transform/CMakeLists.txt new file mode 100644 index 0000000..91de2dc --- /dev/null +++ b/sdk/src/vtx_transform/CMakeLists.txt @@ -0,0 +1,67 @@ +project(vtx_transform) + +# ============================================================================== +# vtx_transform -- tools that modify an existing .vtx (repair, cut, entity filter). +# ============================================================================== +# Sits ABOVE vtx_reader and vtx_writer: cut and filter read a finished .vtx +# through the reader and re-serialize through the writer's formatter policies; +# repair reconstructs a crashed one from its recovery journal (writer sink +# headers, no reader). All of them frame bytes with the shared +# vtx/common/vtx_replay_framing.h. Keeping them in their own module leaves reader +# and writer independent siblings on vtx_common -- the writer only produces new +# recordings. Requires VTX_BUILD_READER and VTX_BUILD_WRITER (enforced in sdk/src). +# ============================================================================== + +# ------------------------------------------------------------------------------ +# 1. Source files and headers +# ------------------------------------------------------------------------------ +file(GLOB_RECURSE HEADERS CONFIGURE_DEPENDS + "${VTX_SDK_SOURCE_DIR}/sdk/include/vtx/transform/*.h" + "${VTX_SDK_SOURCE_DIR}/sdk/include/vtx/transform/*.hpp" +) + +file(GLOB_RECURSE SOURCES CONFIGURE_DEPENDS + "${CMAKE_CURRENT_SOURCE_DIR}/*.cpp" +) + +# ------------------------------------------------------------------------------ +# 2. Target and deps +# ------------------------------------------------------------------------------ +# Library type (STATIC/SHARED) follows BUILD_SHARED_LIBS (VTX_BUILD_SHARED). +add_library(vtx_transform ${SOURCES} ${HEADERS}) + +target_include_directories(vtx_transform + PRIVATE + "${VTX_COMMON_GENERATED_DIR}" +) + +# The public headers only need vtx_common types; reader and writer are +# implementation details (they still propagate as link dependencies). +target_link_libraries(vtx_transform + PUBLIC + vtx_common + PRIVATE + vtx_reader + vtx_writer + $ + $ + $ +) + +# See note in vtx_reader/CMakeLists.txt -- Windows shared DLLs duplicate the +# generated protobuf/flatbuffers objects into every module that references them. +if(BUILD_SHARED_LIBS AND WIN32) + target_link_libraries(vtx_transform PRIVATE $) +endif() + +# ------------------------------------------------------------------------------ +# 3. VS filters +# ------------------------------------------------------------------------------ +source_group(TREE "${VTX_SDK_SOURCE_DIR}/sdk/include" PREFIX "Header Files" FILES ${HEADERS}) +source_group(TREE "${CMAKE_CURRENT_SOURCE_DIR}" PREFIX "Source Files" FILES ${SOURCES}) + +install(TARGETS vtx_transform EXPORT VTXTargets + ARCHIVE DESTINATION lib + LIBRARY DESTINATION lib + RUNTIME DESTINATION bin +) diff --git a/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp new file mode 100644 index 0000000..93f56f3 --- /dev/null +++ b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp @@ -0,0 +1,364 @@ +#include "vtx/transform/vtx_replay_cut.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers +#include "vtx_schema.pb.h" + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" +#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" +#include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" + +namespace VTX { + + namespace { + + // Duration of a sliced tick table (first..last nonzero stamp), or negative when + // the table cannot answer. + double DurationFromTicks(const std::vector& ticks) { + int64_t first = 0; + int64_t last = 0; + for (const int64_t t : ticks) { + if (t != 0) { + if (first == 0) { + first = t; + } + last = t; + } + } + if (first == 0 || last < first) { + return -1.0; + } + return static_cast(last - first) / static_cast(GameTime::TICKS_PER_SECOND); + } + + ReplayCutPlan PlanForChunkSpan(const FileFooter& footer, int32_t first_chunk, int32_t last_chunk, + int32_t first_frame, int32_t last_frame) { + ReplayCutPlan plan; + plan.first_chunk = first_chunk; + plan.last_chunk = last_chunk; + plan.first_frame = first_frame; + plan.last_frame = last_frame; + plan.trims_head = footer.chunk_index[static_cast(first_chunk)].start_frame < first_frame; + plan.trims_tail = footer.chunk_index[static_cast(last_chunk)].end_frame > last_frame; + for (int32_t i = first_chunk; i <= last_chunk; ++i) { + plan.chunk_bytes += footer.chunk_index[static_cast(i)].chunk_size_bytes; + } + plan.valid = true; + return plan; + } + + // Frames [kept_lo, kept_hi] of the source re-serialized as one chunk through the + // shared framing: per-chunk zstd decision mirrored from the source chunk, fresh + // seek-table entry. + template + bool WriteRewrittenChunk(IVtxReaderFacade& reader, std::ofstream& out, int32_t kept_lo, int32_t kept_hi, + int32_t new_chunk_index, int32_t rebase_frame, bool compress, + ChunkIndexData& entry_out, std::string& error) { + std::vector> frames; + frames.reserve(static_cast(kept_hi - kept_lo + 1)); + for (int32_t frame_index = kept_lo; frame_index <= kept_hi; ++frame_index) { + const Frame* source_frame = reader.GetFrameSync(frame_index); + if (!source_frame) { + error = "Could not read frame " + std::to_string(frame_index) + " from the source replay."; + return false; + } + frames.push_back(Policy::FromNative(Frame(*source_frame))); + } + + std::string payload = Policy::SerializeChunk(frames, new_chunk_index, compress); + payload = Framing::CompressIfBeneficial(std::move(payload), compress, Framing::kDefaultCompressionLevel); + + entry_out.chunk_index = new_chunk_index; + entry_out.file_offset = static_cast(out.tellp()); + entry_out.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + entry_out.start_frame = kept_lo - rebase_frame; + entry_out.end_frame = kept_hi - rebase_frame; + entry_out.checksum = Framing::PayloadChecksum(payload); + + const std::string block = Framing::ChunkBlock(payload); + if (!out.write(block.data(), static_cast(block.size()))) { + error = "Write failed for rewritten chunk " + std::to_string(new_chunk_index) + "."; + return false; + } + return true; + } + + template + void RunCut(const std::string& source_path, const FileFooter& footer, const ReplayCutPlan& plan, + const std::string& dest_path, ReplayCutResult& result) { + namespace fs = std::filesystem; + std::string error; + + // Edge chunks that start/end inside the kept range are re-serialized frame + // by frame; open a private reader for that (a caller's reader may be busy on + // its own thread). Whole-chunk cuts never decode a frame. + ReaderContext reader_context; + if (plan.trims_head || plan.trims_tail) { + reader_context = OpenReplayFile(source_path); + if (!reader_context.Loaded()) { + result.error = "Could not reopen the source replay for edge-chunk rewriting: " + + reader_context.GetError().message; + return; + } + if (!reader_context.WaitUntilReady()) { + result.error = "The source replay failed to load: " + reader_context.GetReadyError().message; + return; + } + } + + std::ifstream in(source_path, std::ios::binary); + if (!in) { + result.error = "Could not open the source replay: " + source_path; + return; + } + Framing::FileLayout layout; + if (!Framing::ProbeLayout(in, Policy::GetMagicBytes(), layout, error)) { + result.error = error; + return; + } + for (int32_t i = plan.first_chunk; i <= plan.last_chunk; ++i) { + const ChunkIndexEntry& chunk = footer.chunk_index[static_cast(i)]; + if (chunk.chunk_size_bytes <= Framing::kSizePrefixSize || chunk.file_offset < layout.header_end || + chunk.file_offset + chunk.chunk_size_bytes > layout.footer_offset) { + result.error = "The source replay seek table is inconsistent with the file layout (chunk " + + std::to_string(chunk.chunk_index) + ")."; + return; + } + } + + std::ofstream out(dest_path, std::ios::binary | std::ios::trunc); + if (!out) { + result.error = "Could not create the output file: " + dest_path; + return; + } + const auto abort_output = [&](std::string message) { + out.close(); + std::error_code ec; + fs::remove(dest_path, ec); + result.error = std::move(message); + }; + + // 1) Header block, verbatim: it carries no frame-dependent data, so copying + // it keeps the schema and recording metadata bit-identical. + if (!Framing::CopyRange(in, out, 0, layout.header_end, error)) { + abort_output(error); + return; + } + + // 2) Chunks. Whole chunks are copied verbatim (their seek entry is rebased); + // partial edge chunks are rebuilt with only the kept frames. + std::vector seek_table; + seek_table.reserve(static_cast(plan.last_chunk - plan.first_chunk + 1)); + for (int32_t i = plan.first_chunk; i <= plan.last_chunk; ++i) { + const ChunkIndexEntry& entry = footer.chunk_index[static_cast(i)]; + const int32_t kept_lo = std::max(entry.start_frame, plan.first_frame); + const int32_t kept_hi = std::min(entry.end_frame, plan.last_frame); + const int32_t new_index = static_cast(seek_table.size()); + + if (kept_lo == entry.start_frame && kept_hi == entry.end_frame) { + ChunkIndexData rebased; + rebased.chunk_index = new_index; + rebased.file_offset = static_cast(out.tellp()); + rebased.chunk_size_bytes = entry.chunk_size_bytes; + rebased.start_frame = entry.start_frame - plan.first_frame; + rebased.end_frame = entry.end_frame - plan.first_frame; + rebased.checksum = entry.checksum; + if (!Framing::CopyRange(in, out, entry.file_offset, entry.chunk_size_bytes, error)) { + abort_output(error); + return; + } + seek_table.push_back(rebased); + } else { + const bool compress = Framing::IsZstdAt(in, entry.file_offset + Framing::kSizePrefixSize, + entry.chunk_size_bytes - Framing::kSizePrefixSize); + ChunkIndexData rewritten; + if (!WriteRewrittenChunk(*reader_context.reader, out, kept_lo, kept_hi, new_index, + plan.first_frame, compress, rewritten, error)) { + abort_output(error); + return; + } + seek_table.push_back(rewritten); + ++result.chunks_rewritten; + } + } + + // 3) Footer: time table sliced to the kept frames (tick values stay absolute, + // only frame indices are rebased), duration recomputed, fresh seek table. + const auto slice_ticks = [&](const std::vector& ticks) { + std::vector sliced; + const size_t begin = std::min(static_cast(plan.first_frame), ticks.size()); + const size_t end = std::min(static_cast(plan.last_frame) + 1, ticks.size()); + sliced.reserve(end - begin); + for (size_t i = begin; i < end; ++i) { + sliced.push_back(static_cast(ticks[i])); + } + return sliced; + }; + const auto slice_frame_indices = [&](const std::vector& values) { + std::vector sliced; + for (const uint32_t value : values) { + const int32_t frame = static_cast(value); + if (frame >= plan.first_frame && frame <= plan.last_frame) { + sliced.push_back(frame - plan.first_frame); + } + } + return sliced; + }; + + const std::vector game_times = slice_ticks(footer.times.game_time); + const std::vector created_utc = slice_ticks(footer.times.created_utc); + const std::vector gaps = slice_frame_indices(footer.times.gaps); + const std::vector segments = slice_frame_indices(footer.times.segments); + + SessionFooter session_footer; + session_footer.total_frames = plan.FrameCount(); + double duration = DurationFromTicks(created_utc); + if (duration < 0.0) { + duration = DurationFromTicks(game_times); + } + if (duration < 0.0 && footer.total_frames > 0) { + duration = + static_cast(footer.duration_seconds) * session_footer.total_frames / footer.total_frames; + } + session_footer.duration_seconds = std::max(duration, 0.0); + session_footer.game_times = &game_times; + session_footer.created_utc = &created_utc; + session_footer.gaps = &gaps; + session_footer.segments = &segments; + + std::string footer_payload = Policy::SerializeFooter(seek_table, session_footer); + footer_payload = Framing::CompressIfBeneficial(std::move(footer_payload), layout.footer_compressed, + Framing::kDefaultCompressionLevel); + const std::string footer_block = Framing::FooterBlock(footer_payload, Policy::GetMagicBytes()); + if (!out.write(footer_block.data(), static_cast(footer_block.size()))) { + abort_output("Write failed while finishing the footer."); + return; + } + out.flush(); + if (!out) { + abort_output("Flush failed for the output file."); + return; + } + out.close(); + + result.total_frames = session_footer.total_frames; + result.chunks_written = static_cast(seek_table.size()); + std::error_code ec; + result.output_bytes = fs::file_size(dest_path, ec); + } + + } // namespace + + ReplayCutPlan PlanCutFrames(const FileFooter& footer, int32_t start_frame, int32_t end_frame) { + ReplayCutPlan plan; + if (footer.chunk_index.empty() || footer.total_frames <= 0) { + plan.error = "The replay has no chunk index."; + return plan; + } + start_frame = std::clamp(start_frame, 0, footer.total_frames - 1); + end_frame = std::clamp(end_frame, 0, footer.total_frames - 1); + if (end_frame < start_frame) { + plan.error = "End of range is before its start."; + return plan; + } + + int32_t first_chunk = -1; + int32_t last_chunk = -1; + for (int32_t i = 0; i < static_cast(footer.chunk_index.size()); ++i) { + const auto& entry = footer.chunk_index[static_cast(i)]; + if (first_chunk < 0 && start_frame <= entry.end_frame) { + first_chunk = i; + } + if (end_frame >= entry.start_frame) { + last_chunk = i; + } + } + if (first_chunk < 0 || last_chunk < first_chunk) { + plan.error = "The requested range does not overlap any chunk."; + return plan; + } + // The exact bounds are clamped into the covered span (a start that falls in a + // gap between chunks snaps to the first covered frame). + const int32_t lo = std::max(start_frame, footer.chunk_index[static_cast(first_chunk)].start_frame); + const int32_t hi = std::min(end_frame, footer.chunk_index[static_cast(last_chunk)].end_frame); + return PlanForChunkSpan(footer, first_chunk, last_chunk, lo, hi); + } + + ReplayCutPlan PlanCutChunks(const FileFooter& footer, int32_t first_chunk, int32_t last_chunk) { + ReplayCutPlan plan; + if (footer.chunk_index.empty()) { + plan.error = "The replay has no chunk index."; + return plan; + } + const int32_t max_chunk = static_cast(footer.chunk_index.size()) - 1; + first_chunk = std::clamp(first_chunk, 0, max_chunk); + last_chunk = std::clamp(last_chunk, 0, max_chunk); + if (last_chunk < first_chunk) { + plan.error = "End chunk is before start chunk."; + return plan; + } + return PlanForChunkSpan(footer, first_chunk, last_chunk, + footer.chunk_index[static_cast(first_chunk)].start_frame, + footer.chunk_index[static_cast(last_chunk)].end_frame); + } + + ReplayCutResult CutReplayFile(const std::string& source_path, const FileFooter& footer, const ReplayCutPlan& plan, + const std::string& dest_path) { + namespace fs = std::filesystem; + ReplayCutResult result; + + if (!plan.valid) { + result.error = plan.error.empty() ? "Invalid cut plan." : plan.error; + return result; + } + if (source_path.empty() || dest_path.empty()) { + result.error = "Source and destination paths are required."; + return result; + } + if (plan.first_chunk < 0 || plan.last_chunk < plan.first_chunk || + static_cast(plan.last_chunk) >= footer.chunk_index.size()) { + result.error = "The cut plan does not match the footer's chunk index."; + return result; + } + std::error_code ec; + if (!fs::is_regular_file(source_path, ec)) { + result.error = "Source replay does not exist: " + source_path; + return result; + } + const fs::path source_abs = fs::weakly_canonical(fs::path(source_path), ec); + const fs::path dest_abs = fs::weakly_canonical(fs::path(dest_path), ec); + if (source_abs == dest_abs || (fs::exists(dest_path, ec) && fs::equivalent(source_path, dest_path, ec))) { + result.error = "Destination must be a different file than the source."; + return result; + } + + // The format is decided by the source's leading magic. + std::string magic(Framing::kMagicSize, '\0'); + { + std::ifstream in(source_path, std::ios::binary); + if (!in || !in.read(magic.data(), static_cast(magic.size()))) { + result.error = "Could not read the source replay: " + source_path; + return result; + } + } + if (magic == FlatBuffersVtxPolicy::GetMagicBytes()) { + RunCut(source_path, footer, plan, dest_path, result); + } else if (magic == ProtobufVtxPolicy::GetMagicBytes()) { + RunCut(source_path, footer, plan, dest_path, result); + } else { + result.error = "The source replay does not start with a known VTX magic."; + } + return result; + } + +} // namespace VTX diff --git a/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_filter.cpp b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_filter.cpp new file mode 100644 index 0000000..8941333 --- /dev/null +++ b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_filter.cpp @@ -0,0 +1,645 @@ +#include "vtx/transform/vtx_replay_filter.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers +#include "vtx_schema.pb.h" + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" +#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" +#include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" + +namespace VTX { + + namespace { + + char ToLowerAscii(char c) { + return static_cast(std::tolower(static_cast(c))); + } + + bool CharEquals(char a, char b, bool case_insensitive) { + return case_insensitive ? ToLowerAscii(a) == ToLowerAscii(b) : a == b; + } + + template + std::string ShortestDecimal(T value) { + char buffer[64]; + const auto res = std::to_chars(buffer, buffer + sizeof(buffer), value); + if (res.ec != std::errc {}) { + return {}; + } + return std::string(buffer, res.ptr); + } + + bool IsMatchableField(const PropertyAddress& address) { + if (address.container_type != FieldContainerType::None || address.index < 0) { + return false; + } + switch (address.type_id) { + case FieldType::Bool: + case FieldType::Int32: + case FieldType::Int64: + case FieldType::Float: + case FieldType::Double: + case FieldType::String: + return true; + default: + return false; + } + } + + // String form of one scalar property; false when the slot is absent. + bool PropertyValueToString(const PropertyContainer& entity, const PropertyAddress& address, std::string& out) { + const size_t i = static_cast(address.index); + switch (address.type_id) { + case FieldType::Bool: + if (i >= entity.bool_properties.size()) + return false; + out = entity.bool_properties[i] ? "true" : "false"; + return true; + case FieldType::Int32: + if (i >= entity.int32_properties.size()) + return false; + out = std::to_string(entity.int32_properties[i]); + return true; + case FieldType::Int64: + if (i >= entity.int64_properties.size()) + return false; + out = std::to_string(entity.int64_properties[i]); + return true; + case FieldType::Float: + if (i >= entity.float_properties.size()) + return false; + out = ShortestDecimal(entity.float_properties[i]); + return true; + case FieldType::Double: + if (i >= entity.double_properties.size()) + return false; + out = ShortestDecimal(entity.double_properties[i]); + return true; + case FieldType::String: + if (i >= entity.string_properties.size()) + return false; + out = entity.string_properties[i]; + return true; + default: + return false; + } + } + + const char* KindName(ReplayFilterRuleKind kind) { + switch (kind) { + case ReplayFilterRuleKind::UniqueId: + return "unique id"; + case ReplayFilterRuleKind::StructName: + return "struct"; + case ReplayFilterRuleKind::Property: + return "property"; + case ReplayFilterRuleKind::Bucket: + return "bucket"; + } + return "rule"; + } + + // ------------------------------------------------------------------ + // Rewrite. File framing (size prefixes, footer trailer, zstd rule, chunk + // checksums, layout probe) is the shared vtx_replay_framing.h. + // ------------------------------------------------------------------ + + template + void RunFilter(const std::string& source_path, const std::string& dest_path, IVtxReaderFacade& reader, + const FileFooter& footer, const ReplayFilterMatcher& matcher, bool prune_events, + const ReplayFilterProgress& progress, ReplayFilterResult& result) { + namespace fs = std::filesystem; + std::string error; + const auto fail = [&](std::string message) { + result.error = std::move(message); + }; + + std::ifstream in(source_path, std::ios::binary); + if (!in) { + fail("Could not open the source replay: " + source_path); + return; + } + Framing::FileLayout layout; + if (!Framing::ProbeLayout(in, Policy::GetMagicBytes(), layout, error)) { + fail(error); + return; + } + + // Chunks in frame order. Frame ranges and per-chunk compression are kept + // exactly; only the bytes (and therefore offsets/sizes/checksums) change. + std::vector chunks = footer.chunk_index; + std::sort(chunks.begin(), chunks.end(), + [](const ChunkIndexEntry& a, const ChunkIndexEntry& b) { return a.start_frame < b.start_frame; }); + for (const ChunkIndexEntry& chunk : chunks) { + if (chunk.end_frame < chunk.start_frame || chunk.chunk_size_bytes <= Framing::kSizePrefixSize || + chunk.file_offset < layout.header_end || + chunk.file_offset + chunk.chunk_size_bytes > layout.footer_offset) { + fail("The source replay seek table is inconsistent with the file layout (chunk " + + std::to_string(chunk.chunk_index) + ")."); + return; + } + } + if (!chunks.empty() && chunks.front().file_offset != layout.header_end) { + fail("The source replay has an unexpected layout between its header and first chunk."); + return; + } + + std::ofstream out(dest_path, std::ios::binary | std::ios::trunc); + if (!out) { + fail("Could not create the output file: " + dest_path); + return; + } + const auto abort_output = [&](std::string message) { + out.close(); + std::error_code ec; + fs::remove(dest_path, ec); + fail(std::move(message)); + }; + + // 1) Header block, verbatim (schema, uuid, name, recording timestamp, + // metadata). Nothing in it depends on the entity payload. + if (!Framing::CopyRange(in, out, 0, layout.header_end, error)) { + abort_output(error); + return; + } + + const int32_t chunk_total = static_cast(chunks.size()); + if (progress && !progress(0, chunk_total)) { + abort_output("Filter cancelled."); + return; + } + + const bool track_survivors = prune_events && !footer.events.empty(); + std::unordered_set survivors; + + // 2) Every chunk re-serialized with the surviving entities. + std::vector seek_table; + seek_table.reserve(chunks.size()); + for (int32_t i = 0; i < chunk_total; ++i) { + const ChunkIndexEntry& chunk = chunks[static_cast(i)]; + std::vector> frames; + frames.reserve(static_cast(chunk.end_frame - chunk.start_frame + 1)); + + for (int32_t frame_index = chunk.start_frame; frame_index <= chunk.end_frame; ++frame_index) { + const Frame* source_frame = reader.GetFrameSync(frame_index); + if (!source_frame) { + abort_output("Could not read frame " + std::to_string(frame_index) + + " from the source replay."); + return; + } + Frame frame(*source_frame); + std::vector& buckets = frame.GetMutableBuckets(); + for (size_t b = 0; b < buckets.size(); ++b) { + Bucket& bucket = buckets[b]; + const int32_t bucket_index = static_cast(b); + const size_t before = bucket.entities.size(); + const size_t dropped = + matcher.Apply(bucket_index, bucket, [&](const Bucket& dropped_from, size_t index) { + ++result + .dropped_by_struct[matcher.TypeName(dropped_from.entities[index].entity_type_id)]; + }); + result.entities_seen += before; + result.entities_dropped += dropped; + result.entities_kept += before - dropped; + if (dropped > 0) { + result.dropped_by_bucket[matcher.BucketName(bucket_index)] += dropped; + } + if (track_survivors) { + for (const std::string& id : bucket.unique_ids) { + survivors.insert(id); + } + } + } + frames.push_back(Policy::FromNative(std::move(frame))); + } + + // Per-chunk zstd decision mirrored from the source chunk's bytes. + const bool compress = Framing::IsZstdAt(in, chunk.file_offset + Framing::kSizePrefixSize, + chunk.chunk_size_bytes - Framing::kSizePrefixSize); + std::string payload = Policy::SerializeChunk(frames, i, compress); + payload = + Framing::CompressIfBeneficial(std::move(payload), compress, Framing::kDefaultCompressionLevel); + + ChunkIndexData entry; + entry.chunk_index = i; + entry.file_offset = static_cast(out.tellp()); + entry.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + entry.start_frame = chunk.start_frame; + entry.end_frame = chunk.end_frame; + entry.checksum = Framing::PayloadChecksum(payload); + + const std::string block = Framing::ChunkBlock(payload); + if (!out.write(block.data(), static_cast(block.size()))) { + abort_output("Write failed for chunk " + std::to_string(i) + "."); + return; + } + seek_table.push_back(entry); + ++result.chunks_rewritten; + + if (progress && !progress(i + 1, chunk_total)) { + abort_output("Filter cancelled."); + return; + } + } + + // 3) Footer: same frames, same duration, same per-frame time table; + // fresh seek table; events carried over (pruned to surviving entities). + std::vector game_times(footer.times.game_time.begin(), footer.times.game_time.end()); + std::vector created_utc(footer.times.created_utc.begin(), footer.times.created_utc.end()); + std::vector gaps(footer.times.gaps.begin(), footer.times.gaps.end()); + std::vector segments(footer.times.segments.begin(), footer.times.segments.end()); + + std::vector events; + events.reserve(footer.events.size()); + for (const TimelineEvent& event : footer.events) { + if (track_survivors && !event.entity_unique_id.empty() && !survivors.contains(event.entity_unique_id)) { + ++result.events_dropped; + continue; + } + events.push_back(event); + ++result.events_kept; + } + + SessionFooter session_footer; + session_footer.total_frames = footer.total_frames; + session_footer.duration_seconds = footer.duration_seconds; + const bool has_times = !game_times.empty() || !created_utc.empty() || !gaps.empty() || !segments.empty(); + if (has_times) { + session_footer.game_times = &game_times; + session_footer.created_utc = &created_utc; + session_footer.gaps = &gaps; + session_footer.segments = &segments; + } + session_footer.events = &events; + + std::string footer_payload = Policy::SerializeFooter(seek_table, session_footer); + footer_payload = Framing::CompressIfBeneficial(std::move(footer_payload), layout.footer_compressed, + Framing::kDefaultCompressionLevel); + const std::string footer_block = Framing::FooterBlock(footer_payload, Policy::GetMagicBytes()); + if (!out.write(footer_block.data(), static_cast(footer_block.size()))) { + abort_output("Write failed while finishing the footer."); + return; + } + out.flush(); + if (!out) { + abort_output("Flush failed for the output file."); + return; + } + out.close(); + + result.total_frames = footer.total_frames; + std::error_code ec; + result.output_bytes = fs::file_size(dest_path, ec); + } + + } // namespace + + // ---------------------------------------------------------------------- + // Rules + // ---------------------------------------------------------------------- + + ReplayFilterRule ReplayFilterRule::UniqueId(std::string pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::UniqueId; + rule.pattern = std::move(pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + ReplayFilterRule ReplayFilterRule::StructName(std::string pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::StructName; + rule.pattern = std::move(pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + ReplayFilterRule ReplayFilterRule::Property(std::string struct_name, std::string field_name, + std::string value_pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::Property; + rule.struct_name = std::move(struct_name); + rule.field_name = std::move(field_name); + rule.pattern = std::move(value_pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + ReplayFilterRule ReplayFilterRule::Bucket(std::string pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::Bucket; + rule.pattern = std::move(pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + bool GlobMatch(std::string_view pattern, std::string_view text, bool case_insensitive) { + size_t p = 0; + size_t t = 0; + size_t star = std::string_view::npos; + size_t star_text = 0; + while (t < text.size()) { + if (p < pattern.size() && (pattern[p] == '?' || CharEquals(pattern[p], text[t], case_insensitive))) { + ++p; + ++t; + } else if (p < pattern.size() && pattern[p] == '*') { + star = p++; + star_text = t; + } else if (star != std::string_view::npos) { + p = star + 1; + t = ++star_text; + } else { + return false; + } + } + while (p < pattern.size() && pattern[p] == '*') { + ++p; + } + return p == pattern.size(); + } + + // ---------------------------------------------------------------------- + // Matcher + // ---------------------------------------------------------------------- + + ReplayFilterMatcher::ReplayFilterMatcher(const ReplayFilterSpec& spec, const PropertyAddressCache& cache) + : mode_(spec.mode) { + for (const auto& [type_id, struct_cache] : cache.structs) { + type_names_[type_id] = struct_cache.name; + } + bucket_names_ = cache.bucket_names; + if (spec.rules.empty()) { + error_ = "The filter has no rules; add at least one bucket, unique id, struct or property rule."; + return; + } + for (const ReplayFilterRule& rule : spec.rules) { + if (rule.pattern.empty()) { + error_ = std::string("A ") + KindName(rule.kind) + " rule has an empty pattern."; + return; + } + ResolvedRule resolved; + resolved.kind = rule.kind; + resolved.pattern = rule.pattern; + resolved.case_insensitive = rule.case_insensitive; + if (rule.kind == ReplayFilterRuleKind::Property) { + if (rule.field_name.empty()) { + error_ = "A property rule has no field name."; + return; + } + if (!rule.struct_name.empty()) { + const auto id_it = cache.name_to_id.find(rule.struct_name); + const auto struct_it = + id_it == cache.name_to_id.end() ? cache.structs.end() : cache.structs.find(id_it->second); + if (struct_it == cache.structs.end()) { + error_ = "Property rule: struct '" + rule.struct_name + "' is not in the schema."; + return; + } + const auto field_it = struct_it->second.properties.find(rule.field_name); + if (field_it == struct_it->second.properties.end()) { + error_ = + "Property rule: struct '" + rule.struct_name + "' has no field '" + rule.field_name + "'."; + return; + } + if (!IsMatchableField(field_it->second)) { + error_ = "Property rule: '" + rule.struct_name + "." + rule.field_name + + "' is not a scalar bool/int/float/double/string field."; + return; + } + resolved.slots[id_it->second] = field_it->second; + } else { + for (const auto& [type_id, struct_cache] : cache.structs) { + const auto field_it = struct_cache.properties.find(rule.field_name); + if (field_it != struct_cache.properties.end() && IsMatchableField(field_it->second)) { + resolved.slots[type_id] = field_it->second; + } + } + if (resolved.slots.empty()) { + error_ = "Property rule: no struct declares a scalar field named '" + rule.field_name + "'."; + return; + } + } + } else if (rule.kind == ReplayFilterRuleKind::Bucket) { + const bool numeric = std::all_of(rule.pattern.begin(), rule.pattern.end(), + [](unsigned char c) { return std::isdigit(c) != 0; }); + if (numeric) { + // A digits-only pattern addresses the bucket by index, named or not. + int32_t ordinal = -1; + const auto parsed = + std::from_chars(rule.pattern.data(), rule.pattern.data() + rule.pattern.size(), ordinal); + if (parsed.ec != std::errc {} || ordinal < 0) { + error_ = "Bucket rule: '" + rule.pattern + "' is not a valid bucket index."; + return; + } + resolved.bucket_ordinal = ordinal; + } else { + for (size_t b = 0; b < bucket_names_.size(); ++b) { + if (GlobMatch(rule.pattern, bucket_names_[b], rule.case_insensitive)) { + resolved.bucket_indices.push_back(static_cast(b)); + } + } + if (resolved.bucket_indices.empty()) { + std::string names; + for (const std::string& name : bucket_names_) { + names += (names.empty() ? "'" : ", '") + name + "'"; + } + error_ = "Bucket rule: no schema bucket matches '" + rule.pattern + "'" + + (names.empty() ? std::string(" (the schema names no buckets).") + : " (buckets: " + names + ")."); + return; + } + } + } + rules_.push_back(std::move(resolved)); + } + } + + std::string ReplayFilterMatcher::TypeName(int32_t type_id) const { + const auto it = type_names_.find(type_id); + if (it != type_names_.end()) { + return it->second; + } + return "type#" + std::to_string(type_id); + } + + std::string ReplayFilterMatcher::BucketName(int32_t bucket_index) const { + if (bucket_index >= 0 && static_cast(bucket_index) < bucket_names_.size() && + !bucket_names_[static_cast(bucket_index)].empty()) { + return bucket_names_[static_cast(bucket_index)]; + } + return "bucket#" + std::to_string(bucket_index); + } + + bool ReplayFilterMatcher::BucketMatches(int32_t bucket_index) const { + if (bucket_index < 0) { + return false; + } + for (const ResolvedRule& rule : rules_) { + if (rule.kind != ReplayFilterRuleKind::Bucket) { + continue; + } + if (bucket_index == rule.bucket_ordinal || std::find(rule.bucket_indices.begin(), rule.bucket_indices.end(), + bucket_index) != rule.bucket_indices.end()) { + return true; + } + } + return false; + } + + bool ReplayFilterMatcher::RuleMatches(const ResolvedRule& rule, int32_t bucket_index, const Bucket& bucket, + size_t index) const { + switch (rule.kind) { + case ReplayFilterRuleKind::Bucket: + return bucket_index >= 0 && (bucket_index == rule.bucket_ordinal || + std::find(rule.bucket_indices.begin(), rule.bucket_indices.end(), + bucket_index) != rule.bucket_indices.end()); + case ReplayFilterRuleKind::UniqueId: { + const std::string_view id = + index < bucket.unique_ids.size() ? std::string_view(bucket.unique_ids[index]) : std::string_view(); + return GlobMatch(rule.pattern, id, rule.case_insensitive); + } + case ReplayFilterRuleKind::StructName: { + const auto it = type_names_.find(bucket.entities[index].entity_type_id); + if (it == type_names_.end()) { + return false; + } + return GlobMatch(rule.pattern, it->second, rule.case_insensitive); + } + case ReplayFilterRuleKind::Property: { + const PropertyContainer& entity = bucket.entities[index]; + const auto slot = rule.slots.find(entity.entity_type_id); + if (slot == rule.slots.end()) { + return false; + } + std::string value; + if (!PropertyValueToString(entity, slot->second, value)) { + return false; + } + return GlobMatch(rule.pattern, value, rule.case_insensitive); + } + } + return false; + } + + bool ReplayFilterMatcher::Matches(int32_t bucket_index, const Bucket& bucket, size_t index) const { + if (index >= bucket.entities.size()) { + return false; + } + for (const ResolvedRule& rule : rules_) { + if (RuleMatches(rule, bucket_index, bucket, index)) { + return true; + } + } + return false; + } + + size_t ReplayFilterMatcher::Apply(int32_t bucket_index, Bucket& bucket, + const std::function& on_drop) const { + const size_t count = bucket.entities.size(); + size_t write = 0; + for (size_t read = 0; read < count; ++read) { + if (!Keeps(bucket_index, bucket, read)) { + if (on_drop) { + on_drop(bucket, read); + } + continue; + } + if (write != read) { + bucket.entities[write] = std::move(bucket.entities[read]); + if (read < bucket.unique_ids.size() && write < bucket.unique_ids.size()) { + bucket.unique_ids[write] = std::move(bucket.unique_ids[read]); + } + } + ++write; + } + if (write == count) { + return 0; + } + bucket.entities.resize(write); + if (bucket.unique_ids.size() > write) { + bucket.unique_ids.resize(write); + } + bucket.type_ranges.clear(); + return count - write; + } + + // ---------------------------------------------------------------------- + // Entry point + // ---------------------------------------------------------------------- + + ReplayFilterResult FilterReplayFile(const std::string& source_path, const std::string& dest_path, + const ReplayFilterSpec& spec, const ReplayFilterProgress& progress) { + namespace fs = std::filesystem; + const auto started = std::chrono::steady_clock::now(); + ReplayFilterResult result; + const auto finish = [&]() -> ReplayFilterResult& { + result.elapsed_seconds = std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + return result; + }; + const auto fail = [&](std::string message) -> ReplayFilterResult& { + result.error = std::move(message); + return finish(); + }; + + if (spec.rules.empty()) { + return fail("The filter has no rules; add at least one bucket, unique id, struct or property rule."); + } + if (source_path.empty() || dest_path.empty()) { + return fail("Source and destination paths are required."); + } + std::error_code ec; + if (!fs::is_regular_file(source_path, ec)) { + return fail("Source replay does not exist: " + source_path); + } + const fs::path source_abs = fs::weakly_canonical(fs::path(source_path), ec); + const fs::path dest_abs = fs::weakly_canonical(fs::path(dest_path), ec); + if (source_abs == dest_abs || (fs::exists(dest_path, ec) && fs::equivalent(source_path, dest_path, ec))) { + return fail("Destination must be a different file than the source."); + } + + ReaderContext context = OpenReplayFile(source_path); + if (!context.Loaded()) { + return fail("Could not open the source replay: " + context.GetError().message); + } + if (!context.WaitUntilReady()) { + return fail("The source replay failed to load: " + context.GetReadyError().message); + } + + const FileFooter footer = context->GetFooter(); + const PropertyAddressCache cache = context->GetPropertyAddressCache(); + const ReplayFilterMatcher matcher(spec, cache); + if (!matcher.valid()) { + return fail(matcher.error()); + } + result.source_bytes = fs::file_size(source_path, ec); + + switch (context.format) { + case VtxFormat::FlatBuffers: + RunFilter(source_path, dest_path, *context.reader, footer, matcher, spec.prune_events, + progress, result); + break; + case VtxFormat::Protobuf: + RunFilter(source_path, dest_path, *context.reader, footer, matcher, spec.prune_events, + progress, result); + break; + default: + return fail("Unknown replay format."); + } + return finish(); + } + +} // namespace VTX diff --git a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_recovery.cpp b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_recovery.cpp similarity index 79% rename from sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_recovery.cpp rename to sdk/src/vtx_transform/src/vtx/transform/vtx_replay_recovery.cpp index 9f4e98f..78caf1b 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_recovery.cpp +++ b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_recovery.cpp @@ -1,16 +1,14 @@ -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" #include #include #include -#include #include #include #include #include -#include -#include +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" #include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" @@ -31,63 +29,33 @@ namespace VTX { // crash-truncated (footerless) file almost never does. Used to detect a // complete file with only a leftover journal, so we DON'T rewrite its footer. bool HasValidTrailingFooter(const std::string& path, const std::string& magic, int64_t file_size) { - if (file_size < 8) + if (file_size < static_cast(Framing::kFooterTrailerSize)) return false; std::ifstream in(path, std::ios::binary); if (!in.is_open()) return false; - in.seekg(file_size - 8); - char buf[8]; - in.read(buf, 8); - if (in.gcount() != 8) - return false; + uint64_t footer_offset = 0; uint32_t footer_size = 0; - std::memcpy(&footer_size, buf, sizeof(footer_size)); // host-native, matches the sink - if (std::string(buf + 4, 4) != magic) - return false; - return footer_size > 0 && static_cast(footer_size) + 8 <= file_size; - } - - // Mirror of ChunkedFileSink::CompressIfBeneficial, so a footer synthesized by - // repair is byte-identical to one written by a clean Close() under the same - // (journaled) compression settings. - std::string CompressIfBeneficial(std::string payload, bool use_compression, int8_t level) { - if (!use_compression || payload.size() < 512) - return payload; - const size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - const size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), level); - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) - return payload; - compressed.resize(compressed_size); - return compressed; + std::string ignored; + return Framing::ProbeFooterTrailer(in, magic, static_cast(file_size), footer_offset, footer_size, + ignored); } - // Byte offset where chunks begin (magic(4) + u32 header_size + header), or -1 if - // the header framing is not intact. + // Byte offset where chunks begin (magic + u32 header_size + header), or -1 if the + // header framing is not intact. int64_t ReadHeaderEnd(std::ifstream& in, const std::string& expect_magic, int64_t file_size) { - in.clear(); - in.seekg(0); - char magic[4]; - in.read(magic, 4); - if (in.gcount() != 4 || std::string(magic, 4) != expect_magic) - return -1; - uint32_t header_size = 0; - in.read(reinterpret_cast(&header_size), sizeof(header_size)); - if (in.gcount() != static_cast(sizeof(header_size))) - return -1; - const int64_t header_end = 8 + static_cast(header_size); - if (header_size == 0 || header_end > file_size) + uint64_t header_end = 0; + std::string ignored; + if (!Framing::ProbeHeader(in, expect_magic, static_cast(file_size), header_end, ignored)) return -1; - return header_end; + return static_cast(header_end); } template RepairResult RepairImpl(const std::string& path, const RecoveryJournal::Parsed& journal) { RepairResult r; const int64_t file_size = FileSizeOf(path); - if (file_size < 8) { + if (file_size < static_cast(Framing::kMagicSize + Framing::kSizePrefixSize)) { r.error = "main file too small or missing"; return r; } @@ -110,7 +78,7 @@ namespace VTX { for (const auto& c : journal.chunks) { const int64_t off = c.file_offset; const int64_t end = off + static_cast(c.chunk_size_bytes); - if (off < header_end || c.chunk_size_bytes <= sizeof(uint32_t) || end > file_size) + if (off < header_end || c.chunk_size_bytes <= Framing::kSizePrefixSize || end > file_size) break; // torn tail / beyond EOF std::string bytes; @@ -121,9 +89,9 @@ namespace VTX { if (in.gcount() != static_cast(c.chunk_size_bytes)) break; - // The stored checksum covers the payload after the 4-byte length prefix. - const uint64_t hash = - XXH3_64bits(bytes.data() + sizeof(uint32_t), c.chunk_size_bytes - sizeof(uint32_t)); + // The stored checksum covers the payload after the size prefix. + const uint64_t hash = Framing::PayloadChecksum(bytes.data() + Framing::kSizePrefixSize, + c.chunk_size_bytes - Framing::kSizePrefixSize); if (c.checksum != 0 && hash != c.checksum) break; // corrupt chunk -> stop, drop it and everything after @@ -166,8 +134,8 @@ namespace VTX { if (f->index != expected || f->payload.empty()) break; // gap / duplicate / empty -> stop at the first hole const uint64_t offset = out.Tell(); - const uint32_t size = static_cast(f->payload.size()); - out.Write(&size, sizeof(size)); + const Framing::SizePrefix prefix = Framing::EncodeSizePrefix(static_cast(f->payload.size())); + out.Write(prefix.data(), prefix.size()); out.Write(f->payload.data(), f->payload.size()); ChunkIndexData e; @@ -175,8 +143,8 @@ namespace VTX { e.file_offset = static_cast(offset); e.start_frame = f->index; e.end_frame = f->index; - e.chunk_size_bytes = size + static_cast(sizeof(uint32_t)); - e.checksum = XXH3_64bits(f->payload.data(), f->payload.size()); + e.chunk_size_bytes = Framing::ChunkSizeOnDisk(f->payload.size()); + e.checksum = Framing::PayloadChecksum(f->payload); good.push_back(e); last_frame = f->index; @@ -239,14 +207,13 @@ namespace VTX { footer_data.segments = &segments; // Compress exactly as the sink's Close() would have (settings journaled in // the 'S' record), so large recovered footers also match byte-for-byte. - const std::string footer_payload = CompressIfBeneficial(Policy::SerializeFooter(good, footer_data), - journal.use_compression, journal.compression_level); + const std::string footer_payload = Framing::CompressIfBeneficial( + Policy::SerializeFooter(good, footer_data), journal.use_compression, journal.compression_level); bool footer_ok = out.Write(footer_payload.data(), footer_payload.size()); - const uint32_t footer_size = static_cast(footer_payload.size()); - footer_ok = out.Write(&footer_size, sizeof(footer_size)) && footer_ok; - const std::string magic = Policy::GetMagicBytes(); - footer_ok = out.Write(magic.data(), magic.size()) && footer_ok; + const std::string trailer = + Framing::FooterTrailer(static_cast(footer_payload.size()), Policy::GetMagicBytes()); + footer_ok = out.Write(trailer.data(), trailer.size()) && footer_ok; footer_ok = out.Sync() && footer_ok; // Good() also catches any failure in the pending-frame append writes above. const bool all_ok = footer_ok && out.Good(); @@ -296,10 +263,10 @@ namespace VTX { r.error = "cannot open main file: " + path; return r; } - char magic[4] = {0, 0, 0, 0}; - in.read(magic, 4); + char magic[Framing::kMagicSize] = {}; + in.read(magic, Framing::kMagicSize); in.close(); - const std::string m(magic, 4); + const std::string m(magic, Framing::kMagicSize); const int64_t file_size = FileSizeOf(path); diff --git a/sdk/src/vtx_writer/CMakeLists.txt b/sdk/src/vtx_writer/CMakeLists.txt index 41b535e..560ca2f 100644 --- a/sdk/src/vtx_writer/CMakeLists.txt +++ b/sdk/src/vtx_writer/CMakeLists.txt @@ -46,6 +46,19 @@ if(BUILD_SHARED_LIBS AND WIN32) target_link_libraries(vtx_writer PRIVATE $) endif() +# Linux + shared build: IXWebSocket and mbedTLS are linked into this .so from static +# archives. GNU ld would export their symbols with default visibility, so a consumer +# that links its own copy of the same archives (vtx_tests does, for its in-process +# WebSocket server) ends up with ONE interposed set of their globals -- constructed +# by both images and destroyed twice: `free(): double free detected` at exit. +# Hiding every archive keeps the .so self-contained with a private copy, which is +# what the Windows DLL already does (a DLL never exports the symbols of static +# libraries it links). Nothing here is public API: consumers get zstd/flatbuffers +# through vtx_common's link interface, never through vtx_writer. +if(BUILD_SHARED_LIBS AND UNIX AND NOT APPLE) + target_link_options(vtx_writer PRIVATE "LINKER:--exclude-libs,ALL") +endif() + # ============================================================================== # 3. VS filters # ============================================================================== diff --git a/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp b/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp index e0bab92..0f875d8 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp +++ b/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp @@ -118,7 +118,20 @@ std::string VTX::FlatBuffersVtxPolicy::SerializeFooter(const std::vector> {}); + std::vector> event_offsets; + if (footer_data.events) { + event_offsets.reserve(footer_data.events->size()); + for (const auto& ev : *footer_data.events) { + // Strings are always created (the reader dereferences them unconditionally). + const auto type_off = builder.CreateString(ev.event_type); + const auto label_off = builder.CreateString(ev.label); + const auto id_off = builder.CreateString(ev.entity_unique_id); + const fbsvtx::Vector location(ev.location.x, ev.location.y, ev.location.z); + event_offsets.push_back( + fbsvtx::CreateTimelineEvent(builder, ev.game_time, type_off, label_off, &location, id_off)); + } + } + auto events_vec = builder.CreateVector(event_offsets); auto footer_offset = fbsvtx::CreateFileFooter(builder, footer_data.total_frames, static_cast(footer_data.duration_seconds), diff --git a/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp b/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp index 7a038eb..f199ea7 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp +++ b/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp @@ -126,6 +126,21 @@ std::string VTX::ProtobufVtxPolicy::SerializeFooter(const std::vectorReserve(static_cast(footerData.events->size())); + for (const auto& ev : *footerData.events) { + auto* pb_ev = footer_msg.add_events(); + pb_ev->set_game_time(ev.game_time); + pb_ev->set_event_type(ev.event_type); + pb_ev->set_label(ev.label); + auto* location = pb_ev->mutable_location(); + location->set_x(ev.location.x); + location->set_y(ev.location.y); + location->set_z(ev.location.z); + pb_ev->set_entity_unique_id(ev.entity_unique_id); + } + } + footer_msg.mutable_chunk_index()->Reserve(static_cast(seekTable.size())); for (const auto& entry : seekTable) { diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 328ec0f..be39fd0 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -49,6 +49,7 @@ include(GoogleTest) add_executable(vtx_tests util/test_main.cpp util/test_fixtures.h + util/replay_snapshot.h common/test_flat_array.cpp common/test_flat_array_edges.cpp @@ -67,6 +68,7 @@ add_executable(vtx_tests common/test_native_loader.cpp common/test_binary_loader.cpp common/test_diagnostics.cpp + common/test_replay_framing.cpp writer/test_writer_basic.cpp writer/test_writer_edges.cpp @@ -76,7 +78,6 @@ add_executable(vtx_tests writer/test_schema_in_memory.cpp writer/test_frame_post_processor.cpp writer/test_frame_finalization.cpp - writer/test_crash_recovery.cpp writer/test_network_sink.cpp writer/test_pipe_source.cpp writer/test_websocket_source.cpp @@ -105,6 +106,18 @@ target_link_libraries(vtx_tests PRIVATE $ ) +# The tools that modify an existing replay (repair / cut / filter) live in +# vtx_transform, which only exists when reader + writer are built; their tests join +# the executable only then. +if(TARGET vtx_transform) + target_sources(vtx_tests PRIVATE + transform/test_crash_recovery.cpp + transform/test_replay_cut.cpp + transform/test_replay_filter.cpp + ) + target_link_libraries(vtx_tests PRIVATE vtx_transform) +endif() + target_include_directories(vtx_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}" "${VTX_COMMON_GENERATED_DIR}" @@ -134,6 +147,7 @@ source_group("common" REGULAR_EXPRESSION "common/.*") source_group("writer" REGULAR_EXPRESSION "writer/.*") source_group("reader" REGULAR_EXPRESSION "reader/.*") source_group("differ" REGULAR_EXPRESSION "differ/.*") +source_group("transform" REGULAR_EXPRESSION "transform/.*") # Register every TEST(suite, name) with ctest gtest_discover_tests(vtx_tests diff --git a/tests/common/test_replay_framing.cpp b/tests/common/test_replay_framing.cpp new file mode 100644 index 0000000..dc32295 --- /dev/null +++ b/tests/common/test_replay_framing.cpp @@ -0,0 +1,265 @@ +// vtx_replay_framing.h is the one definition of the .vtx on-disk framing. These +// tests pin the byte layout of every helper and prove that a file written by the +// real sink is described exactly by ProbeLayout / PayloadChecksum -- which is what +// lets the sinks, repair, cut and filter share it without drifting apart. + +#include + +#include +#include +#include +#include +#include +#include + +#include +#include // the round-trip check decompresses directly + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/writer/core/vtx_writer_facade.h" + +#include "util/replay_snapshot.h" +#include "util/test_fixtures.h" + +using namespace VTX::Framing; + +namespace { + + std::string Repeated(size_t n, char c = 'a') { + return std::string(n, c); + } + + std::string RandomBytes(size_t n, uint32_t seed = 7) { + std::mt19937 rng(seed); + std::string out(n, '\0'); + for (char& c : out) { + c = static_cast(rng() & 0xFF); + } + return out; + } + + // Host-order bytes of a u32, the way the sink has always written them. + std::string HostBytes(uint32_t v) { + std::string s(sizeof(v), '\0'); + std::memcpy(s.data(), &v, sizeof(v)); + return s; + } + +} // namespace + +TEST(ReplayFraming, SizePrefixRoundTripsAndMatchesHostOrder) { + for (uint32_t v : {0u, 1u, 512u, 0xDEADBEEFu, 0xFFFFFFFFu}) { + const SizePrefix p = EncodeSizePrefix(v); + EXPECT_EQ(std::string(p.data(), p.size()), HostBytes(v)) << v; + EXPECT_EQ(DecodeSizePrefix(p.data()), v); + } + EXPECT_EQ(ChunkSizeOnDisk(0), kSizePrefixSize); + EXPECT_EQ(ChunkSizeOnDisk(100), 104u); +} + +TEST(ReplayFraming, BlocksHaveTheDocumentedLayout) { + const std::string payload = "PAYLOAD"; + const std::string magic = "VTXF"; + + const std::string chunk = ChunkBlock(payload); + ASSERT_EQ(chunk.size(), kSizePrefixSize + payload.size()); + EXPECT_EQ(chunk.substr(0, 4), HostBytes(7)); + EXPECT_EQ(chunk.substr(4), payload); + + const std::string header = HeaderBlock(magic, payload); + ASSERT_EQ(header.size(), 4 + 4 + payload.size()); + EXPECT_EQ(header.substr(0, 4), magic); + EXPECT_EQ(header.substr(4, 4), HostBytes(7)); + EXPECT_EQ(header.substr(8), payload); + + const std::string trailer = FooterTrailer(7, magic); + ASSERT_EQ(trailer.size(), kFooterTrailerSize); + EXPECT_EQ(trailer.substr(0, 4), HostBytes(7)); + EXPECT_EQ(trailer.substr(4), magic); + + const std::string footer = FooterBlock(payload, magic); + EXPECT_EQ(footer, payload + trailer); +} + +TEST(ReplayFraming, CompressionRuleMatchesTheWriter) { + EXPECT_FALSE(ShouldCompress(kCompressMinBytes - 1, true)); + EXPECT_TRUE(ShouldCompress(kCompressMinBytes, true)); + EXPECT_FALSE(ShouldCompress(1 << 20, false)); + + // Compressible data shrinks and carries the zstd magic; the sniff agrees. + const std::string big = Repeated(64 * 1024); + const std::string compressed = Compress(big, kDefaultCompressionLevel); + EXPECT_LT(compressed.size(), big.size()); + EXPECT_TRUE(LooksLikeZstd(compressed)); + EXPECT_FALSE(LooksLikeZstd(big)); + + // Incompressible data is returned as-is (never grown). + const std::string noise = RandomBytes(4096); + EXPECT_EQ(Compress(noise, kDefaultCompressionLevel), noise); + + // The combined rule: threshold, enabled flag, then Compress. + EXPECT_EQ(CompressIfBeneficial(Repeated(100)), Repeated(100)); // below threshold + EXPECT_EQ(CompressIfBeneficial(big, /*enabled=*/false), big); + EXPECT_EQ(CompressIfBeneficial(big), compressed); + + // Decompresses back to the input. + std::string round(big.size(), '\0'); + const size_t n = ZSTD_decompress(round.data(), round.size(), compressed.data(), compressed.size()); + ASSERT_FALSE(ZSTD_isError(n)); + EXPECT_EQ(round, big); +} + +TEST(ReplayFraming, ChecksumIsXxHash64OfThePayload) { + const std::string a = RandomBytes(1000, 1); + const std::string b = RandomBytes(1000, 2); + EXPECT_EQ(PayloadChecksum(a), XXH3_64bits(a.data(), a.size())); + EXPECT_EQ(PayloadChecksum(a.data(), a.size()), PayloadChecksum(a)); + EXPECT_NE(PayloadChecksum(a), PayloadChecksum(b)); +} + +TEST(ReplayFraming, ProbeLayoutLocatesHeaderAndFooterOfASyntheticFile) { + const std::string magic = "VTXF"; + const std::string header = Repeated(40, 'h'); + const std::string chunk0 = Repeated(300, 'c'); + const std::string chunk1 = Repeated(20, 'd'); + const std::string footer = Repeated(64, 'f'); + + std::stringstream file; + file << HeaderBlock(magic, header) << ChunkBlock(chunk0) << ChunkBlock(chunk1) << FooterBlock(footer, magic); + + FileLayout layout; + std::string error; + ASSERT_TRUE(ProbeLayout(file, magic, layout, error)) << error; + EXPECT_EQ(layout.file_size, file.str().size()); + EXPECT_EQ(layout.header_end, 4 + 4 + header.size()); + EXPECT_EQ(layout.footer_size, footer.size()); + EXPECT_EQ(layout.footer_offset, layout.file_size - kFooterTrailerSize - footer.size()); + EXPECT_FALSE(layout.footer_compressed); + // The chunk region is exactly what sits between header and footer. + EXPECT_EQ(layout.footer_offset - layout.header_end, ChunkBlock(chunk0).size() + ChunkBlock(chunk1).size()); + EXPECT_TRUE(IsZstdAt(file, layout.header_end + kSizePrefixSize, chunk0.size()) == false); + + // A compressed footer is detected through the sniff. + const std::string zfooter = Compress(Repeated(4096, 'f'), kDefaultCompressionLevel); + std::stringstream zfile; + zfile << HeaderBlock(magic, header) << ChunkBlock(chunk0) << FooterBlock(zfooter, magic); + ASSERT_TRUE(ProbeLayout(zfile, magic, layout, error)) << error; + EXPECT_TRUE(layout.footer_compressed); + EXPECT_EQ(layout.footer_size, zfooter.size()); +} + +TEST(ReplayFraming, ProbeRejectsBrokenLayouts) { + const std::string magic = "VTXF"; + const std::string header = Repeated(40, 'h'); + FileLayout layout; + std::string error; + + // Wrong magic. + std::stringstream wrong; + wrong << HeaderBlock("VTXP", header) << FooterBlock("f", "VTXP"); + EXPECT_FALSE(ProbeLayout(wrong, magic, layout, error)); + EXPECT_NE(error.find("magic"), std::string::npos) << error; + + // Footerless (crash-truncated): the header probes fine, the trailer does not. + std::stringstream truncated; + truncated << HeaderBlock(magic, header) << ChunkBlock(Repeated(300)); + uint64_t header_end = 0; + ASSERT_TRUE(ProbeHeader(truncated, magic, StreamSize(truncated), header_end, error)) << error; + EXPECT_EQ(header_end, 4 + 4 + header.size()); + EXPECT_FALSE(ProbeLayout(truncated, magic, layout, error)); + EXPECT_NE(error.find("no footer"), std::string::npos) << error; + + // A footer size that reaches back into the header. + std::stringstream overlap; + overlap << HeaderBlock(magic, header) << Repeated(10, 'x') << FooterTrailer(1000, magic); + EXPECT_FALSE(ProbeLayout(overlap, magic, layout, error)); + EXPECT_FALSE(error.empty()); + + // Empty / too small. + std::stringstream empty; + EXPECT_FALSE(ProbeLayout(empty, magic, layout, error)); +} + +TEST(ReplayFraming, CopyRangeStreamsExactBytes) { + const std::string data = RandomBytes(10 * 1024 + 17); + std::stringstream in(data); + std::stringstream out; + std::string error; + ASSERT_TRUE(CopyRange(in, out, 100, 5000, error)) << error; + EXPECT_EQ(out.str(), data.substr(100, 5000)); + + std::stringstream none; + ASSERT_TRUE(CopyRange(in, none, 0, 0, error)) << error; + EXPECT_TRUE(none.str().empty()); + + std::stringstream past; + EXPECT_FALSE(CopyRange(in, past, data.size() - 10, 100, error)); + EXPECT_NE(error.find("Read failed"), std::string::npos) << error; +} + +// --------------------------------------------------------------------------- +// The real writer and the framing helpers must agree byte for byte: probe a +// freshly recorded file and re-derive every seek-table field from the bytes. +// --------------------------------------------------------------------------- +class ReplayFramingOnDisk : public ::testing::TestWithParam {}; + +TEST_P(ReplayFramingOnDisk, SinkOutputIsDescribedByTheSharedFraming) { + const bool flat = GetParam() == VTX::VtxFormat::FlatBuffers; + const std::string magic = flat ? "VTXF" : "VTXP"; + + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = VtxTest::OutputPath(std::string("framing_") + (flat ? "fb" : "pb") + ".vtx"); + cfg.schema_json_path = VtxTest::FixturePath("test_schema.json"); + cfg.replay_name = "FramingTest"; + cfg.replay_uuid = "uuid-framing"; + cfg.chunk_max_frames = 25; + cfg.use_compression = true; + auto writer = flat ? VTX::CreateFlatBuffersWriterFacade(cfg) : VTX::CreateProtobufWriterFacade(cfg); + ASSERT_TRUE(writer); + for (int i = 0; i < 60; ++i) { + VTX::Frame frame; + auto& bucket = frame.CreateBucket("entity"); + VTX::PropertyContainer pc; + pc.entity_type_id = 0; + pc.string_properties = {"p", "n"}; + pc.int32_properties = {1, i, 0}; + pc.float_properties = {float(i), 1.0f}; + bucket.unique_ids.push_back("p"); + bucket.entities.push_back(std::move(pc)); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / 60.0f; + writer->RecordFrame(frame, t); + } + writer->Stop(); + + const VtxTest::ReplaySnapshot s = VtxTest::ReadReplay(cfg.output_filepath); + ASSERT_TRUE(s.ok); + + std::ifstream in(cfg.output_filepath, std::ios::binary); + ASSERT_TRUE(in); + FileLayout layout; + std::string error; + ASSERT_TRUE(ProbeLayout(in, magic, layout, error)) << error; + + // Chunks begin right after the header block and end right before the footer. + ASSERT_FALSE(s.footer.chunk_index.empty()); + EXPECT_EQ(s.footer.chunk_index.front().file_offset, layout.header_end); + const auto& last = s.footer.chunk_index.back(); + EXPECT_EQ(last.file_offset + last.chunk_size_bytes, layout.footer_offset); + + // Every entry's size prefix and checksum re-derive from the bytes on disk. + VtxTest::ExpectSeekTableMatchesFile(cfg.output_filepath, s.footer); + + // Each chunk is either a zstd frame or raw, exactly as the sniff says. + for (const auto& e : s.footer.chunk_index) { + const bool z = IsZstdAt(in, e.file_offset + kSizePrefixSize, e.chunk_size_bytes - kSizePrefixSize); + EXPECT_TRUE(z || e.chunk_size_bytes - kSizePrefixSize < kCompressMinBytes) << "chunk " << e.chunk_index; + } +} + +INSTANTIATE_TEST_SUITE_P(Formats, ReplayFramingOnDisk, + ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), + [](const ::testing::TestParamInfo& info) { + return info.param == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; + }); diff --git a/tests/writer/test_crash_recovery.cpp b/tests/transform/test_crash_recovery.cpp similarity index 99% rename from tests/writer/test_crash_recovery.cpp rename to tests/transform/test_crash_recovery.cpp index 1b57934..af873e5 100644 --- a/tests/writer/test_crash_recovery.cpp +++ b/tests/transform/test_crash_recovery.cpp @@ -39,8 +39,8 @@ #include "vtx/common/vtx_types.h" #include "vtx/reader/core/vtx_reader_facade.h" #include "vtx/reader/core/vtx_replay_validation.h" +#include "vtx/transform/vtx_replay_recovery.h" #include "vtx/writer/core/vtx_frame_post_processor.h" -#include "vtx/writer/core/vtx_replay_recovery.h" #include "vtx/writer/core/vtx_writer_facade.h" #include "vtx/writer/core/writer.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" diff --git a/tests/transform/test_replay_cut.cpp b/tests/transform/test_replay_cut.cpp new file mode 100644 index 0000000..fb5399e --- /dev/null +++ b/tests/transform/test_replay_cut.cpp @@ -0,0 +1,335 @@ +// Replay cut tests: CutReplayFile() writes a copy of a replay holding a sub-range +// of its frames. Whole chunks are copied verbatim (same checksum), edge chunks are +// re-serialized with only the kept frames, frames are renumbered from 0, the time +// table is sliced with absolute tick values, and the seek table describes the bytes +// on disk. Runs as a TEST_P over both backends. + +#include + +#include +#include +#include +#include +#include + +#include "vtx/common/vtx_types.h" +#include "vtx/transform/vtx_replay_cut.h" +#include "vtx/writer/core/vtx_writer_facade.h" + +#include "util/replay_snapshot.h" +#include "util/test_fixtures.h" + +namespace { + + constexpr int32_t kFrames = 100; // 3 chunks: [0,39] [40,79] [80,99] + constexpr int32_t kChunkFrames = 40; + constexpr float kFps = 60.0f; + constexpr int64_t kBaseUtc = 17'000'000'000'000'000LL; // unix-relative 100 ns ticks + constexpr int64_t kFrameTicks = 166'666; + + const char* FormatName(VTX::VtxFormat f) { + return f == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; + } + + // Fixture schema (tests/fixtures/test_schema.json), Player (type 0): strings + // [UniqueID, Name], int32 [Team, Score, Deaths], float [Health, Armor], ... + VTX::Frame BuildFrame(int32_t frame) { + VTX::Frame f; + auto& bucket = f.CreateBucket("entity"); + for (int32_t p = 0; p < 2; ++p) { + VTX::PropertyContainer pc; + pc.entity_type_id = 0; + const std::string id = "player_" + std::to_string(p); + pc.string_properties = {id, p == 0 ? "Alpha" : "Bravo"}; + pc.int32_properties = {p + 1, frame, frame / 10}; // Score carries the frame index + pc.float_properties = {100.0f - float(frame % 50), 25.0f + float(p)}; + pc.vector_properties = {VTX::Vector {double(frame), double(p), 0.0}, VTX::Vector {1.0, 0.0, 0.0}}; + pc.quat_properties = {VTX::Quat {0.0f, 0.0f, 0.0f, 1.0f}}; + pc.bool_properties = {frame % 7 != 0}; + bucket.unique_ids.push_back(id); + bucket.entities.push_back(std::move(pc)); + } + return f; + } + + // Output frame k must equal source frame (first + k), entity by entity. + void ExpectFramesAreSourceRange(const VtxTest::ReplaySnapshot& source, const VtxTest::ReplaySnapshot& output, + int32_t first, int32_t last) { + ASSERT_EQ(output.frames.size(), static_cast(last - first + 1)); + for (size_t k = 0; k < output.frames.size(); ++k) { + const VTX::Frame& s = source.frames[static_cast(first) + k]; + const VTX::Frame& o = output.frames[k]; + ASSERT_EQ(o.GetBuckets().size(), s.GetBuckets().size()) << "frame " << k; + for (size_t b = 0; b < s.GetBuckets().size(); ++b) { + const VTX::Bucket& sb = s.GetBuckets()[b]; + const VTX::Bucket& ob = o.GetBuckets()[b]; + ASSERT_EQ(ob.unique_ids, sb.unique_ids) << "frame " << k; + ASSERT_EQ(ob.entities.size(), sb.entities.size()) << "frame " << k; + for (size_t i = 0; i < sb.entities.size(); ++i) { + VtxTest::ExpectEntityEqual(sb.entities[i], ob.entities[i], + "frame " + std::to_string(k) + " entity " + sb.unique_ids[i]); + } + } + } + } + + // The footer's time table is the source table sliced to [first, last], values untouched. + void ExpectTimeTableSliced(const VtxTest::ReplaySnapshot& source, const VtxTest::ReplaySnapshot& output, + int32_t first, int32_t last) { + const size_t n = static_cast(last - first + 1); + ASSERT_EQ(output.footer.times.created_utc.size(), n); + ASSERT_EQ(output.footer.times.game_time.size(), n); + for (size_t k = 0; k < n; ++k) { + EXPECT_EQ(output.footer.times.created_utc[k], + source.footer.times.created_utc[static_cast(first) + k]) + << k; + EXPECT_EQ(output.footer.times.game_time[k], source.footer.times.game_time[static_cast(first) + k]) + << k; + } + const double expected_duration = double(source.footer.times.created_utc[static_cast(last)] - + source.footer.times.created_utc[static_cast(first)]) / + double(VTX::GameTime::TICKS_PER_SECOND); + EXPECT_NEAR(output.footer.duration_seconds, expected_duration, 1e-3); + EXPECT_EQ(output.footer.total_frames, last - first + 1); + } + +} // namespace + +class ReplayCutTest : public ::testing::TestWithParam { +protected: + std::string Path(const std::string& suffix) const { + return VtxTest::OutputPath(std::string("replay_cut_") + FormatName(GetParam()) + "_" + suffix + ".vtx"); + } + + std::string WriteSource(const std::string& suffix) const { + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = Path(suffix + "_src"); + cfg.schema_json_path = VtxTest::FixturePath("test_schema.json"); + cfg.replay_name = "ReplayCutTest"; + cfg.replay_uuid = "uuid-cut-" + suffix; + cfg.default_fps = kFps; + cfg.chunk_max_frames = kChunkFrames; + cfg.use_compression = true; + + auto writer = GetParam() == VTX::VtxFormat::FlatBuffers ? VTX::CreateFlatBuffersWriterFacade(cfg) + : VTX::CreateProtobufWriterFacade(cfg); + EXPECT_TRUE(writer); + if (!writer) { + return {}; + } + for (int32_t i = 0; i < kFrames; ++i) { + auto frame = BuildFrame(i); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / kFps; + t.created_utc_time = kBaseUtc + int64_t(i) * kFrameTicks; + writer->RecordFrame(frame, t); + } + writer->Stop(); + return cfg.output_filepath; + } +}; + +// --------------------------------------------------------------------------- +// A whole-chunk cut copies the chunk verbatim: same bytes (same checksum), frames +// renumbered from 0, time table sliced, header identical. +// --------------------------------------------------------------------------- +TEST_P(ReplayCutTest, WholeChunkCutCopiesVerbatim) { + const std::string src = WriteSource("wholechunk"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + ASSERT_EQ(source.footer.chunk_index.size(), 3u); + + const VTX::ReplayCutPlan plan = VTX::PlanCutChunks(source.footer, 1, 1); + ASSERT_TRUE(plan.valid) << plan.error; + EXPECT_EQ(plan.first_frame, 40); + EXPECT_EQ(plan.last_frame, 79); + EXPECT_FALSE(plan.trims_head); + EXPECT_FALSE(plan.trims_tail); + EXPECT_EQ(plan.FrameCount(), 40); + EXPECT_EQ(plan.chunk_bytes, source.footer.chunk_index[1].chunk_size_bytes); + + const std::string dst = Path("wholechunk_out"); + const VTX::ReplayCutResult r = VTX::CutReplayFile(src, source.footer, plan, dst); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, 40); + EXPECT_EQ(r.chunks_written, 1); + EXPECT_EQ(r.chunks_rewritten, 0); + EXPECT_GT(r.output_bytes, 0u); + + const VtxTest::ReplaySnapshot output = VtxTest::ReadReplay(dst); + ASSERT_TRUE(output.ok); + EXPECT_EQ(output.format, source.format); + EXPECT_EQ(output.header.replay_uuid, source.header.replay_uuid); + EXPECT_EQ(output.header.recorded_utc_timestamp, source.header.recorded_utc_timestamp); + EXPECT_EQ(output.schema.property_mapping, source.schema.property_mapping); + + ASSERT_EQ(output.footer.chunk_index.size(), 1u); + EXPECT_EQ(output.footer.chunk_index[0].chunk_index, 0); + EXPECT_EQ(output.footer.chunk_index[0].start_frame, 0); + EXPECT_EQ(output.footer.chunk_index[0].end_frame, 39); + EXPECT_EQ(output.footer.chunk_index[0].checksum, source.footer.chunk_index[1].checksum); // verbatim copy + EXPECT_EQ(output.footer.chunk_index[0].chunk_size_bytes, source.footer.chunk_index[1].chunk_size_bytes); + + ExpectFramesAreSourceRange(source, output, 40, 79); + ExpectTimeTableSliced(source, output, 40, 79); + VtxTest::ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// An exact-frame cut rewrites both edge chunks and keeps the middle one verbatim. +// --------------------------------------------------------------------------- +TEST_P(ReplayCutTest, ExactFrameCutRewritesEdges) { + const std::string src = WriteSource("exact"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + + const VTX::ReplayCutPlan plan = VTX::PlanCutFrames(source.footer, 30, 85); + ASSERT_TRUE(plan.valid) << plan.error; + EXPECT_EQ(plan.first_chunk, 0); + EXPECT_EQ(plan.last_chunk, 2); + EXPECT_EQ(plan.first_frame, 30); + EXPECT_EQ(plan.last_frame, 85); + EXPECT_TRUE(plan.trims_head); + EXPECT_TRUE(plan.trims_tail); + EXPECT_EQ(plan.FrameCount(), 56); + + const std::string dst = Path("exact_out"); + const VTX::ReplayCutResult r = VTX::CutReplayFile(src, source.footer, plan, dst); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, 56); + EXPECT_EQ(r.chunks_written, 3); + EXPECT_EQ(r.chunks_rewritten, 2); + + const VtxTest::ReplaySnapshot output = VtxTest::ReadReplay(dst); + ASSERT_TRUE(output.ok); + ASSERT_EQ(output.footer.chunk_index.size(), 3u); + EXPECT_EQ(output.footer.chunk_index[0].start_frame, 0); + EXPECT_EQ(output.footer.chunk_index[0].end_frame, 9); // source 30..39 + EXPECT_EQ(output.footer.chunk_index[1].start_frame, 10); + EXPECT_EQ(output.footer.chunk_index[1].end_frame, 49); // source 40..79, verbatim + EXPECT_EQ(output.footer.chunk_index[1].checksum, source.footer.chunk_index[1].checksum); + EXPECT_EQ(output.footer.chunk_index[2].start_frame, 50); + EXPECT_EQ(output.footer.chunk_index[2].end_frame, 55); // source 80..85 + EXPECT_NE(output.footer.chunk_index[0].checksum, source.footer.chunk_index[0].checksum); // rewritten + EXPECT_NE(output.footer.chunk_index[2].checksum, source.footer.chunk_index[2].checksum); + + ExpectFramesAreSourceRange(source, output, 30, 85); + ExpectTimeTableSliced(source, output, 30, 85); + VtxTest::ExpectSeekTableMatchesFile(dst, output.footer); +} + +// Bounds are clamped to the replay and a head cut alone rewrites one chunk. +TEST_P(ReplayCutTest, ClampsBoundsAndTrimsOnlyOneEdge) { + const std::string src = WriteSource("clamp"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + + const VTX::ReplayCutPlan plan = VTX::PlanCutFrames(source.footer, 95, 100'000); + ASSERT_TRUE(plan.valid) << plan.error; + EXPECT_EQ(plan.first_frame, 95); + EXPECT_EQ(plan.last_frame, kFrames - 1); + EXPECT_TRUE(plan.trims_head); + EXPECT_FALSE(plan.trims_tail); + + const std::string dst = Path("clamp_out"); + const VTX::ReplayCutResult r = VTX::CutReplayFile(src, source.footer, plan, dst); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, 5); + EXPECT_EQ(r.chunks_written, 1); + EXPECT_EQ(r.chunks_rewritten, 1); + + const VtxTest::ReplaySnapshot output = VtxTest::ReadReplay(dst); + ASSERT_TRUE(output.ok); + ExpectFramesAreSourceRange(source, output, 95, kFrames - 1); + VtxTest::ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// Failure paths report an error and never leave a destination behind. +// --------------------------------------------------------------------------- +TEST_P(ReplayCutTest, ErrorsAreReported) { + namespace fs = std::filesystem; + const std::string src = WriteSource("errors"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + const std::string dst = Path("errors_out"); + + { + VTX::ReplayCutPlan invalid; // valid == false + const auto r = VTX::CutReplayFile(src, source.footer, invalid, dst); + EXPECT_FALSE(r.ok()); + EXPECT_FALSE(fs::exists(dst)); + } + { + const VTX::ReplayCutPlan plan = VTX::PlanCutChunks(source.footer, 0, 0); + ASSERT_TRUE(plan.valid); + const auto r = VTX::CutReplayFile(src, source.footer, plan, src); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("different file"), std::string::npos) << r.error; + EXPECT_TRUE(fs::exists(src)); + } + { + const VTX::ReplayCutPlan plan = VTX::PlanCutChunks(source.footer, 0, 0); + const auto r = VTX::CutReplayFile(Path("does_not_exist"), source.footer, plan, dst); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("does not exist"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } + { + // A plan built against a different footer must not index past this one. + VTX::ReplayCutPlan stale = VTX::PlanCutChunks(source.footer, 0, 0); + stale.last_chunk = 42; + const auto r = VTX::CutReplayFile(src, source.footer, stale, dst); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("chunk index"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } +} + +INSTANTIATE_TEST_SUITE_P(Formats, ReplayCutTest, + ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), + [](const ::testing::TestParamInfo& info) { return FormatName(info.param); }); + +// =========================================================================== +// Plan resolution on a synthetic footer, no file involved. +// =========================================================================== +TEST(ReplayCutPlan, ResolvesRangesAgainstTheChunkIndex) { + VTX::FileFooter footer; + footer.total_frames = 100; + for (int32_t i = 0; i < 3; ++i) { + VTX::ChunkIndexEntry e; + e.chunk_index = i; + e.start_frame = i * 40; + e.end_frame = std::min(i * 40 + 39, 99); + e.chunk_size_bytes = 1000 + static_cast(i); + footer.chunk_index.push_back(e); + } + + const auto whole = VTX::PlanCutChunks(footer, 0, 2); + ASSERT_TRUE(whole.valid); + EXPECT_EQ(whole.first_frame, 0); + EXPECT_EQ(whole.last_frame, 99); + EXPECT_FALSE(whole.trims_head || whole.trims_tail); + EXPECT_EQ(whole.chunk_bytes, 3003u); + + const auto exact = VTX::PlanCutFrames(footer, 45, 45); + ASSERT_TRUE(exact.valid); + EXPECT_EQ(exact.first_chunk, 1); + EXPECT_EQ(exact.last_chunk, 1); + EXPECT_TRUE(exact.trims_head); + EXPECT_TRUE(exact.trims_tail); + EXPECT_EQ(exact.FrameCount(), 1); + + const auto boundary = VTX::PlanCutFrames(footer, 40, 79); + ASSERT_TRUE(boundary.valid); + EXPECT_FALSE(boundary.trims_head || boundary.trims_tail); // exactly chunk 1 + + EXPECT_FALSE(VTX::PlanCutFrames(footer, 50, 10).valid); + EXPECT_FALSE(VTX::PlanCutChunks(footer, 2, 1).valid); + EXPECT_FALSE(VTX::PlanCutFrames(VTX::FileFooter {}, 0, 0).valid); + EXPECT_EQ(VTX::ReplayCutPlan {}.FrameCount(), 0); +} diff --git a/tests/transform/test_replay_filter.cpp b/tests/transform/test_replay_filter.cpp new file mode 100644 index 0000000..97e1a50 --- /dev/null +++ b/tests/transform/test_replay_filter.cpp @@ -0,0 +1,907 @@ +// Replay entity filter tests: FilterReplayFile() writes a copy of a replay with +// entities dropped (blacklist) or exclusively kept (whitelist) by unique id glob, +// struct name or property value. Every frame survives; the header is byte-identical; +// the footer keeps the frame count, duration and per-frame time table; the seek +// table is rebuilt and its checksums match the bytes on disk; timeline events are +// carried over and pruned to the entities that still exist. +// +// Runs as a TEST_P over both backends (FlatBuffers + Protobuf). + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers +#include "vtx_schema.pb.h" + +#include "vtx/common/readers/schema_reader/schema_registry.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" +#include "vtx/transform/vtx_replay_filter.h" +#include "vtx/writer/core/vtx_writer_facade.h" +#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" +#include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" + +#include "util/replay_snapshot.h" +#include "util/test_fixtures.h" + +namespace { + + // Read-back helpers shared with the cut / framing tests. + using Snapshot = VtxTest::ReplaySnapshot; + using VtxTest::ExpectEntityEqual; + using VtxTest::ExpectSeekTableMatchesFile; + using VtxTest::ReadReplay; + + constexpr int kFrames = 100; // 3 chunks: [0,39] [40,79] [80,99] + constexpr int kChunkFrames = 40; + constexpr float kFps = 60.0f; + constexpr int64_t kBaseUtc = 17'000'000'000'000'000LL; // unix-relative 100 ns ticks + constexpr int64_t kFrameTicks = 166'666; + + std::string ReadFileText(const std::string& path) { + std::ifstream in(path, std::ios::binary); + return std::string((std::istreambuf_iterator(in)), std::istreambuf_iterator()); + } + + // The fixture schema with three buckets instead of one; the struct definitions + // are unchanged so the entity builders below stay valid. + std::string MultiBucketSchemaJson() { + std::string json = ReadFileText(VtxTest::FixturePath("test_schema.json")); + const std::string single = "\"buckets\": [\"entity\"]"; + const size_t at = json.find(single); + if (at != std::string::npos) { + json.replace(at, single.size(), "\"buckets\": [\"entity\", \"fx\", \"telemetry\"]"); + } + return json; + } + + const char* FormatName(VTX::VtxFormat f) { + return f == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; + } + + // Fixture schema (tests/fixtures/test_schema.json), one bucket "entity": + // Player (0): strings [UniqueID, Name] int32 [Team, Score, Deaths] float [Health, Armor] + // vector [Position, Velocity] quat [Rotation] bool [IsAlive] + // Projectile (1): strings [UniqueID, OwnerID, Type] vector [Position, Velocity] float [Damage] + // MatchState (2): strings [UniqueID, Phase] int32 [ScoreTeam1, ScoreTeam2, Round] float [TimeRemaining] + constexpr int32_t kPlayer = 0; + constexpr int32_t kProjectile = 1; + constexpr int32_t kMatchState = 2; + + VTX::PropertyContainer MakePlayer(const std::string& id, const std::string& name, int team, int frame) { + VTX::PropertyContainer pc; + pc.entity_type_id = kPlayer; + pc.string_properties = {id, name}; + pc.int32_properties = {team, frame, frame / 10}; + pc.float_properties = {100.0f - float(frame % 50), 25.0f + float(team)}; + pc.vector_properties = {VTX::Vector {double(frame), double(team), 0.0}, VTX::Vector {1.0, 0.0, 0.0}}; + pc.quat_properties = {VTX::Quat {0.0f, 0.0f, 0.0f, 1.0f}}; + pc.bool_properties = {frame % 7 != 0}; + return pc; + } + + VTX::PropertyContainer MakeProjectile(const std::string& id, const std::string& owner, const std::string& type, + int frame) { + VTX::PropertyContainer pc; + pc.entity_type_id = kProjectile; + pc.string_properties = {id, owner, type}; + pc.vector_properties = {VTX::Vector {double(frame) * 2.0, 1.0, 2.0}, VTX::Vector {0.0, 5.0, 0.0}}; + pc.float_properties = {10.0f + float(frame % 3)}; + return pc; + } + + VTX::PropertyContainer MakeMatch(int frame) { + VTX::PropertyContainer pc; + pc.entity_type_id = kMatchState; + pc.string_properties = {"match", frame < 50 ? "warmup" : "live"}; + pc.int32_properties = {frame / 20, frame / 30, 1 + frame / 50}; + pc.float_properties = {300.0f - float(frame)}; + return pc; + } + + void Add(VTX::Bucket& bucket, const std::string& id, VTX::PropertyContainer pc) { + bucket.unique_ids.push_back(id); + bucket.entities.push_back(std::move(pc)); + } + + std::string ProjectileId(int frame) { + return "proj_" + std::to_string(frame % 3); + } + + // Two players, one projectile (id cycles proj_0..proj_2) and the match state. + VTX::Frame BuildFrame(int frame) { + VTX::Frame f; + auto& bucket = f.CreateBucket("entity"); + Add(bucket, "player_0", MakePlayer("player_0", "Alpha", 1, frame)); + Add(bucket, "player_1", MakePlayer("player_1", "Bravo", 2, frame)); + Add(bucket, ProjectileId(frame), + MakeProjectile(ProjectileId(frame), frame % 2 ? "player_1" : "player_0", frame % 2 ? "rocket" : "bullet", + frame)); + Add(bucket, "match", MakeMatch(frame)); + return f; + } + + // Same entities spread over three buckets: players in "entity" (0), the + // projectile in "fx" (1), the match state in "telemetry" (2). + VTX::Frame BuildMultiBucketFrame(int frame) { + VTX::Frame f; + auto& entity = f.CreateBucket("entity"); + Add(entity, "player_0", MakePlayer("player_0", "Alpha", 1, frame)); + Add(entity, "player_1", MakePlayer("player_1", "Bravo", 2, frame)); + auto& fx = f.CreateBucket("fx"); + Add(fx, ProjectileId(frame), MakeProjectile(ProjectileId(frame), "player_0", "bullet", frame)); + auto& telemetry = f.CreateBucket("telemetry"); + Add(telemetry, "match", MakeMatch(frame)); + return f; + } + + // The output bucket must be exactly the source bucket minus the entities `dropped` + // rejects, in the original order, with every survivor unchanged. + template + void ExpectBucketFiltered(const VTX::Bucket& source, const VTX::Bucket& output, DropPredicate dropped, + const std::string& what) { + std::vector expected; + for (size_t i = 0; i < source.entities.size(); ++i) { + if (!dropped(source, i)) { + expected.push_back(i); + } + } + ASSERT_EQ(output.entities.size(), expected.size()) << what; + ASSERT_EQ(output.unique_ids.size(), expected.size()) << what; + for (size_t o = 0; o < expected.size(); ++o) { + const size_t s = expected[o]; + EXPECT_EQ(output.unique_ids[o], source.unique_ids[s]) << what << " entity " << o; + ExpectEntityEqual(source.entities[s], output.entities[o], what + " entity " + source.unique_ids[s]); + } + } + + template + void ExpectFramesFiltered(const Snapshot& source, const Snapshot& output, DropPredicate dropped) { + ASSERT_EQ(output.frames.size(), source.frames.size()); + for (size_t f = 0; f < source.frames.size(); ++f) { + const auto& sb = source.frames[f].GetBuckets(); + const auto& ob = output.frames[f].GetBuckets(); + ASSERT_EQ(ob.size(), sb.size()) << "frame " << f; + for (size_t b = 0; b < sb.size(); ++b) { + ExpectBucketFiltered(sb[b], ob[b], dropped, + "frame " + std::to_string(f) + " bucket " + std::to_string(b)); + } + } + } + + // Like ExpectFramesFiltered, with the predicate also given the bucket index. + template + void ExpectFramesFilteredByBucket(const Snapshot& source, const Snapshot& output, DropPredicate dropped) { + ASSERT_EQ(output.frames.size(), source.frames.size()); + for (size_t f = 0; f < source.frames.size(); ++f) { + const auto& sb = source.frames[f].GetBuckets(); + const auto& ob = output.frames[f].GetBuckets(); + ASSERT_EQ(ob.size(), sb.size()) << "frame " << f; + for (size_t b = 0; b < sb.size(); ++b) { + ExpectBucketFiltered( + sb[b], ob[b], [&](const VTX::Bucket& bucket, size_t i) { return dropped(b, bucket, i); }, + "frame " + std::to_string(f) + " bucket " + std::to_string(b)); + } + } + } + + // Header bit-identical fields, same frame count/duration/time table, same chunk + // frame ranges. + void ExpectSameHeaderFooterFraming(const Snapshot& source, const Snapshot& output) { + EXPECT_EQ(output.format, source.format); + EXPECT_EQ(output.header.replay_uuid, source.header.replay_uuid); + EXPECT_EQ(output.header.replay_name, source.header.replay_name); + EXPECT_EQ(output.header.recorded_utc_timestamp, source.header.recorded_utc_timestamp); + EXPECT_EQ(output.header.custom_json_metadata, source.header.custom_json_metadata); + EXPECT_EQ(output.header.version.format_major, source.header.version.format_major); + EXPECT_EQ(output.header.version.format_minor, source.header.version.format_minor); + EXPECT_EQ(output.header.version.schema_version, source.header.version.schema_version); + EXPECT_EQ(output.schema.property_mapping, source.schema.property_mapping); + EXPECT_EQ(output.schema.data_identifier, source.schema.data_identifier); + + EXPECT_EQ(output.footer.total_frames, source.footer.total_frames); + EXPECT_FLOAT_EQ(output.footer.duration_seconds, source.footer.duration_seconds); + EXPECT_EQ(output.footer.times.game_time, source.footer.times.game_time); + EXPECT_EQ(output.footer.times.created_utc, source.footer.times.created_utc); + EXPECT_EQ(output.footer.times.gaps, source.footer.times.gaps); + EXPECT_EQ(output.footer.times.segments, source.footer.times.segments); + EXPECT_FALSE(source.footer.times.created_utc.empty()) << "fixture should stamp created_utc"; + + ASSERT_EQ(output.footer.chunk_index.size(), source.footer.chunk_index.size()); + for (size_t i = 0; i < source.footer.chunk_index.size(); ++i) { + EXPECT_EQ(output.footer.chunk_index[i].chunk_index, source.footer.chunk_index[i].chunk_index) << i; + EXPECT_EQ(output.footer.chunk_index[i].start_frame, source.footer.chunk_index[i].start_frame) << i; + EXPECT_EQ(output.footer.chunk_index[i].end_frame, source.footer.chunk_index[i].end_frame) << i; + } + } + + VTX::TimelineEvent MakeEvent(float game_time, const std::string& type, const std::string& label, + const std::string& entity_id) { + VTX::TimelineEvent ev; + ev.game_time = game_time; + ev.event_type = type; + ev.label = label; + ev.location = VTX::Vector {1.0, 2.0, 3.0}; + ev.entity_unique_id = entity_id; + return ev; + } + + // The public writer never emits timeline events, so fabricate a source that has + // them: replace the file's footer block with one serialized through the same + // policy the sink uses, carrying the original seek table and time table plus the + // given events. + void RewriteFooterWithEvents(const std::string& path, VTX::VtxFormat format, + const std::vector& events) { + const Snapshot s = ReadReplay(path); + ASSERT_TRUE(s.ok); + + std::vector seek_table; + for (const VTX::ChunkIndexEntry& e : s.footer.chunk_index) { + VTX::ChunkIndexData d; + d.chunk_index = e.chunk_index; + d.file_offset = static_cast(e.file_offset); + d.chunk_size_bytes = e.chunk_size_bytes; + d.start_frame = e.start_frame; + d.end_frame = e.end_frame; + d.checksum = e.checksum; + seek_table.push_back(d); + } + const std::vector game_times(s.footer.times.game_time.begin(), s.footer.times.game_time.end()); + const std::vector created_utc(s.footer.times.created_utc.begin(), s.footer.times.created_utc.end()); + const std::vector gaps(s.footer.times.gaps.begin(), s.footer.times.gaps.end()); + const std::vector segments(s.footer.times.segments.begin(), s.footer.times.segments.end()); + + VTX::SessionFooter sf; + sf.total_frames = s.footer.total_frames; + sf.duration_seconds = s.footer.duration_seconds; + sf.game_times = &game_times; + sf.created_utc = &created_utc; + sf.gaps = &gaps; + sf.segments = &segments; + sf.events = &events; + + const bool flat = format == VTX::VtxFormat::FlatBuffers; + const std::string payload = flat ? VTX::FlatBuffersVtxPolicy::SerializeFooter(seek_table, sf) + : VTX::ProtobufVtxPolicy::SerializeFooter(seek_table, sf); + const std::string magic = + flat ? VTX::FlatBuffersVtxPolicy::GetMagicBytes() : VTX::ProtobufVtxPolicy::GetMagicBytes(); + + // Locate the existing footer block: [payload][u32 size][magic] at the tail. + uint64_t file_size = 0; + uint32_t old_footer_size = 0; + { + std::ifstream in(path, std::ios::binary); + ASSERT_TRUE(in); + in.seekg(0, std::ios::end); + file_size = static_cast(in.tellg()); + in.seekg(static_cast(file_size - magic.size() - sizeof(uint32_t))); + ASSERT_TRUE(in.read(reinterpret_cast(&old_footer_size), sizeof(old_footer_size))); + } + const uint64_t footer_offset = file_size - magic.size() - sizeof(uint32_t) - old_footer_size; + std::filesystem::resize_file(path, footer_offset); + + std::ofstream out(path, std::ios::binary | std::ios::app); + ASSERT_TRUE(out); + const uint32_t footer_size = static_cast(payload.size()); + out.write(payload.data(), static_cast(payload.size())); + out.write(reinterpret_cast(&footer_size), sizeof(footer_size)); + out.write(magic.data(), static_cast(magic.size())); + ASSERT_TRUE(out.good()); + } + +} // namespace + +class ReplayFilterTest : public ::testing::TestWithParam { +protected: + std::string Path(const std::string& suffix) const { + return VtxTest::OutputPath(std::string("replay_filter_") + FormatName(GetParam()) + "_" + suffix + ".vtx"); + } + + std::string WriteSource(const std::string& suffix) const { + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = Path(suffix + "_src"); + cfg.schema_json_path = VtxTest::FixturePath("test_schema.json"); + cfg.replay_name = "ReplayFilterTest"; + cfg.replay_uuid = "uuid-filter-" + suffix; + cfg.default_fps = kFps; + cfg.chunk_max_frames = kChunkFrames; + cfg.use_compression = true; + + auto writer = GetParam() == VTX::VtxFormat::FlatBuffers ? VTX::CreateFlatBuffersWriterFacade(cfg) + : VTX::CreateProtobufWriterFacade(cfg); + EXPECT_TRUE(writer); + if (!writer) { + return {}; + } + for (int i = 0; i < kFrames; ++i) { + auto frame = BuildFrame(i); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / kFps; + t.created_utc_time = kBaseUtc + int64_t(i) * kFrameTicks; + writer->RecordFrame(frame, t); + } + writer->Stop(); + return cfg.output_filepath; + } + + std::string WriteMultiBucketSource(const std::string& suffix) const { + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = Path(suffix + "_src"); + cfg.schema_json_content = MultiBucketSchemaJson(); + cfg.replay_name = "ReplayFilterTest"; + cfg.replay_uuid = "uuid-filter-" + suffix; + cfg.default_fps = kFps; + cfg.chunk_max_frames = kChunkFrames; + cfg.use_compression = true; + + auto writer = GetParam() == VTX::VtxFormat::FlatBuffers ? VTX::CreateFlatBuffersWriterFacade(cfg) + : VTX::CreateProtobufWriterFacade(cfg); + EXPECT_TRUE(writer); + if (!writer) { + return {}; + } + for (int i = 0; i < kFrames; ++i) { + auto frame = BuildMultiBucketFrame(i); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / kFps; + t.created_utc_time = kBaseUtc + int64_t(i) * kFrameTicks; + writer->RecordFrame(frame, t); + } + writer->Stop(); + return cfg.output_filepath; + } +}; + +// --------------------------------------------------------------------------- +// Blacklist by struct name: every Projectile goes, everything else is untouched, +// header/footer framing survives, the seek table describes the bytes on disk. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, DropByStructName) { + const std::string src = WriteSource("dropstruct"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropstruct_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Drop; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + + std::vector> ticks; + const auto r = VTX::FilterReplayFile(src, dst, spec, [&](int32_t done, int32_t total) { + ticks.emplace_back(done, total); + return true; + }); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, kFrames); + EXPECT_EQ(r.chunks_rewritten, 3); + EXPECT_EQ(r.entities_seen, 4u * kFrames); + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.entities_kept, 3u * kFrames); + ASSERT_EQ(r.dropped_by_struct.size(), 1u); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), 1u * kFrames); + EXPECT_EQ(r.events_kept, 0u); + EXPECT_EQ(r.events_dropped, 0u); + EXPECT_GT(r.output_bytes, 0u); + EXPECT_LE(r.output_bytes, r.source_bytes); + EXPECT_GE(r.elapsed_seconds, 0.0); + ASSERT_EQ(ticks.size(), 4u); // (0,3) then one per chunk + EXPECT_EQ(ticks.front(), std::make_pair(0, 3)); + EXPECT_EQ(ticks.back(), std::make_pair(3, 3)); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFiltered(source, output, + [](const VTX::Bucket& b, size_t i) { return b.entities[i].entity_type_id == kProjectile; }); + ExpectSeekTableMatchesFile(dst, output.footer); + // The reader exposes the seek-table checksums through GetFooter() for the + // source too (they used to be dropped in the footer conversion). + ExpectSeekTableMatchesFile(src, source.footer); +} + +// --------------------------------------------------------------------------- +// Whitelist by unique id glob: only the players remain. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, KeepByUniqueIdGlob) { + const std::string src = WriteSource("keepid"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("keepid_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Keep; + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("player_*")); + + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_kept, 2u * kFrames); + EXPECT_EQ(r.entities_dropped, 2u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), 1u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("MatchState"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFiltered(source, output, + [](const VTX::Bucket& b, size_t i) { return b.unique_ids[i].rfind("player_", 0) != 0; }); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 1u); + EXPECT_EQ(frame.GetBuckets()[0].unique_ids, (std::vector {"player_0", "player_1"})); + } + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// Property rules: on one struct, and on "any struct with that field". +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, DropByPropertyValue) { + const std::string src = WriteSource("dropprop"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropprop_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Name", "Bra*")); + // Empty struct = every struct that declares a scalar "UniqueID" (all three do). + spec.rules.push_back(VTX::ReplayFilterRule::Property("", "UniqueID", "proj_1")); + + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + int proj1_frames = 0; + for (int i = 0; i < kFrames; ++i) { + proj1_frames += (i % 3 == 1) ? 1 : 0; + } + EXPECT_EQ(r.entities_dropped, uint64_t(kFrames + proj1_frames)); + EXPECT_EQ(r.dropped_by_struct.at("Player"), uint64_t(kFrames)); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), uint64_t(proj1_frames)); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFiltered(source, output, [](const VTX::Bucket& b, size_t i) { + return b.unique_ids[i] == "player_1" || b.unique_ids[i] == "proj_1"; + }); + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// A numeric property, matched through its decimal string form. +TEST_P(ReplayFilterTest, DropByNumericProperty) { + const std::string src = WriteSource("dropnum"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropnum_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Team", "2")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectFramesFiltered(source, output, [](const VTX::Bucket& b, size_t i) { return b.unique_ids[i] == "player_1"; }); +} + +// --------------------------------------------------------------------------- +// Case-insensitive glob on the struct name. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, CaseInsensitiveStructGlob) { + const std::string src = WriteSource("cistruct"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("cistruct_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("MATCH*", /*case_insensitive=*/true)); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("MatchState"), 1u * kFrames); + + // Case-sensitive by default: the same pattern matches nothing. + VTX::ReplayFilterSpec strict; + strict.rules.push_back(VTX::ReplayFilterRule::StructName("MATCH*")); + const auto r2 = VTX::FilterReplayFile(src, Path("cistruct_strict_out"), strict); + ASSERT_TRUE(r2.ok()) << r2.error; + EXPECT_EQ(r2.entities_dropped, 0u); + EXPECT_EQ(r2.entities_kept, 4u * kFrames); +} + +// --------------------------------------------------------------------------- +// A whitelist that matches nothing still writes every frame, with empty buckets. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, KeepNothingLeavesEmptyBuckets) { + const std::string src = WriteSource("keepnone"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("keepnone_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Keep; + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("nothing_has_this_id")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_kept, 0u); + EXPECT_EQ(r.entities_dropped, 4u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ASSERT_EQ(output.frames.size(), size_t(kFrames)); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 1u); + EXPECT_TRUE(frame.GetBuckets()[0].entities.empty()); + EXPECT_TRUE(frame.GetBuckets()[0].unique_ids.empty()); + } + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// Timeline events ride along and are pruned to the entities that still exist. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, EventsCarriedAndPruned) { + const std::string src = WriteSource("events"); + ASSERT_FALSE(src.empty()); + const std::vector events = { + MakeEvent(0.5f, "Fire", "proj_0 launched", "proj_0"), + MakeEvent(1.0f, "Spawn", "player_0 spawned", "player_0"), + MakeEvent(1.5f, "Round", "round started", ""), + }; + RewriteFooterWithEvents(src, GetParam(), events); + { + const Snapshot s = ReadReplay(src); + ASSERT_TRUE(s.ok); + ASSERT_EQ(s.footer.events.size(), 3u) << "fixture should now carry events"; + EXPECT_EQ(s.footer.total_frames, kFrames); + } + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + spec.prune_events = true; + const std::string pruned = Path("events_pruned"); + const auto r = VTX::FilterReplayFile(src, pruned, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.events_kept, 2u); + EXPECT_EQ(r.events_dropped, 1u); + { + const Snapshot out = ReadReplay(pruned); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.footer.events.size(), 2u); + EXPECT_EQ(out.footer.events[0].entity_unique_id, "player_0"); + EXPECT_EQ(out.footer.events[0].event_type, "Spawn"); + EXPECT_EQ(out.footer.events[0].label, "player_0 spawned"); + EXPECT_FLOAT_EQ(out.footer.events[0].game_time, 1.0f); + EXPECT_EQ(out.footer.events[0].location, (VTX::Vector {1.0, 2.0, 3.0})); + EXPECT_EQ(out.footer.events[1].entity_unique_id, ""); + EXPECT_EQ(out.footer.events[1].label, "round started"); + EXPECT_EQ(out.footer.total_frames, kFrames); + } + + spec.prune_events = false; + const std::string carried = Path("events_carried"); + const auto r2 = VTX::FilterReplayFile(src, carried, spec); + ASSERT_TRUE(r2.ok()) << r2.error; + EXPECT_EQ(r2.events_kept, 3u); + EXPECT_EQ(r2.events_dropped, 0u); + const Snapshot out = ReadReplay(carried); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.footer.events.size(), 3u); + EXPECT_EQ(out.footer.events[0].entity_unique_id, "proj_0"); +} + +// --------------------------------------------------------------------------- +// Failure paths report an error and never leave a destination behind. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, ErrorsAreReported) { + const std::string src = WriteSource("errors"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("errors_out"); + namespace fs = std::filesystem; + + { + VTX::ReplayFilterSpec empty; + const auto r = VTX::FilterReplayFile(src, dst, empty); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("no rules"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Nope", "X", "*")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("not in the schema"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Nope", "*")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("has no field"), std::string::npos) << r.error; + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Position", "*")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("not a scalar"), std::string::npos) << r.error; + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("empty pattern"), std::string::npos) << r.error; + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + const auto r = VTX::FilterReplayFile(src, src, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("different file"), std::string::npos) << r.error; + EXPECT_TRUE(fs::exists(src)); + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + const auto r = VTX::FilterReplayFile(Path("does_not_exist"), dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("does not exist"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } +} + +TEST_P(ReplayFilterTest, CancelRemovesDestination) { + const std::string src = WriteSource("cancel"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("cancel_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + const auto r = VTX::FilterReplayFile(src, dst, spec, [](int32_t done, int32_t) { return done < 1; }); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("cancel"), std::string::npos) << r.error; + EXPECT_EQ(r.chunks_rewritten, 1); + EXPECT_FALSE(std::filesystem::exists(dst)); +} + +// --------------------------------------------------------------------------- +// Bucket rules: a dropped bucket is emptied in every frame and keeps its slot. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, DropByBucketName) { + const std::string src = WriteMultiBucketSource("dropbucket"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropbucket_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("fx")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_seen, 4u * kFrames); + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.entities_kept, 3u * kFrames); + ASSERT_EQ(r.dropped_by_bucket.size(), 1u); + EXPECT_EQ(r.dropped_by_bucket.at("fx"), 1u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFilteredByBucket(source, output, [](size_t bucket, const VTX::Bucket&, size_t) { return bucket == 1; }); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 3u); // the emptied bucket keeps its slot + EXPECT_TRUE(frame.GetBuckets()[1].entities.empty()); + EXPECT_TRUE(frame.GetBuckets()[1].unique_ids.empty()); + EXPECT_EQ(frame.GetBuckets()[0].unique_ids, (std::vector {"player_0", "player_1"})); + EXPECT_EQ(frame.GetBuckets()[2].unique_ids, (std::vector {"match"})); + } + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// A whitelisted bucket keeps all of its entities; other rules still apply elsewhere. +TEST_P(ReplayFilterTest, KeepBucketCombinesWithOtherRules) { + const std::string src = WriteMultiBucketSource("keepbucket"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("keepbucket_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Keep; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("telemetry")); + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Player")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_kept, 3u * kFrames); + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.dropped_by_bucket.at("fx"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFilteredByBucket(source, output, [](size_t bucket, const VTX::Bucket& b, size_t i) { + return bucket != 2 && b.entities[i].entity_type_id != kPlayer; + }); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 3u); + EXPECT_EQ(frame.GetBuckets()[0].entities.size(), 2u); + EXPECT_TRUE(frame.GetBuckets()[1].entities.empty()); + EXPECT_EQ(frame.GetBuckets()[2].unique_ids, (std::vector {"match"})); + } +} + +// Buckets can be addressed by index or by (case-insensitive) name glob; a rule +// that matches no schema bucket is an error rather than a silent no-op. +TEST_P(ReplayFilterTest, BucketByIndexAndGlob) { + const std::string src = WriteMultiBucketSource("bucketidx"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("bucketidx_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("2")); + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("F?", /*case_insensitive=*/true)); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_dropped, 2u * kFrames); + EXPECT_EQ(r.dropped_by_bucket.at("fx"), 1u * kFrames); + EXPECT_EQ(r.dropped_by_bucket.at("telemetry"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectFramesFilteredByBucket(source, output, [](size_t bucket, const VTX::Bucket&, size_t) { return bucket >= 1; }); + + VTX::ReplayFilterSpec strict; + strict.rules.push_back(VTX::ReplayFilterRule::Bucket("F?")); // case-sensitive: matches nothing + const auto r2 = VTX::FilterReplayFile(src, Path("bucketidx_strict_out"), strict); + EXPECT_FALSE(r2.ok()); + EXPECT_NE(r2.error.find("no schema bucket"), std::string::npos) << r2.error; + EXPECT_NE(r2.error.find("'telemetry'"), std::string::npos) << r2.error; // lists the available names + + VTX::ReplayFilterSpec bad_index; + bad_index.rules.push_back(VTX::ReplayFilterRule::Bucket("99999999999")); + const auto r3 = VTX::FilterReplayFile(src, Path("bucketidx_bad_out"), bad_index); + EXPECT_FALSE(r3.ok()); + EXPECT_NE(r3.error.find("not a valid bucket index"), std::string::npos) << r3.error; +} + +INSTANTIATE_TEST_SUITE_P(Formats, ReplayFilterTest, + ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), + [](const ::testing::TestParamInfo& info) { return FormatName(info.param); }); + +// =========================================================================== +// Rule primitives, no file involved. +// =========================================================================== +TEST(ReplayFilterGlob, Semantics) { + EXPECT_TRUE(VTX::GlobMatch("*", "")); + EXPECT_TRUE(VTX::GlobMatch("*", "anything")); + EXPECT_TRUE(VTX::GlobMatch("", "")); + EXPECT_FALSE(VTX::GlobMatch("", "a")); + EXPECT_TRUE(VTX::GlobMatch("abc", "abc")); + EXPECT_FALSE(VTX::GlobMatch("abc", "abcd")); + EXPECT_TRUE(VTX::GlobMatch("a*", "abc")); + EXPECT_TRUE(VTX::GlobMatch("*c", "abc")); + EXPECT_TRUE(VTX::GlobMatch("a?c", "abc")); + EXPECT_FALSE(VTX::GlobMatch("a?c", "ac")); + EXPECT_TRUE(VTX::GlobMatch("*.vtx", "capture.vtx")); + EXPECT_TRUE(VTX::GlobMatch("*x*y*", "axbyc")); + EXPECT_TRUE(VTX::GlobMatch("a*b*c", "axxbxxc")); + EXPECT_FALSE(VTX::GlobMatch("a*b*c", "acb")); + EXPECT_TRUE(VTX::GlobMatch("**a", "ba")); + EXPECT_FALSE(VTX::GlobMatch("abc", "ABC")); + EXPECT_TRUE(VTX::GlobMatch("abc", "ABC", /*case_insensitive=*/true)); + EXPECT_TRUE(VTX::GlobMatch("SystemHealth*", "SystemHealthProcess")); + EXPECT_FALSE(VTX::GlobMatch("SystemHealth*", "PlayerState")); +} + +TEST(ReplayFilterMatcher, ApplyRemovesRejectedEntitiesInPlace) { + VTX::SchemaRegistry registry; + ASSERT_TRUE(registry.LoadFromJson(VtxTest::FixturePath("test_schema.json"))); + const VTX::PropertyAddressCache& cache = registry.GetPropertyCache(); + ASSERT_FALSE(cache.structs.empty()); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Drop; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("player_1")); + const VTX::ReplayFilterMatcher matcher(spec, cache); + ASSERT_TRUE(matcher.valid()) << matcher.error(); + EXPECT_EQ(matcher.TypeName(kProjectile), "Projectile"); + EXPECT_EQ(matcher.TypeName(999), "type#999"); + + VTX::Frame frame = BuildFrame(5); // player_0, player_1, proj_2, match + VTX::Bucket& bucket = frame.GetBucket("entity"); + ASSERT_EQ(bucket.entities.size(), 4u); + EXPECT_TRUE(matcher.Keeps(0, bucket, 0)); + EXPECT_FALSE(matcher.Keeps(0, bucket, 1)); + EXPECT_FALSE(matcher.Keeps(0, bucket, 2)); + EXPECT_TRUE(matcher.Keeps(0, bucket, 3)); + EXPECT_FALSE(matcher.Matches(0, bucket, 42)); // out of range never matches + + std::vector dropped_ids; + const size_t dropped = matcher.Apply( + 0, bucket, [&](const VTX::Bucket& b, size_t index) { dropped_ids.push_back(b.unique_ids[index]); }); + EXPECT_EQ(dropped, 2u); + EXPECT_EQ(dropped_ids, (std::vector {"player_1", "proj_2"})); + EXPECT_EQ(bucket.unique_ids, (std::vector {"player_0", "match"})); + ASSERT_EQ(bucket.entities.size(), 2u); + EXPECT_EQ(bucket.entities[0].entity_type_id, kPlayer); + EXPECT_EQ(bucket.entities[1].entity_type_id, kMatchState); + EXPECT_TRUE(bucket.type_ranges.empty()); + + // Nothing left to remove: the bucket is untouched and 0 is reported. + EXPECT_EQ(matcher.Apply(0, bucket), 0u); + EXPECT_EQ(bucket.entities.size(), 2u); + + // Keep mode with a property rule resolved against the schema. + VTX::ReplayFilterSpec keep; + keep.mode = VTX::ReplayFilterMode::Keep; + keep.rules.push_back(VTX::ReplayFilterRule::Property("MatchState", "Phase", "warm*")); + const VTX::ReplayFilterMatcher keeper(keep, cache); + ASSERT_TRUE(keeper.valid()) << keeper.error(); + VTX::Frame early = BuildFrame(5); // Phase = warmup + VTX::Frame late = BuildFrame(75); // Phase = live + EXPECT_EQ(keeper.Apply(0, early.GetBucket("entity")), 3u); + EXPECT_EQ(early.GetBucket("entity").unique_ids, (std::vector {"match"})); + EXPECT_EQ(keeper.Apply(0, late.GetBucket("entity")), 4u); + EXPECT_TRUE(late.GetBucket("entity").entities.empty()); + + // Unresolvable rules are reported, not silently ignored. + VTX::ReplayFilterSpec bad; + bad.rules.push_back(VTX::ReplayFilterRule::Property("", "NoSuchField", "*")); + const VTX::ReplayFilterMatcher invalid(bad, cache); + EXPECT_FALSE(invalid.valid()); + EXPECT_NE(invalid.error().find("NoSuchField"), std::string::npos) << invalid.error(); +} + +TEST(ReplayFilterMatcher, BucketRulesResolveAgainstSchemaBuckets) { + VTX::SchemaRegistry registry; + ASSERT_TRUE(registry.LoadFromRawString(MultiBucketSchemaJson())); + const VTX::PropertyAddressCache& cache = registry.GetPropertyCache(); + ASSERT_EQ(cache.bucket_names, (std::vector {"entity", "fx", "telemetry"})); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("tele*")); + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("7")); // by index, beyond the named ones + const VTX::ReplayFilterMatcher matcher(spec, cache); + ASSERT_TRUE(matcher.valid()) << matcher.error(); + EXPECT_FALSE(matcher.BucketMatches(0)); + EXPECT_FALSE(matcher.BucketMatches(1)); + EXPECT_TRUE(matcher.BucketMatches(2)); + EXPECT_TRUE(matcher.BucketMatches(7)); + EXPECT_FALSE(matcher.BucketMatches(-1)); + EXPECT_EQ(matcher.BucketName(1), "fx"); + EXPECT_EQ(matcher.BucketName(7), "bucket#7"); + + VTX::Frame frame = BuildMultiBucketFrame(3); + VTX::Bucket& telemetry = frame.GetBucket("telemetry"); + VTX::Bucket& fx = frame.GetBucket("fx"); + EXPECT_TRUE(matcher.Matches(2, telemetry, 0)); + EXPECT_FALSE(matcher.Matches(1, fx, 0)); + EXPECT_FALSE(matcher.Keeps(2, telemetry, 0)); // Drop mode + EXPECT_TRUE(matcher.Keeps(1, fx, 0)); + EXPECT_EQ(matcher.Apply(2, telemetry), 1u); + EXPECT_TRUE(telemetry.entities.empty()); + EXPECT_EQ(matcher.Apply(1, fx), 0u); + + VTX::ReplayFilterSpec unknown; + unknown.rules.push_back(VTX::ReplayFilterRule::Bucket("nope")); + const VTX::ReplayFilterMatcher invalid(unknown, cache); + EXPECT_FALSE(invalid.valid()); + EXPECT_NE(invalid.error().find("no schema bucket"), std::string::npos) << invalid.error(); +} diff --git a/tests/util/replay_snapshot.h b/tests/util/replay_snapshot.h new file mode 100644 index 0000000..293fddd --- /dev/null +++ b/tests/util/replay_snapshot.h @@ -0,0 +1,109 @@ +#pragma once +// Read-back helpers shared by the replay rewrite tests (framing, cut, filter): +// open a .vtx, pull every frame, and check a footer's seek table against the +// bytes actually on disk through the shared framing definitions. + +#include + +#include +#include +#include +#include + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" + +namespace VtxTest { + + struct ReplaySnapshot { + bool ok = false; + VTX::VtxFormat format = VTX::VtxFormat::Unknown; + VTX::FileHeader header; + VTX::ContextualSchema schema; + VTX::FileFooter footer; + std::vector frames; + }; + + /// Opens @p path and copies out header, schema, footer and every frame. + inline ReplaySnapshot ReadReplay(const std::string& path) { + ReplaySnapshot s; + VTX::ReaderContext ctx = VTX::OpenReplayFile(path); + if (!ctx.Loaded()) { + ADD_FAILURE() << "open failed for " << path << ": " << ctx.GetError().message; + return s; + } + if (!ctx.WaitUntilReady()) { + ADD_FAILURE() << "load failed for " << path << ": " << ctx.GetReadyError().message; + return s; + } + s.format = ctx.format; + s.header = ctx->GetHeader(); + s.schema = ctx->GetContextualSchema(); + s.footer = ctx->GetFooter(); + const int32_t total = ctx->GetTotalFrames(); + s.frames.reserve(static_cast(total)); + for (int32_t i = 0; i < total; ++i) { + const VTX::Frame* frame = ctx->GetFrameSync(i); + if (!frame) { + ADD_FAILURE() << "frame " << i << " unreadable in " << path; + return s; + } + s.frames.push_back(*frame); + } + s.ok = true; + return s; + } + + /// Field-by-field equality of two entities (scalars, math types, the common arrays, container counts). + inline void ExpectEntityEqual(const VTX::PropertyContainer& a, const VTX::PropertyContainer& b, + const std::string& what) { + EXPECT_EQ(a.entity_type_id, b.entity_type_id) << what; + EXPECT_EQ(a.content_hash, b.content_hash) << what; + EXPECT_EQ(a.bool_properties, b.bool_properties) << what; + EXPECT_EQ(a.int32_properties, b.int32_properties) << what; + EXPECT_EQ(a.int64_properties, b.int64_properties) << what; + EXPECT_EQ(a.float_properties, b.float_properties) << what; + EXPECT_EQ(a.double_properties, b.double_properties) << what; + EXPECT_EQ(a.string_properties, b.string_properties) << what; + EXPECT_EQ(a.transform_properties, b.transform_properties) << what; + EXPECT_EQ(a.vector_properties, b.vector_properties) << what; + EXPECT_EQ(a.quat_properties, b.quat_properties) << what; + EXPECT_EQ(a.range_properties, b.range_properties) << what; + EXPECT_EQ(a.int32_arrays.data, b.int32_arrays.data) << what; + EXPECT_EQ(a.int32_arrays.offsets, b.int32_arrays.offsets) << what; + EXPECT_EQ(a.float_arrays.data, b.float_arrays.data) << what; + EXPECT_EQ(a.float_arrays.offsets, b.float_arrays.offsets) << what; + EXPECT_EQ(a.string_arrays.data, b.string_arrays.data) << what; + EXPECT_EQ(a.string_arrays.offsets, b.string_arrays.offsets) << what; + EXPECT_EQ(a.vector_arrays.data, b.vector_arrays.data) << what; + EXPECT_EQ(a.any_struct_properties.size(), b.any_struct_properties.size()) << what; + EXPECT_EQ(a.map_properties.size(), b.map_properties.size()) << what; + } + + /// Every seek-table entry must describe the bytes on disk: the size prefix equals + /// chunk_size_bytes - prefix, the xxHash64 of the payload matches, and chunks are + /// contiguous from the first one onwards. + inline void ExpectSeekTableMatchesFile(const std::string& path, const VTX::FileFooter& footer) { + std::ifstream in(path, std::ios::binary); + ASSERT_TRUE(in) << path; + const uint64_t file_size = VTX::Framing::StreamSize(in); + ASSERT_FALSE(footer.chunk_index.empty()); + for (const VTX::ChunkIndexEntry& entry : footer.chunk_index) { + ASSERT_LE(entry.file_offset + entry.chunk_size_bytes, file_size) << "chunk " << entry.chunk_index; + char prefix[VTX::Framing::kSizePrefixSize]; + ASSERT_TRUE(VTX::Framing::ReadAt(in, entry.file_offset, prefix, sizeof(prefix))); + const uint32_t payload_size = VTX::Framing::DecodeSizePrefix(prefix); + ASSERT_EQ(VTX::Framing::ChunkSizeOnDisk(payload_size), entry.chunk_size_bytes) + << "chunk " << entry.chunk_index; + std::string payload(payload_size, '\0'); + ASSERT_TRUE(in.read(payload.data(), payload_size)); + EXPECT_EQ(VTX::Framing::PayloadChecksum(payload), entry.checksum) << "chunk " << entry.chunk_index; + } + for (size_t i = 1; i < footer.chunk_index.size(); ++i) { + EXPECT_EQ(footer.chunk_index[i].file_offset, + footer.chunk_index[i - 1].file_offset + footer.chunk_index[i - 1].chunk_size_bytes); + } + } + +} // namespace VtxTest diff --git a/tools/inspector/CMakeLists.txt b/tools/inspector/CMakeLists.txt index 6409e1c..a8223c5 100644 --- a/tools/inspector/CMakeLists.txt +++ b/tools/inspector/CMakeLists.txt @@ -29,10 +29,19 @@ target_include_directories(vtx_inspector PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include ) +# File > Repair Replay / Cut Replay / Filter Entities run through vtx_transform, which +# is only built when reader + writer are; fail at configure time rather than at link time. +if(NOT TARGET vtx_transform) + message(FATAL_ERROR + "vtx_inspector requires the vtx_transform module (File > Repair Replay / Cut Replay / Filter Entities). " + "Enable VTX_BUILD_TRANSFORM together with VTX_BUILD_READER and VTX_BUILD_WRITER, " + "or configure with -DBUILD_VTX_INSPECTOR=OFF.") +endif() + target_link_libraries(vtx_inspector PRIVATE vtx_tool_shared vtx_reader - vtx_writer # VTX::RepairReplayFile for File > Repair Replay + vtx_transform # VTX::RepairReplayFile / VTX::CutReplayFile / VTX::FilterReplayFile ) set_target_properties(vtx_inspector PROPERTIES diff --git a/tools/inspector/include/inspector_layout.h b/tools/inspector/include/inspector_layout.h index 19405dd..31fc384 100644 --- a/tools/inspector/include/inspector_layout.h +++ b/tools/inspector/include/inspector_layout.h @@ -4,6 +4,7 @@ #include "gui/gui_layer.h" class CutReplayWindow; +class FilterEntitiesWindow; class GuiScaleController; class InspectorSession; class RepairReplayWindow; @@ -32,4 +33,7 @@ class InspectorLayout : public IGuiLayer { // File > Cut Replay: floating, operates on the loaded replay. At most one. std::shared_ptr cut_window_; + + // File > Filter Entities: floating, operates on the loaded replay. At most one. + std::shared_ptr filter_window_; }; diff --git a/tools/inspector/include/services/replay_cut_service.h b/tools/inspector/include/services/replay_cut_service.h deleted file mode 100644 index 15e3780..0000000 --- a/tools/inspector/include/services/replay_cut_service.h +++ /dev/null @@ -1,47 +0,0 @@ -#pragma once - -#include -#include - -#include "vtx/common/vtx_types.h" - -namespace VtxServices { - - // Resolved cut request. Interior chunks are always copied verbatim; when the - // requested range starts or ends inside a chunk (trims_head/trims_tail) that - // edge chunk is re-serialized with only the kept frames, and its seek-table - // entry (size, checksum, frame range) is regenerated. - struct ReplayCutPlan { - bool valid = false; - std::string error; // set when !valid - - int first_chunk = 0; // index into footer.chunk_index - int last_chunk = 0; - int first_frame = 0; // exact kept range, original frame numbering - int last_frame = 0; - bool trims_head = false; // first chunk starts before first_frame - bool trims_tail = false; // last chunk ends after last_frame - uint64_t chunk_bytes = 0; // on-disk bytes of the involved source chunks - }; - - // Cuts a new .vtx out of an existing flatbuffers replay: header block copied - // verbatim, whole chunks copied verbatim, partial edge chunks rewritten, and - // the footer rebuilt (frame numbering rebased to 0, time table sliced, - // duration recomputed). Timeline events are not carried over. - class ReplayCutService { - public: - // Exact cut: keeps [start_frame, end_frame] inclusive, trimming inside - // the edge chunks when the bounds do not fall on chunk boundaries. - static ReplayCutPlan PlanCutFrames(const VTX::FileFooter& footer, int start_frame, int end_frame); - - // Whole-chunk cut: keeps chunks [first_chunk, last_chunk] untouched. - static ReplayCutPlan PlanCutChunks(const VTX::FileFooter& footer, int first_chunk, int last_chunk); - - // Writes the cut to dest_path. Returns false and fills `error` on failure. - // dest_path must differ from source_path. When the plan trims an edge - // chunk, the source is reopened with the SDK reader to re-serialize it. - static bool ExecuteCut(const std::string& source_path, const VTX::FileFooter& footer, const ReplayCutPlan& plan, - const std::string& dest_path, std::string& error); - }; - -} // namespace VtxServices diff --git a/tools/inspector/include/windows/cut_replay_window.h b/tools/inspector/include/windows/cut_replay_window.h index 87b405c..3306a34 100644 --- a/tools/inspector/include/windows/cut_replay_window.h +++ b/tools/inspector/include/windows/cut_replay_window.h @@ -5,7 +5,8 @@ #include #include "gui/gui_layer.h" -#include "services/replay_cut_service.h" +#include "vtx/common/vtx_types.h" +#include "vtx/transform/vtx_replay_cut.h" class InspectorSession; @@ -43,8 +44,8 @@ class CutReplayWindow : public IGuiLayer { // Resolves the current inputs to an inclusive frame range; false when the // inputs cannot be parsed (error_out explains why). bool ResolveRequestedFrames(int& start_frame, int& end_frame, std::string& error_out) const; - void StartCut(const VtxServices::ReplayCutPlan& plan, const std::string& dest_path); - static Outcome RunCut(std::string source_path, VTX::FileFooter footer, VtxServices::ReplayCutPlan plan, + void StartCut(const VTX::ReplayCutPlan& plan, const std::string& dest_path); + static Outcome RunCut(std::string source_path, VTX::FileFooter footer, VTX::ReplayCutPlan plan, std::string dest_path); std::shared_ptr session_; diff --git a/tools/inspector/include/windows/filter_entities_window.h b/tools/inspector/include/windows/filter_entities_window.h new file mode 100644 index 0000000..4037b28 --- /dev/null +++ b/tools/inspector/include/windows/filter_entities_window.h @@ -0,0 +1,109 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include "gui/gui_layer.h" +#include "vtx/common/vtx_property_cache.h" +#include "vtx/transform/vtx_replay_filter.h" + +class InspectorSession; + +// Floating window (File > Filter Entities...) that writes a new .vtx with entities +// of the loaded replay dropped (blacklist) or exclusively kept (whitelist) by +// struct name, unique id glob or property value. Every frame survives; only the +// entity payload changes. The rewrite is VTX::FilterReplayFile on a worker thread +// with live chunk progress and cancel. +// +// Implements IGuiLayer directly (rather than the shared ImGuiWindow base) for the +// same reason as CutReplayWindow: gui/gui_window.h's `class ImGuiWindow` clashes +// with Dear ImGui's `struct ImGuiWindow` in translation units that include +// imgui_internal.h (inspector_layout.cpp). +class FilterEntitiesWindow : public IGuiLayer { +public: + explicit FilterEntitiesWindow(std::shared_ptr session); + ~FilterEntitiesWindow() override; + + void OnUpdate() override {} + void OnRender() override; + + void SetOpen(bool open) { is_open_ = open; } + bool IsOpen() const { return is_open_; } + +private: + enum class Phase { Idle, Running, Done }; + + struct BucketRow { + std::string name; + int32_t index = -1; ///< Bucket number within a frame (schema order). + bool selected = false; + }; + + struct StructRow { + std::string name; + int32_t type_id = -1; + bool selected = false; + }; + + struct PropertyRow { + char struct_name[128] = {}; + char field_name[128] = {}; + char pattern[256] = {}; + }; + + // Shared with the worker thread. + struct Progress { + std::atomic done {0}; + std::atomic total {0}; + std::atomic cancel {false}; + }; + + // What the current rules do to the current frame (recomputed when either changes). + struct Preview { + std::string key; + std::string error; ///< Rule resolution error from the matcher, if any. + bool has_matcher = false; + int32_t frame_index = -1; + uint64_t seen = 0; + uint64_t kept = 0; + std::vector>> buckets; ///< name -> (kept, seen) + std::vector per_struct_seen; ///< indexed like structs_ + std::vector per_bucket_seen; ///< indexed by bucket number + }; + + void DrawContent(); + void DrawRules(); + void DrawPreview(const VTX::ReplayFilterSpec& spec); + void DrawRunControls(const VTX::ReplayFilterSpec& spec); + void DrawOutcome(); + VTX::ReplayFilterSpec BuildSpec() const; + static std::string SpecSignature(const VTX::ReplayFilterSpec& spec); + void RefreshPreview(const VTX::ReplayFilterSpec& spec, const std::string& key); + void StartFilter(const VTX::ReplayFilterSpec& spec, const std::string& dest_path); + void JoinWorker(); + + std::shared_ptr session_; + VTX::PropertyAddressCache schema_cache_; + std::vector buckets_; + std::vector structs_; + std::vector property_rules_; + char unique_id_patterns_[4096] = {}; + int mode_ = 0; // 0 = drop matching (blacklist), 1 = keep only matching (whitelist) + bool case_insensitive_ = false; + bool prune_events_ = true; + + Preview preview_; + + Phase phase_ = Phase::Idle; + std::shared_ptr progress_; + std::future future_; + VTX::ReplayFilterResult outcome_; + std::string outcome_dest_; + bool outcome_logged_ = false; + bool is_open_ = true; +}; diff --git a/tools/inspector/include/windows/repair_replay_window.h b/tools/inspector/include/windows/repair_replay_window.h index a17dfb3..0f0a50f 100644 --- a/tools/inspector/include/windows/repair_replay_window.h +++ b/tools/inspector/include/windows/repair_replay_window.h @@ -4,7 +4,7 @@ #include #include "gui/gui_layer.h" -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" // Floating window that repairs a crashed .vtx from its ".recovery" sidecar via // VTX::RepairReplayFile. The user selects the .vtx (and, if it lives elsewhere, the diff --git a/tools/inspector/src/inspector_layout.cpp b/tools/inspector/src/inspector_layout.cpp index a6b2754..3b08406 100644 --- a/tools/inspector/src/inspector_layout.cpp +++ b/tools/inspector/src/inspector_layout.cpp @@ -12,6 +12,7 @@ #include "inspector_session.h" #include "windows/analysis_window_factory.h" #include "windows/cut_replay_window.h" +#include "windows/filter_entities_window.h" #include "windows/repair_replay_window.h" namespace { @@ -270,6 +271,15 @@ void InspectorLayout::OnRender() { } } + // Rewrite the loaded replay into a new .vtx without the selected entities. + if (ImGui::MenuItem("Filter Entities...", nullptr, false, session_->HasLoadedReplay())) { + if (!filter_window_) { + filter_window_ = std::make_shared(session_); + } else { + filter_window_->SetOpen(true); + } + } + ImGui::Separator(); if (ImGui::MenuItem("Exit")) { @@ -350,4 +360,12 @@ void InspectorLayout::OnRender() { cut_window_.reset(); } } + + // Filter-entities window (floating, single instance), same lifecycle as cut. + if (filter_window_) { + filter_window_->OnRender(); + if (!filter_window_->IsOpen()) { + filter_window_.reset(); + } + } } diff --git a/tools/inspector/src/services/replay_cut_service.cpp b/tools/inspector/src/services/replay_cut_service.cpp deleted file mode 100644 index 6aa805b..0000000 --- a/tools/inspector/src/services/replay_cut_service.cpp +++ /dev/null @@ -1,328 +0,0 @@ -#include "services/replay_cut_service.h" - -#include -#include -#include -#include -#include -#include - -#include -#include - -#include "vtx/reader/core/vtx_reader_facade.h" -#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" - -namespace { - - constexpr double kTicksPerSecond = 10'000'000.0; - - // Streams [offset, offset+bytes) from `in` to `out` through a bounded buffer. - bool CopyRange(std::ifstream& in, std::ofstream& out, uint64_t offset, uint64_t bytes, std::string& error) { - constexpr size_t kBufferSize = 4 * 1024 * 1024; - std::vector buffer(std::min(bytes, kBufferSize)); - in.seekg(static_cast(offset)); - uint64_t remaining = bytes; - while (remaining > 0) { - const size_t step = static_cast(std::min(remaining, buffer.size())); - if (!in.read(buffer.data(), static_cast(step))) { - error = "Read failed at offset " + std::to_string(offset + (bytes - remaining)); - return false; - } - if (!out.write(buffer.data(), static_cast(step))) { - error = "Write failed"; - return false; - } - remaining -= step; - } - return true; - } - - // Mirror of ChunkedFileSink::CompressIfBeneficial with the writer defaults - // (skip tiny payloads, keep the raw payload when zstd does not shrink it). - std::string CompressIfBeneficial(std::string payload) { - constexpr int kCompressionLevel = 10; - if (payload.size() < 512) { - return payload; - } - const size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - const size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), kCompressionLevel); - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) { - return payload; - } - compressed.resize(compressed_size); - return compressed; - } - - // Duration of the sliced range from a tick table (first..last nonzero), or a - // negative value when the table cannot answer. - double DurationFromTicks(const std::vector& ticks) { - int64_t first = 0; - int64_t last = 0; - for (const int64_t t : ticks) { - if (t != 0) { - if (first == 0) { - first = t; - } - last = t; - } - } - if (first == 0 || last < first) { - return -1.0; - } - return static_cast(last - first) / kTicksPerSecond; - } - - // Serializes frames [kept_lo, kept_hi] of the source into a fresh chunk - // payload (mirrors ChunkedFileSink::SaveChunk framing: length-prefixed, - // compressed when beneficial, checksummed on the on-disk bytes). - bool WriteRewrittenChunk(VTX::IVtxReaderFacade& reader, std::ofstream& out, int kept_lo, int kept_hi, - int new_chunk_index, int rebase_frame, VTX::ChunkIndexData& entry_out, - std::string& error) { - std::vector> frames; - frames.reserve(static_cast(kept_hi - kept_lo + 1)); - for (int frame = kept_lo; frame <= kept_hi; ++frame) { - const VTX::Frame* source_frame = reader.GetFrameSync(frame); - if (!source_frame) { - error = "Could not read frame " + std::to_string(frame) + " from the source replay."; - return false; - } - frames.push_back(std::make_unique(*source_frame)); - } - - std::string payload = - VTX::FlatBuffersVtxPolicy::SerializeChunk(frames, new_chunk_index, /*is_compressed=*/true); - payload = CompressIfBeneficial(std::move(payload)); - - entry_out.chunk_index = new_chunk_index; - entry_out.file_offset = static_cast(out.tellp()); - entry_out.chunk_size_bytes = static_cast(payload.size() + sizeof(uint32_t)); - entry_out.start_frame = kept_lo - rebase_frame; - entry_out.end_frame = kept_hi - rebase_frame; - entry_out.checksum = XXH3_64bits(payload.data(), payload.size()); - - const uint32_t payload_size = static_cast(payload.size()); - if (!out.write(reinterpret_cast(&payload_size), sizeof(payload_size)) || - !out.write(payload.data(), static_cast(payload.size()))) { - error = "Write failed for rewritten chunk " + std::to_string(new_chunk_index); - return false; - } - return true; - } - - VtxServices::ReplayCutPlan PlanForChunkSpan(const VTX::FileFooter& footer, int first_chunk, int last_chunk, - int first_frame, int last_frame) { - VtxServices::ReplayCutPlan plan; - plan.first_chunk = first_chunk; - plan.last_chunk = last_chunk; - plan.first_frame = first_frame; - plan.last_frame = last_frame; - plan.trims_head = footer.chunk_index[static_cast(first_chunk)].start_frame < first_frame; - plan.trims_tail = footer.chunk_index[static_cast(last_chunk)].end_frame > last_frame; - for (int i = first_chunk; i <= last_chunk; ++i) { - plan.chunk_bytes += footer.chunk_index[static_cast(i)].chunk_size_bytes; - } - plan.valid = true; - return plan; - } - -} // namespace - -namespace VtxServices { - - ReplayCutPlan ReplayCutService::PlanCutFrames(const VTX::FileFooter& footer, int start_frame, int end_frame) { - ReplayCutPlan plan; - if (footer.chunk_index.empty() || footer.total_frames <= 0) { - plan.error = "The loaded replay has no chunk index."; - return plan; - } - start_frame = std::clamp(start_frame, 0, footer.total_frames - 1); - end_frame = std::clamp(end_frame, 0, footer.total_frames - 1); - if (end_frame < start_frame) { - plan.error = "End of range is before its start."; - return plan; - } - - int first_chunk = -1; - int last_chunk = -1; - for (int i = 0; i < static_cast(footer.chunk_index.size()); ++i) { - const auto& entry = footer.chunk_index[static_cast(i)]; - if (first_chunk < 0 && start_frame <= entry.end_frame) { - first_chunk = i; - } - if (end_frame >= entry.start_frame) { - last_chunk = i; - } - } - if (first_chunk < 0 || last_chunk < first_chunk) { - plan.error = "The requested range does not overlap any chunk."; - return plan; - } - // Clamp the exact bounds into the covered span (start may fall in a gap - // between chunks; snap it to the first covered frame). - const int lo = std::max(start_frame, footer.chunk_index[static_cast(first_chunk)].start_frame); - const int hi = std::min(end_frame, footer.chunk_index[static_cast(last_chunk)].end_frame); - return PlanForChunkSpan(footer, first_chunk, last_chunk, lo, hi); - } - - ReplayCutPlan ReplayCutService::PlanCutChunks(const VTX::FileFooter& footer, int first_chunk, int last_chunk) { - ReplayCutPlan plan; - if (footer.chunk_index.empty()) { - plan.error = "The loaded replay has no chunk index."; - return plan; - } - const int max_chunk = static_cast(footer.chunk_index.size()) - 1; - first_chunk = std::clamp(first_chunk, 0, max_chunk); - last_chunk = std::clamp(last_chunk, 0, max_chunk); - if (last_chunk < first_chunk) { - plan.error = "End chunk is before start chunk."; - return plan; - } - return PlanForChunkSpan(footer, first_chunk, last_chunk, - footer.chunk_index[static_cast(first_chunk)].start_frame, - footer.chunk_index[static_cast(last_chunk)].end_frame); - } - - bool ReplayCutService::ExecuteCut(const std::string& source_path, const VTX::FileFooter& footer, - const ReplayCutPlan& plan, const std::string& dest_path, std::string& error) { - if (!plan.valid) { - error = plan.error.empty() ? "Invalid cut plan." : plan.error; - return false; - } - namespace fs = std::filesystem; - std::error_code ec; - if (fs::equivalent(fs::path(source_path), fs::path(dest_path), ec)) { - error = "Destination must be a different file than the source."; - return false; - } - - // Edge chunks that start/end inside the kept range must be re-serialized - // frame by frame; open a private reader for that (the caller's reader may - // be busy on the UI thread). - VTX::ReaderContext reader_context; - if (plan.trims_head || plan.trims_tail) { - reader_context = VTX::OpenReplayFile(source_path); - if (!reader_context.Loaded() || !reader_context.WaitUntilReady()) { - error = - "Could not reopen the source replay for edge-chunk rewriting: " + reader_context.GetError().message; - return false; - } - } - - std::ifstream in(source_path, std::ios::binary); - if (!in) { - error = "Could not open the source replay: " + source_path; - return false; - } - std::ofstream out(dest_path, std::ios::binary | std::ios::trunc); - if (!out) { - error = "Could not create the output file: " + dest_path; - return false; - } - - // 1) Header region, verbatim: everything before the first chunk of the - // ORIGINAL file (magic + length-prefixed header payload). The header - // carries no frame-dependent data; copying it keeps the schema and - // recording metadata bit-identical. - const uint64_t header_end = footer.chunk_index.front().file_offset; - if (!CopyRange(in, out, 0, header_end, error)) { - return false; - } - - // 2) Chunks. Whole chunks are copied verbatim; partial edge chunks are - // rebuilt with only the kept frames and get fresh index metadata. - std::vector seek_table; - seek_table.reserve(static_cast(plan.last_chunk - plan.first_chunk + 1)); - for (int i = plan.first_chunk; i <= plan.last_chunk; ++i) { - const auto& entry = footer.chunk_index[static_cast(i)]; - const int kept_lo = std::max(entry.start_frame, plan.first_frame); - const int kept_hi = std::min(entry.end_frame, plan.last_frame); - const int new_index = static_cast(seek_table.size()); - - if (kept_lo == entry.start_frame && kept_hi == entry.end_frame) { - VTX::ChunkIndexData rebased; - rebased.chunk_index = new_index; - rebased.file_offset = static_cast(out.tellp()); - rebased.chunk_size_bytes = entry.chunk_size_bytes; - rebased.start_frame = entry.start_frame - plan.first_frame; - rebased.end_frame = entry.end_frame - plan.first_frame; - rebased.checksum = entry.checksum; - seek_table.push_back(rebased); - if (!CopyRange(in, out, entry.file_offset, entry.chunk_size_bytes, error)) { - return false; - } - } else { - VTX::ChunkIndexData rewritten; - if (!WriteRewrittenChunk(*reader_context.reader, out, kept_lo, kept_hi, new_index, plan.first_frame, - rewritten, error)) { - return false; - } - seek_table.push_back(rewritten); - } - } - - // 3) Footer: time table sliced to the kept frames (tick values stay - // absolute; only frame indices are rebased), duration recomputed. - const auto slice_ticks = [&](const std::vector& ticks) { - std::vector sliced; - const size_t begin = std::min(static_cast(plan.first_frame), ticks.size()); - const size_t end = std::min(static_cast(plan.last_frame) + 1, ticks.size()); - sliced.reserve(end - begin); - for (size_t i = begin; i < end; ++i) { - sliced.push_back(static_cast(ticks[i])); - } - return sliced; - }; - const auto slice_frame_indices = [&](const std::vector& values) { - std::vector sliced; - for (const uint32_t value : values) { - const int frame = static_cast(value); - if (frame >= plan.first_frame && frame <= plan.last_frame) { - sliced.push_back(frame - plan.first_frame); - } - } - return sliced; - }; - - const std::vector game_times = slice_ticks(footer.times.game_time); - const std::vector created_utc = slice_ticks(footer.times.created_utc); - const std::vector gaps = slice_frame_indices(footer.times.gaps); - const std::vector segments = slice_frame_indices(footer.times.segments); - - VTX::SessionFooter session_footer; - session_footer.total_frames = plan.last_frame - plan.first_frame + 1; - double duration = DurationFromTicks(created_utc); - if (duration < 0.0) { - duration = DurationFromTicks(game_times); - } - if (duration < 0.0 && footer.total_frames > 0) { - duration = static_cast(footer.duration_seconds) * session_footer.total_frames / footer.total_frames; - } - session_footer.duration_seconds = std::max(duration, 0.0); - session_footer.game_times = &game_times; - session_footer.created_utc = &created_utc; - session_footer.gaps = &gaps; - session_footer.segments = &segments; - - // Written uncompressed; the reader sniffs the zstd magic and passes - // uncompressed payloads through. - const std::string footer_payload = VTX::FlatBuffersVtxPolicy::SerializeFooter(seek_table, session_footer); - const uint32_t footer_size = static_cast(footer_payload.size()); - const std::string magic = VTX::FlatBuffersVtxPolicy::GetMagicBytes(); - if (!out.write(footer_payload.data(), static_cast(footer_payload.size())) || - !out.write(reinterpret_cast(&footer_size), sizeof(footer_size)) || - !out.write(magic.data(), static_cast(magic.size()))) { - error = "Write failed while finishing the footer."; - return false; - } - out.flush(); - if (!out) { - error = "Flush failed for the output file."; - return false; - } - return true; - } - -} // namespace VtxServices diff --git a/tools/inspector/src/windows/cut_replay_window.cpp b/tools/inspector/src/windows/cut_replay_window.cpp index 9cb97a2..661eebd 100644 --- a/tools/inspector/src/windows/cut_replay_window.cpp +++ b/tools/inspector/src/windows/cut_replay_window.cpp @@ -184,15 +184,17 @@ bool CutReplayWindow::ResolveRequestedFrames(int& start_frame, int& end_frame, s } CutReplayWindow::Outcome CutReplayWindow::RunCut(std::string source_path, VTX::FileFooter footer, - VtxServices::ReplayCutPlan plan, std::string dest_path) { + VTX::ReplayCutPlan plan, std::string dest_path) { // Touches only its by-value arguments -- safe to run detached from the window. Outcome out; out.dest_path = dest_path; - out.ok = VtxServices::ReplayCutService::ExecuteCut(source_path, footer, plan, dest_path, out.error); + const VTX::ReplayCutResult result = VTX::CutReplayFile(source_path, footer, plan, dest_path); + out.ok = result.ok(); + out.error = result.error; return out; } -void CutReplayWindow::StartCut(const VtxServices::ReplayCutPlan& plan, const std::string& dest_path) { +void CutReplayWindow::StartCut(const VTX::ReplayCutPlan& plan, const std::string& dest_path) { phase_ = Phase::Running; spinner_tick_ = 0; outcome_ = Outcome {}; @@ -212,10 +214,6 @@ void CutReplayWindow::DrawContent() { range_initialized_ = false; return; } - if (session_->GetFormat() != VTX::VtxFormat::FlatBuffers) { - ImGui::TextColored(kWarn, "Cut currently supports flatbuffers (.vtx \"VTXF\") replays only."); - return; - } if (!range_initialized_) { ResetRangeToFullReplay(); range_initialized_ = true; @@ -273,15 +271,15 @@ void CutReplayWindow::DrawContent() { ImGui::Separator(); // Live plan preview. - VtxServices::ReplayCutPlan plan; + VTX::ReplayCutPlan plan; if (range_mode_ == 3) { - plan = VtxServices::ReplayCutService::PlanCutChunks(footer, chunk_start_, chunk_end_); + plan = VTX::PlanCutChunks(footer, chunk_start_, chunk_end_); } else { int requested_start = 0; int requested_end = 0; std::string input_error; if (ResolveRequestedFrames(requested_start, requested_end, input_error)) { - plan = VtxServices::ReplayCutService::PlanCutFrames(footer, requested_start, requested_end); + plan = VTX::PlanCutFrames(footer, requested_start, requested_end); } else { plan.error = input_error; } diff --git a/tools/inspector/src/windows/filter_entities_window.cpp b/tools/inspector/src/windows/filter_entities_window.cpp new file mode 100644 index 0000000..8ba8bfe --- /dev/null +++ b/tools/inspector/src/windows/filter_entities_window.cpp @@ -0,0 +1,491 @@ +#include "windows/filter_entities_window.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "gui/portable-file-dialogs.h" +#include "inspector_session.h" +#include "vtx/reader/core/vtx_reader_facade.h" + +namespace { + + const ImVec4 kOk {0.40f, 0.85f, 0.45f, 1.0f}; + const ImVec4 kWarn {0.95f, 0.80f, 0.30f, 1.0f}; + const ImVec4 kErr {0.95f, 0.45f, 0.45f, 1.0f}; + const ImVec4 kDim {0.65f, 0.65f, 0.65f, 1.0f}; + + std::string Trim(std::string s) { + const auto not_space = [](unsigned char c) { + return !std::isspace(c); + }; + s.erase(s.begin(), std::find_if(s.begin(), s.end(), not_space)); + s.erase(std::find_if(s.rbegin(), s.rend(), not_space).base(), s.end()); + return s; + } + + // Non-empty, trimmed lines of a text buffer. + std::vector Lines(const char* text) { + std::vector out; + std::istringstream in(text); + std::string line; + while (std::getline(in, line)) { + line = Trim(line); + if (!line.empty()) { + out.push_back(line); + } + } + return out; + } + + std::string FormatBytes(uint64_t bytes) { + char buffer[64]; + if (bytes >= 1024ull * 1024ull) { + std::snprintf(buffer, sizeof(buffer), "%.2f MB", static_cast(bytes) / (1024.0 * 1024.0)); + } else if (bytes >= 1024ull) { + std::snprintf(buffer, sizeof(buffer), "%.1f KB", static_cast(bytes) / 1024.0); + } else { + std::snprintf(buffer, sizeof(buffer), "%llu B", static_cast(bytes)); + } + return buffer; + } + +} // namespace + +FilterEntitiesWindow::FilterEntitiesWindow(std::shared_ptr session) + : session_(std::move(session)) { + if (session_ && session_->HasLoadedReplay()) { + if (VTX::IVtxReaderFacade* reader = session_->GetReader()) { + schema_cache_ = reader->GetPropertyAddressCache(); + for (const auto& [type_id, struct_cache] : schema_cache_.structs) { + structs_.push_back(StructRow {struct_cache.name, type_id, false}); + } + std::sort(structs_.begin(), structs_.end(), + [](const StructRow& a, const StructRow& b) { return a.name < b.name; }); + for (size_t b = 0; b < schema_cache_.bucket_names.size(); ++b) { + buckets_.push_back(BucketRow {schema_cache_.bucket_names[b], static_cast(b), false}); + } + } + } +} + +FilterEntitiesWindow::~FilterEntitiesWindow() { + JoinWorker(); +} + +void FilterEntitiesWindow::JoinWorker() { + if (future_.valid()) { + if (progress_) { + progress_->cancel.store(true); + } + future_.wait(); + } +} + +void FilterEntitiesWindow::OnRender() { + if (!is_open_) { + return; + } + ImGui::SetNextWindowSize(ImVec2(640, 700), ImGuiCond_FirstUseEver); + if (ImGui::Begin("Filter Entities", &is_open_)) { + DrawContent(); + } + ImGui::End(); +} + +VTX::ReplayFilterSpec FilterEntitiesWindow::BuildSpec() const { + VTX::ReplayFilterSpec spec; + spec.mode = mode_ == 1 ? VTX::ReplayFilterMode::Keep : VTX::ReplayFilterMode::Drop; + spec.prune_events = prune_events_; + for (const BucketRow& row : buckets_) { + if (row.selected) { + spec.rules.push_back(VTX::ReplayFilterRule::Bucket(std::to_string(row.index))); + } + } + for (const StructRow& row : structs_) { + if (row.selected) { + spec.rules.push_back(VTX::ReplayFilterRule::StructName(row.name, case_insensitive_)); + } + } + for (const std::string& pattern : Lines(unique_id_patterns_)) { + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId(pattern, case_insensitive_)); + } + for (const PropertyRow& row : property_rules_) { + const std::string field = Trim(row.field_name); + const std::string pattern = Trim(row.pattern); + if (field.empty() && pattern.empty()) { + continue; // untouched row + } + spec.rules.push_back(VTX::ReplayFilterRule::Property(Trim(row.struct_name), field, pattern, case_insensitive_)); + } + return spec; +} + +std::string FilterEntitiesWindow::SpecSignature(const VTX::ReplayFilterSpec& spec) { + std::string key = spec.mode == VTX::ReplayFilterMode::Keep ? "keep" : "drop"; + key += spec.prune_events ? "|prune" : "|carry"; + for (const VTX::ReplayFilterRule& rule : spec.rules) { + key += '|'; + key += std::to_string(static_cast(rule.kind)); + key += rule.case_insensitive ? "i:" : "c:"; + key += rule.struct_name; + key += '.'; + key += rule.field_name; + key += '='; + key += rule.pattern; + } + return key; +} + +void FilterEntitiesWindow::RefreshPreview(const VTX::ReplayFilterSpec& spec, const std::string& key) { + preview_ = Preview {}; + preview_.key = key; + preview_.per_struct_seen.assign(structs_.size(), 0); + preview_.per_bucket_seen.assign(buckets_.size(), 0); + + VTX::IVtxReaderFacade* reader = session_ ? session_->GetReader() : nullptr; + if (!reader) { + return; + } + preview_.frame_index = std::max(0, session_->GetCurrentFrame()); + const VTX::Frame* frame = reader->GetFrameSync(preview_.frame_index); + if (!frame) { + return; + } + + std::unordered_map struct_rows; + for (size_t i = 0; i < structs_.size(); ++i) { + struct_rows[structs_[i].type_id] = i; + } + + std::unique_ptr matcher; + if (!spec.rules.empty()) { + matcher = std::make_unique(spec, schema_cache_); + if (!matcher->valid()) { + preview_.error = matcher->error(); + matcher.reset(); + } + } + preview_.has_matcher = matcher != nullptr; + + const auto& buckets = frame->GetBuckets(); + for (size_t b = 0; b < buckets.size(); ++b) { + const VTX::Bucket& bucket = buckets[b]; + if (b < preview_.per_bucket_seen.size()) { + preview_.per_bucket_seen[b] = bucket.entities.size(); + } + uint64_t kept = 0; + for (size_t i = 0; i < bucket.entities.size(); ++i) { + const auto row = struct_rows.find(bucket.entities[i].entity_type_id); + if (row != struct_rows.end()) { + ++preview_.per_struct_seen[row->second]; + } + if (matcher && matcher->Keeps(static_cast(b), bucket, i)) { + ++kept; + } + } + std::string name = b < schema_cache_.bucket_names.size() ? schema_cache_.bucket_names[b] : std::string(); + if (name.empty()) { + name = "bucket " + std::to_string(b); + } + preview_.buckets.emplace_back(std::move(name), + std::make_pair(kept, static_cast(bucket.entities.size()))); + preview_.seen += bucket.entities.size(); + preview_.kept += kept; + } +} + +void FilterEntitiesWindow::StartFilter(const VTX::ReplayFilterSpec& spec, const std::string& dest_path) { + JoinWorker(); + phase_ = Phase::Running; + outcome_ = VTX::ReplayFilterResult {}; + outcome_dest_ = dest_path; + outcome_logged_ = false; + progress_ = std::make_shared(); + + // The worker touches only its by-value captures and the shared progress block. + const std::shared_ptr progress = progress_; + const std::string source = session_->current_file_path_; + future_ = std::async(std::launch::async, [source, dest_path, spec, progress]() { + return VTX::FilterReplayFile(source, dest_path, spec, [progress](int32_t done, int32_t total) { + progress->done.store(done); + progress->total.store(total); + return !progress->cancel.load(); + }); + }); +} + +void FilterEntitiesWindow::DrawContent() { + if (phase_ == Phase::Running && future_.valid() && + future_.wait_for(std::chrono::seconds(0)) == std::future_status::ready) { + outcome_ = future_.get(); + phase_ = Phase::Done; + if (!outcome_logged_ && session_) { + outcome_logged_ = true; + if (outcome_.ok()) { + session_->AddGuiInfoLog("Filter Entities: wrote " + outcome_dest_ + " (" + + std::to_string(outcome_.entities_kept) + " entities kept, " + + std::to_string(outcome_.entities_dropped) + " dropped)"); + } else { + session_->AddGuiErrorLog("Filter Entities failed: " + outcome_.error); + } + } + } + + if (!session_ || !session_->HasLoadedReplay()) { + ImGui::TextColored(kWarn, "Open a replay first."); + return; + } + + const std::filesystem::path source(session_->current_file_path_); + ImGui::TextWrapped("Source: %s", source.filename().string().c_str()); + ImGui::TextColored(kDim, + "%d frames, %d buckets, %d structs in the schema. Every frame is kept; only the entity " + "payload changes.", + session_->GetTotalFrames(), static_cast(buckets_.size()), + static_cast(structs_.size())); + ImGui::Separator(); + + const bool busy = phase_ == Phase::Running; + ImGui::BeginDisabled(busy); + ImGui::TextUnformatted("Mode"); + ImGui::SameLine(); + ImGui::RadioButton("Drop matching (blacklist)", &mode_, 0); + ImGui::SameLine(); + ImGui::RadioButton("Keep only matching (whitelist)", &mode_, 1); + ImGui::Spacing(); + DrawRules(); + ImGui::EndDisabled(); + + const VTX::ReplayFilterSpec spec = BuildSpec(); + ImGui::Separator(); + DrawPreview(spec); + ImGui::Separator(); + DrawRunControls(spec); + DrawOutcome(); +} + +void FilterEntitiesWindow::DrawRules() { + if (ImGui::CollapsingHeader("Buckets", ImGuiTreeNodeFlags_DefaultOpen)) { + if (ImGui::SmallButton("Select all##buckets")) { + for (BucketRow& row : buckets_) { + row.selected = true; + } + } + ImGui::SameLine(); + if (ImGui::SmallButton("Clear##buckets")) { + for (BucketRow& row : buckets_) { + row.selected = false; + } + } + ImGui::SameLine(); + ImGui::TextColored(kDim, "(a dropped bucket is emptied in every frame; its slot stays)"); + if (buckets_.empty()) { + ImGui::TextColored(kDim, "The schema names no buckets."); + } + for (size_t i = 0; i < buckets_.size(); ++i) { + BucketRow& row = buckets_[i]; + ImGui::PushID(static_cast(i) + 2000); + const uint64_t count = i < preview_.per_bucket_seen.size() ? preview_.per_bucket_seen[i] : 0; + char label[256]; + std::snprintf(label, sizeof(label), "%d: %s (%llu)", row.index, row.name.c_str(), + static_cast(count)); + ImGui::Checkbox(label, &row.selected); + ImGui::PopID(); + } + } + + if (ImGui::CollapsingHeader("Structs", ImGuiTreeNodeFlags_DefaultOpen)) { + if (ImGui::SmallButton("Select all")) { + for (StructRow& row : structs_) { + row.selected = true; + } + } + ImGui::SameLine(); + if (ImGui::SmallButton("Clear")) { + for (StructRow& row : structs_) { + row.selected = false; + } + } + ImGui::SameLine(); + ImGui::TextColored(kDim, "(count = entities of that struct in the current frame)"); + + const float row_height = ImGui::GetFrameHeightWithSpacing(); + const float height = + std::min(220.0f, row_height * static_cast(std::max(structs_.size(), 1)) + 8.0f); + if (ImGui::BeginChild("##structs", ImVec2(0, height), ImGuiChildFlags_Border)) { + for (size_t i = 0; i < structs_.size(); ++i) { + StructRow& row = structs_[i]; + ImGui::PushID(static_cast(i)); + const uint64_t count = i < preview_.per_struct_seen.size() ? preview_.per_struct_seen[i] : 0; + char label[256]; + std::snprintf(label, sizeof(label), "%s (%llu)", row.name.c_str(), + static_cast(count)); + ImGui::Checkbox(label, &row.selected); + ImGui::PopID(); + } + } + ImGui::EndChild(); + } + + if (ImGui::CollapsingHeader("Unique id patterns", ImGuiTreeNodeFlags_DefaultOpen)) { + ImGui::TextColored(kDim, "One glob per line: * matches any run of characters, ? exactly one."); + ImGui::InputTextMultiline("##unique_ids", unique_id_patterns_, sizeof(unique_id_patterns_), + ImVec2(-FLT_MIN, 70)); + } + + if (ImGui::CollapsingHeader("Property rules", ImGuiTreeNodeFlags_DefaultOpen)) { + ImGui::TextColored(kDim, "Match one scalar field (bool/int/float/double/string) by its text value. " + "Leave the struct empty for any struct with that field."); + int remove_index = -1; + for (size_t i = 0; i < property_rules_.size(); ++i) { + PropertyRow& row = property_rules_[i]; + ImGui::PushID(static_cast(i) + 1000); + ImGui::SetNextItemWidth(170); + ImGui::InputTextWithHint("##struct", "struct (any)", row.struct_name, sizeof(row.struct_name)); + ImGui::SameLine(); + ImGui::SetNextItemWidth(150); + ImGui::InputTextWithHint("##field", "field", row.field_name, sizeof(row.field_name)); + ImGui::SameLine(); + ImGui::SetNextItemWidth(-40); + ImGui::InputTextWithHint("##value", "value glob", row.pattern, sizeof(row.pattern)); + ImGui::SameLine(); + if (ImGui::SmallButton("X")) { + remove_index = static_cast(i); + } + ImGui::PopID(); + } + if (remove_index >= 0) { + property_rules_.erase(property_rules_.begin() + remove_index); + } + if (ImGui::SmallButton("Add property rule")) { + property_rules_.emplace_back(); + } + } + + ImGui::Spacing(); + ImGui::Checkbox("Case-insensitive patterns", &case_insensitive_); + ImGui::SameLine(); + ImGui::Checkbox("Prune timeline events of dropped entities", &prune_events_); +} + +void FilterEntitiesWindow::DrawPreview(const VTX::ReplayFilterSpec& spec) { + const std::string key = SpecSignature(spec) + "#" + std::to_string(session_->GetCurrentFrame()); + if (key != preview_.key) { + RefreshPreview(spec, key); + } + + if (spec.rules.empty()) { + ImGui::TextColored(kWarn, "Add at least one rule: tick a bucket or struct, enter a unique id pattern or add a " + "property rule."); + return; + } + if (!preview_.error.empty()) { + ImGui::TextColored(kErr, "Rules cannot be resolved:"); + ImGui::TextWrapped("%s", preview_.error.c_str()); + return; + } + ImGui::Text("Preview on frame %d: keeps %llu of %llu entities, drops %llu", preview_.frame_index, + static_cast(preview_.kept), static_cast(preview_.seen), + static_cast(preview_.seen - preview_.kept)); + for (const auto& [name, counts] : preview_.buckets) { + ImGui::TextColored(kDim, " %s: %llu of %llu kept", name.c_str(), static_cast(counts.first), + static_cast(counts.second)); + } + ImGui::TextColored(kDim, "The rewrite applies the same rules to every frame."); +} + +void FilterEntitiesWindow::DrawRunControls(const VTX::ReplayFilterSpec& spec) { + const bool busy = phase_ == Phase::Running; + const bool can_run = !busy && !spec.rules.empty() && preview_.error.empty(); + + ImGui::BeginDisabled(!can_run); + if (ImGui::Button("Filter && Save As...", ImVec2(170, 0))) { + namespace fs = std::filesystem; + const fs::path source(session_->current_file_path_); + const std::string default_name = source.stem().string() + "_filtered.vtx"; + auto dialog = pfd::save_file("Save filtered replay", (source.parent_path() / default_name).string(), + {"VTX Files (.vtx)", "*.vtx", "All Files", "*"}); + std::string dest = dialog.result(); + if (!dest.empty()) { + if (fs::path(dest).extension().empty()) { + dest += ".vtx"; + } + StartFilter(spec, dest); + } + } + ImGui::EndDisabled(); + + if (busy) { + ImGui::SameLine(); + if (ImGui::Button("Cancel") && progress_) { + progress_->cancel.store(true); + } + } + if (phase_ == Phase::Done) { + ImGui::SameLine(); + if (ImGui::Button("Reset")) { + phase_ = Phase::Idle; + outcome_ = VTX::ReplayFilterResult {}; + outcome_dest_.clear(); + outcome_logged_ = false; + } + } + + if (busy && progress_) { + const int32_t done = progress_->done.load(); + const int32_t total = progress_->total.load(); + const float fraction = total > 0 ? static_cast(done) / static_cast(total) : 0.0f; + char overlay[64]; + std::snprintf(overlay, sizeof(overlay), "%d / %d chunks", done, total); + ImGui::ProgressBar(fraction, ImVec2(-FLT_MIN, 0), overlay); + ImGui::TextColored(kDim, "Reading every chunk, dropping entities and rewriting it..."); + } +} + +void FilterEntitiesWindow::DrawOutcome() { + if (phase_ != Phase::Done) { + return; + } + ImGui::Spacing(); + if (!outcome_.ok()) { + ImGui::TextColored(kErr, "Filter FAILED"); + ImGui::TextWrapped("%s", outcome_.error.c_str()); + return; + } + ImGui::TextColored(kOk, "Filter SUCCEEDED"); + ImGui::TextWrapped("Wrote %s", outcome_dest_.c_str()); + ImGui::Text("%d frames in %d chunks, %.2f s", outcome_.total_frames, outcome_.chunks_rewritten, + outcome_.elapsed_seconds); + ImGui::Text("Entities: %llu kept, %llu dropped (of %llu)", static_cast(outcome_.entities_kept), + static_cast(outcome_.entities_dropped), + static_cast(outcome_.entities_seen)); + if (outcome_.events_kept > 0 || outcome_.events_dropped > 0) { + ImGui::Text("Timeline events: %llu kept, %llu dropped", static_cast(outcome_.events_kept), + static_cast(outcome_.events_dropped)); + } + ImGui::Text("Size: %s -> %s", FormatBytes(outcome_.source_bytes).c_str(), + FormatBytes(outcome_.output_bytes).c_str()); + if (!outcome_.dropped_by_bucket.empty()) { + ImGui::TextColored(kDim, "Dropped by bucket:"); + for (const auto& [name, count] : outcome_.dropped_by_bucket) { + ImGui::TextColored(kDim, " %s: %llu", name.c_str(), static_cast(count)); + } + } + if (!outcome_.dropped_by_struct.empty()) { + ImGui::TextColored(kDim, "Dropped by struct:"); + for (const auto& [name, count] : outcome_.dropped_by_struct) { + ImGui::TextColored(kDim, " %s: %llu", name.c_str(), static_cast(count)); + } + } + ImGui::TextColored(kDim, "The new file opens like any other replay: same frames, same timestamps, same schema."); +}