From db45d07a04d4a4c1d7e31c41214ed539035bede2 Mon Sep 17 00:00:00 2001 From: onur-zenos Date: Wed, 9 Sep 2026 18:38:24 +0200 Subject: [PATCH 1/4] feat(writer): FilterReplayFile rewrites a replay without selected entities; inspector File > Filter Entities - VTX::FilterReplayFile(src, dst, spec, progress): drop (blacklist) or keep (whitelist) entities by unique id glob, struct name glob or scalar property value (struct.field resolved through the file's schema; empty struct = any struct with that field). Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums) and the footer keeps total_frames, duration and the per-frame time table; only the seek table changes. Timeline events are carried through the new SessionFooter::events (both policies) and pruned to entities that still exist. Progress callback with cancel; ReplayFilterResult reports kept/dropped, per-struct drops, event counts and sizes. ReplayFilterMatcher / GlobMatch expose the rule engine for previews. vtx_writer links vtx_reader privately; the entry point is compiled only when VTX_BUILD_READER is on. - tools/inspector: File > Filter Entities window -- struct checklist with current-frame counts, unique id globs, property rules, case-insensitive and event-pruning toggles, live preview on the current frame, worker thread with chunk progress and Cancel, result summary. - reader: GetFooter() now carries chunk_index[i].checksum on both backends (was always 0) and, for FlatBuffers, each event's entity_unique_id and location. - tests: ReplayFilterTest (both backends), ReplayFilterGlob, ReplayFilterMatcher. Docs: SDK_API.md section, CHANGELOG. --- CHANGELOG.md | 10 + docs/SDK_API.md | 25 + sdk/include/vtx/common/vtx_types.h | 1 + .../vtx/writer/core/vtx_replay_filter.h | 173 ++++ .../formatters/flatbuffer_reader_policy.cpp | 7 + .../formatters/protobuff_reader_policy.cpp | 1 + .../serialization/flatbuffers_to_vtx.cpp | 1 + sdk/src/vtx_writer/CMakeLists.txt | 10 + .../src/vtx/writer/core/vtx_replay_filter.cpp | 690 ++++++++++++++++ .../formatters/flatbuffers_vtx_policy.cpp | 15 +- .../formatters/protobuff_vtx_policy.cpp | 15 + tests/CMakeLists.txt | 1 + tests/writer/test_replay_filter.cpp | 775 ++++++++++++++++++ tools/inspector/include/inspector_layout.h | 4 + .../include/windows/filter_entities_window.h | 101 +++ tools/inspector/src/inspector_layout.cpp | 18 + .../src/windows/filter_entities_window.cpp | 442 ++++++++++ 17 files changed, 2288 insertions(+), 1 deletion(-) create mode 100644 sdk/include/vtx/writer/core/vtx_replay_filter.h create mode 100644 sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp create mode 100644 tests/writer/test_replay_filter.cpp create mode 100644 tools/inspector/include/windows/filter_entities_window.h create mode 100644 tools/inspector/src/windows/filter_entities_window.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 477da35..b8e1759 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **writer**: **`FilterReplayFile`** (`vtx/writer/core/vtx_replay_filter.h`) -- writes a copy of a replay with entities dropped (blacklist) or exclusively kept (whitelist) by **unique id glob**, **struct name glob** or **scalar property value** (`struct.field` resolved through the file's schema; empty struct = any struct with that field); rules combine as a union. Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums), and the footer is rebuilt with the same `total_frames`, duration and per-frame time table -- only the seek table changes. Timeline events are carried over and (by default) pruned to entities that still exist. Progress callback with cancel (the partial destination is removed); `ReplayFilterResult` reports kept/dropped counts, per-struct drops, event counts and sizes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for previews on in-memory frames. Both backends, output keeps the source format. `vtx_writer` now links `vtx_reader` privately; the entry point is compiled only when `VTX_BUILD_READER` is on. Covered by `ReplayFilterTest.*` (both backends: struct / unique id / property / numeric / case-insensitive rules, keep-nothing, events carried + pruned, error paths, cancel) plus `ReplayFilterGlob` / `ReplayFilterMatcher` unit tests +- **writer/serialization**: `SessionFooter::events` -- both footer serializers now write timeline events when the pointer is set. The writer facade still writes none; the replay filter uses it to carry a source file's events into the rewritten footer +- **tools/inspector**: **File > Filter Entities** -- rewrite the loaded replay into a new `.vtx` without the selected entities (or with only them). Struct checklist with per-struct counts for the current frame, unique id globs (one per line), property rules (struct / field / value glob), case-insensitive and event-pruning toggles; a live preview shows how many entities of the current frame the rules keep per bucket and reports unresolvable rules before anything runs. The rewrite runs on a worker thread (`VTX::FilterReplayFile`) with a chunk progress bar and Cancel, then reports frames, kept/dropped entities, per-struct drops, event counts and sizes + +### Fixed + +- **reader**: `GetFooter().chunk_index[i].checksum` was always 0 on both backends -- the footer-to-native conversions copied every seek-table field except the xxHash64 checksum (the internal seek table used for chunk loading did carry it). Visible as `checksum: 0` for every chunk in the CLI's `chunks` command, and the inspector's cut copied 0 into the seek-table entries of verbatim chunks. The FlatBuffers footer conversion also dropped each timeline event's `entity_unique_id` and `location`. All three conversions now carry the fields; `ReplayFilterTest.DropByStructName` asserts the footer checksums of a freshly written file match the bytes on disk and `ReplayFilterTest.EventsCarriedAndPruned` reads events back with their entity id and location on both backends + ## [0.5.0] - 2026-07-27 ### Added diff --git a/docs/SDK_API.md b/docs/SDK_API.md index 6b38d62..3777aa8 100644 --- a/docs/SDK_API.md +++ b/docs/SDK_API.md @@ -298,6 +298,31 @@ Durability knobs on the file sink (`ChunkedFileSink::Config`; facade users curre | `enable_recovery_journal` | `true` | Maintain the `.recovery` sidecar. Opting out removes any stale sidecar at session start; a crash then leaves an unrecoverable (footerless) file. | | `journal_compact_threshold_bytes` | `0` (64 MB) | How many superseded journal bytes accrue before the sidecar is compacted (rewritten via an atomic rename). | +### Filtering entities into a new replay + +`FilterReplayFile` writes a copy of a replay with some entities removed -- a blacklist (`Drop`: entities matching any rule go) or a whitelist (`Keep`: only matching entities stay). Rules select by **unique id glob**, **schema struct name glob**, or the **string form of one scalar property**; several rules combine as a union. Every frame survives: the header block is copied byte for byte, each chunk is re-serialized with the surviving entities (same frame range, same per-chunk compression, fresh xxHash64 checksums), and the footer keeps `total_frames`, `duration_seconds` and the per-frame time table -- only the seek table is rebuilt. Timeline events ride along and are, by default, pruned to the entities that still appear somewhere in the output. + +```cpp +#include "vtx/writer/core/vtx_replay_filter.h" + +VTX::ReplayFilterSpec spec; +spec.mode = VTX::ReplayFilterMode::Drop; // or Keep (whitelist) +spec.rules.push_back(VTX::ReplayFilterRule::StructName("SystemHealth*")); // glob on the struct name +spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); // glob on the unique id +spec.rules.push_back(VTX::ReplayFilterRule::Property("Vehicles", "Team", "2")); // one scalar field, as text + +VTX::ReplayFilterResult r = VTX::FilterReplayFile("capture.vtx", "capture_filtered.vtx", spec, + [](int32_t done, int32_t total) { return true; }); // optional; false cancels +if (r.ok()) { + // r.total_frames, r.chunks_rewritten, r.entities_kept / r.entities_dropped, + // r.dropped_by_struct (struct name -> count), r.events_kept / r.events_dropped, r.output_bytes +} else { + // r.error -- the destination is removed on any failure or cancel +} +``` + +A `Property` rule with an empty struct name means "every struct declaring that field". Matchable field types are Bool (`true`/`false`), Int32, Int64, Float, Double (shortest round-trip decimal) and String; arrays, maps, nested structs and the compound value types cannot be matched, and unresolvable rules (unknown struct or field, empty pattern) fail before anything is written. Only top-level bucket entities are filtered -- nested containers inside a kept entity are never touched -- and a bucket that ends up empty stays in place, since bucket index is positional. The schema is never pruned: dropping every instance of a struct leaves the struct declared. Both backends are supported and the output keeps the source format. `ReplayFilterMatcher` exposes the same rule resolution for previews on in-memory frames (`Keeps(bucket, index)`, `Apply(bucket)`), and `GlobMatch` is the pattern primitive (`*` any run, `?` one character). Requires the reader module (`VTX_BUILD_READER`). + ### Sink performance observer To see where write time goes, attach a perf observer — it receives per-stage timings synchronously from the writer thread (implementations must be cheap; leave null for a zero-cost no-op): diff --git a/sdk/include/vtx/common/vtx_types.h b/sdk/include/vtx/common/vtx_types.h index 128fa70..040ce0e 100644 --- a/sdk/include/vtx/common/vtx_types.h +++ b/sdk/include/vtx/common/vtx_types.h @@ -1216,6 +1216,7 @@ namespace VTX { const std::vector* created_utc = nullptr; const std::vector* gaps = nullptr; const std::vector* segments = nullptr; + const std::vector* events = nullptr; ///< Optional timeline events; null or empty writes none. }; struct ChunkIndexData { diff --git a/sdk/include/vtx/writer/core/vtx_replay_filter.h b/sdk/include/vtx/writer/core/vtx_replay_filter.h new file mode 100644 index 0000000..5a23b71 --- /dev/null +++ b/sdk/include/vtx/writer/core/vtx_replay_filter.h @@ -0,0 +1,173 @@ +/** + * @file vtx_replay_filter.h + * @brief Write a copy of a .vtx replay with some entities removed. + * + * @details FilterReplayFile() rewrites an existing replay into a new file, keeping + * every frame but dropping (or keeping only) the entities selected by a small rule + * set: unique id globs, schema struct names, or the value of one scalar property. + * The header block (schema, uuid, name, recording timestamp, metadata) is copied + * byte for byte, every chunk is re-serialized with the surviving entities, and the + * footer is rebuilt with the same frame count, duration and per-frame time table; + * only the seek table (offsets, sizes, checksums) changes. Timeline events are + * carried over, optionally pruned to the entities that still exist. + * + * VTX::ReplayFilterSpec spec; + * spec.mode = VTX::ReplayFilterMode::Drop; + * spec.rules.push_back(VTX::ReplayFilterRule::StructName("SystemHealth*")); + * spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); + * const auto r = VTX::FilterReplayFile("in.vtx", "out.vtx", spec); + * if (!r.ok()) { ... r.error ... } + * + * Requires the reader module (VTX_BUILD_READER): the source replay is read through + * the standard reader, so the function is only compiled when vtx_reader is built. + * + * @author Zenos Interactive + */ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vtx/common/vtx_property_cache.h" +#include "vtx/common/vtx_types.h" + +namespace VTX { + + /// How the rule set selects the entities that survive. + enum class ReplayFilterMode : uint8_t { + Drop, ///< Blacklist: an entity matching any rule is removed. + Keep, ///< Whitelist: only entities matching at least one rule are kept. + }; + + enum class ReplayFilterRuleKind : uint8_t { + UniqueId, ///< Glob against the entity's unique id (Bucket::unique_ids). + StructName, ///< Glob against the schema struct name of the entity's type id. + Property, ///< Glob against the string form of one scalar property value. + }; + + /** + * @brief One selection rule. Patterns are globs: '*' matches any run of + * characters, '?' exactly one; everything else is literal. + * @details Property rules resolve `struct_name.field_name` through the file's + * schema; an empty struct_name means "every struct declaring that + * field". Supported field types: Bool ("true"/"false"), Int32, Int64, + * Float, Double (shortest round-trip decimal) and String. Arrays, maps, + * nested structs and the compound value types are not matchable. + */ + struct ReplayFilterRule { + ReplayFilterRuleKind kind = ReplayFilterRuleKind::UniqueId; + std::string pattern; ///< Glob matched against the id / struct name / value. + std::string struct_name; ///< Property only. Empty = any struct with that field. + std::string field_name; ///< Property only. + bool case_insensitive = false; ///< ASCII case-insensitive match. + + static ReplayFilterRule UniqueId(std::string pattern, bool case_insensitive = false); + static ReplayFilterRule StructName(std::string pattern, bool case_insensitive = false); + static ReplayFilterRule Property(std::string struct_name, std::string field_name, std::string value_pattern, + bool case_insensitive = false); + }; + + struct ReplayFilterSpec { + ReplayFilterMode mode = ReplayFilterMode::Drop; + std::vector rules; ///< Combined as a union: an entity matches if any rule matches. + bool prune_events = true; ///< Drop timeline events whose entity_unique_id no longer appears in any frame. + }; + + struct ReplayFilterResult { + std::string error; ///< Non-empty on failure (the destination is removed). + + int32_t total_frames = 0; ///< Frames in the output (same as the source). + int32_t chunks_rewritten = 0; ///< Chunks re-serialized (all of them). + uint64_t entities_seen = 0; ///< Top-level bucket entities visited, summed over all frames. + uint64_t entities_kept = 0; + uint64_t entities_dropped = 0; + uint64_t events_kept = 0; + uint64_t events_dropped = 0; + uint64_t source_bytes = 0; + uint64_t output_bytes = 0; + double elapsed_seconds = 0.0; + /// Dropped entity count per struct name (entities of an unknown type id are + /// listed as "type#"). + std::map dropped_by_struct; + + bool ok() const { return error.empty(); } + }; + + /// Called before the first chunk (0, total) and after every rewritten chunk; + /// return false to cancel (the partial destination is removed and the result + /// carries an error). + using ReplayFilterProgress = std::function; + + /// Glob matcher used by the rules ('*' any run, '?' one character). + bool GlobMatch(std::string_view pattern, std::string_view text, bool case_insensitive = false); + + /** + * @brief Resolves a rule set against a file's schema and evaluates entities. + * @details Built once per file; used by FilterReplayFile() and available to + * callers that want to preview a spec on in-memory frames. + */ + class ReplayFilterMatcher { + public: + ReplayFilterMatcher(const ReplayFilterSpec& spec, const PropertyAddressCache& cache); + + bool valid() const { return error_.empty(); } + const std::string& error() const { return error_; } + ReplayFilterMode mode() const { return mode_; } + + /// True if entity @p index of @p bucket matches at least one rule. + bool Matches(const Bucket& bucket, size_t index) const; + /// Applies the mode: true if the entity survives the filter. + bool Keeps(const Bucket& bucket, size_t index) const { + return Matches(bucket, index) == (mode_ == ReplayFilterMode::Keep); + } + + /// Struct name for a type id ("type#" when the schema does not know it). + std::string TypeName(int32_t type_id) const; + + /** + * @brief Removes every entity Keeps() rejects, in place. + * @param on_drop Optional; called with the bucket and the index of each + * dropped entity before it is removed. + * @return Number of removed entities. When anything was removed the bucket's + * type_ranges are cleared (serialization rebuilds them from + * entity_type_id). + */ + size_t Apply(Bucket& bucket, const std::function& on_drop = nullptr) const; + + private: + struct ResolvedRule { + ReplayFilterRuleKind kind = ReplayFilterRuleKind::UniqueId; + std::string pattern; + bool case_insensitive = false; + std::unordered_map slots; ///< Property: type id -> address. + }; + + bool RuleMatches(const ResolvedRule& rule, const Bucket& bucket, size_t index) const; + + ReplayFilterMode mode_ = ReplayFilterMode::Drop; + std::vector rules_; + std::unordered_map type_names_; + std::string error_; + }; + + /** + * @brief Writes @p dest_path as a copy of @p source_path with the entities + * selected by @p spec removed (Drop) or exclusively kept (Keep). + * @details The source is opened with the standard reader (both FlatBuffers and + * Protobuf files are supported; the output keeps the source format). + * Header bytes are copied verbatim, chunks keep their frame ranges and + * per-chunk compression, the footer keeps total_frames, duration and the + * per-frame time table. Nested containers inside a kept entity are never + * touched; buckets that end up empty stay in place (bucket index is + * positional). dest_path must differ from source_path. + */ + ReplayFilterResult FilterReplayFile(const std::string& source_path, const std::string& dest_path, + const ReplayFilterSpec& spec, const ReplayFilterProgress& progress = nullptr); + +} // namespace VTX diff --git a/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp b/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp index 33b9086..fb33b35 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/formatters/flatbuffer_reader_policy.cpp @@ -179,6 +179,7 @@ VTX::FileFooter VTX::FlatBuffersReaderPolicy::GetVTXFooter(const FooterType& fbs entry.end_frame = chunk->end_frame; entry.file_offset = chunk->file_offset; + entry.checksum = chunk->checksum; out.chunk_index.push_back(entry); } } @@ -190,6 +191,12 @@ VTX::FileFooter VTX::FlatBuffersReaderPolicy::GetVTXFooter(const FooterType& fbs out_ev.game_time = ev->game_time; out_ev.event_type = ev->event_type; out_ev.label = ev->label; + out_ev.entity_unique_id = ev->entity_unique_id; + if (ev->location) { + out_ev.location.x = ev->location->x(); + out_ev.location.y = ev->location->y(); + out_ev.location.z = ev->location->z(); + } out.events.push_back(out_ev); } } diff --git a/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp b/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp index 2473175..c67a2be 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/formatters/protobuff_reader_policy.cpp @@ -151,6 +151,7 @@ VTX::FileFooter VTX::ProtobufReaderPolicy::GetVTXFooter(const FooterType& pb_foo entry.end_frame = pb_chunk.end_frame(); entry.file_offset = pb_chunk.file_offset(); entry.chunk_size_bytes = pb_chunk.chunk_size_bytes(); + entry.checksum = pb_chunk.checksum(); out.chunk_index.push_back(entry); } diff --git a/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp b/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp index 78ca304..9d9abd4 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/serialization/flatbuffers_to_vtx.cpp @@ -254,6 +254,7 @@ void VTX::Serialization::FromFlat(const fbsvtx::ChunkIndexEntry* src, VTX::Chunk dst.end_frame = src->end_frame(); dst.file_offset = src->file_offset(); dst.chunk_size_bytes = src->chunk_size_bytes(); + dst.checksum = src->checksum(); } void VTX::Serialization::FromFlat(const fbsvtx::ReplayTimeData* src, VTX::ReplayTimeData& dst) { diff --git a/sdk/src/vtx_writer/CMakeLists.txt b/sdk/src/vtx_writer/CMakeLists.txt index 41b535e..a994b7c 100644 --- a/sdk/src/vtx_writer/CMakeLists.txt +++ b/sdk/src/vtx_writer/CMakeLists.txt @@ -16,6 +16,12 @@ file(GLOB_RECURSE SOURCES CONFIGURE_DEPENDS # 2. Target and deps # ============================================================================== # Library type (STATIC/SHARED) follows BUILD_SHARED_LIBS (VTX_BUILD_SHARED). +# vtx_replay_filter reads the source replay through the reader module, so it is +# only part of the writer when vtx_reader is built (VTX_BUILD_READER). +if(NOT VTX_BUILD_READER) + list(FILTER SOURCES EXCLUDE REGEX "vtx_replay_filter\\.cpp$") +endif() + add_library(vtx_writer ${SOURCES} ${HEADERS}) target_include_directories(vtx_writer @@ -42,6 +48,10 @@ target_link_libraries(vtx_writer # See note in vtx_reader/CMakeLists.txt -- Windows shared DLLs duplicate the # generated protobuf/flatbuffers objects into every module that references them. +if(VTX_BUILD_READER) + target_link_libraries(vtx_writer PRIVATE vtx_reader) # VTX::FilterReplayFile opens the source replay +endif() + if(BUILD_SHARED_LIBS AND WIN32) target_link_libraries(vtx_writer PRIVATE $) endif() diff --git a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp b/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp new file mode 100644 index 0000000..4c0d4ad --- /dev/null +++ b/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp @@ -0,0 +1,690 @@ +#include "vtx/writer/core/vtx_replay_filter.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers +#include "vtx_schema.pb.h" + +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" +#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" +#include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" + +namespace VTX { + + namespace { + + // Mirrors ChunkedFileSink::CompressIfBeneficial with the writer defaults: + // level 10, tiny payloads left raw, raw kept when zstd does not shrink it. + constexpr int kCompressionLevel = 10; + constexpr size_t kCompressMinBytes = 512; + // zstd frame magic, little-endian on disk: 28 B5 2F FD. + constexpr unsigned char kZstdMagic[4] = {0x28, 0xB5, 0x2F, 0xFD}; + + char ToLowerAscii(char c) { + return static_cast(std::tolower(static_cast(c))); + } + + bool CharEquals(char a, char b, bool case_insensitive) { + return case_insensitive ? ToLowerAscii(a) == ToLowerAscii(b) : a == b; + } + + template + std::string ShortestDecimal(T value) { + char buffer[64]; + const auto res = std::to_chars(buffer, buffer + sizeof(buffer), value); + if (res.ec != std::errc {}) { + return {}; + } + return std::string(buffer, res.ptr); + } + + bool IsMatchableField(const PropertyAddress& address) { + if (address.container_type != FieldContainerType::None || address.index < 0) { + return false; + } + switch (address.type_id) { + case FieldType::Bool: + case FieldType::Int32: + case FieldType::Int64: + case FieldType::Float: + case FieldType::Double: + case FieldType::String: + return true; + default: + return false; + } + } + + // String form of one scalar property; false when the slot is absent. + bool PropertyValueToString(const PropertyContainer& entity, const PropertyAddress& address, std::string& out) { + const size_t i = static_cast(address.index); + switch (address.type_id) { + case FieldType::Bool: + if (i >= entity.bool_properties.size()) + return false; + out = entity.bool_properties[i] ? "true" : "false"; + return true; + case FieldType::Int32: + if (i >= entity.int32_properties.size()) + return false; + out = std::to_string(entity.int32_properties[i]); + return true; + case FieldType::Int64: + if (i >= entity.int64_properties.size()) + return false; + out = std::to_string(entity.int64_properties[i]); + return true; + case FieldType::Float: + if (i >= entity.float_properties.size()) + return false; + out = ShortestDecimal(entity.float_properties[i]); + return true; + case FieldType::Double: + if (i >= entity.double_properties.size()) + return false; + out = ShortestDecimal(entity.double_properties[i]); + return true; + case FieldType::String: + if (i >= entity.string_properties.size()) + return false; + out = entity.string_properties[i]; + return true; + default: + return false; + } + } + + const char* KindName(ReplayFilterRuleKind kind) { + switch (kind) { + case ReplayFilterRuleKind::UniqueId: + return "unique id"; + case ReplayFilterRuleKind::StructName: + return "struct"; + case ReplayFilterRuleKind::Property: + return "property"; + } + return "rule"; + } + + // ------------------------------------------------------------------ + // File plumbing (mirrors ChunkedFileSink framing: [magic][u32 header] + // [u32 chunk]* [footer][u32][magic]; xxHash64 over the on-disk payload). + // ------------------------------------------------------------------ + + struct SourceLayout { + uint64_t file_size = 0; + uint64_t header_end = 0; ///< First byte after the header block. + uint64_t footer_offset = 0; + bool footer_compressed = false; + }; + + bool ReadBytesAt(std::ifstream& in, uint64_t offset, void* dst, size_t size) { + in.clear(); + in.seekg(static_cast(offset)); + return static_cast(in.read(static_cast(dst), static_cast(size))); + } + + bool IsZstdAt(std::ifstream& in, uint64_t offset, uint64_t available) { + unsigned char magic[4]; + if (available < sizeof(magic) || !ReadBytesAt(in, offset, magic, sizeof(magic))) { + return false; + } + return std::memcmp(magic, kZstdMagic, sizeof(magic)) == 0; + } + + bool ProbeSource(std::ifstream& in, const std::string& expected_magic, SourceLayout& layout, + std::string& error) { + in.clear(); + in.seekg(0, std::ios::end); + const std::streamoff end = in.tellg(); + if (end < 0) { + error = "Could not determine the source replay size."; + return false; + } + layout.file_size = static_cast(end); + const size_t magic_size = expected_magic.size(); + // magic + header size + footer size + trailing magic. + if (layout.file_size < magic_size * 2 + sizeof(uint32_t) * 2) { + error = "The source replay is too small to be a valid .vtx file."; + return false; + } + + std::string magic(magic_size, '\0'); + if (!ReadBytesAt(in, 0, magic.data(), magic_size) || magic != expected_magic) { + error = "The source replay does not start with the expected magic bytes."; + return false; + } + uint32_t header_size = 0; + if (!ReadBytesAt(in, magic_size, &header_size, sizeof(header_size))) { + error = "Could not read the header size of the source replay."; + return false; + } + layout.header_end = magic_size + sizeof(uint32_t) + header_size; + + std::string trailing(magic_size, '\0'); + if (!ReadBytesAt(in, layout.file_size - magic_size, trailing.data(), magic_size) || + trailing != expected_magic) { + error = "The source replay has no footer (incomplete recording?). Repair it first."; + return false; + } + uint32_t footer_size = 0; + if (!ReadBytesAt(in, layout.file_size - magic_size - sizeof(uint32_t), &footer_size, sizeof(footer_size))) { + error = "Could not read the footer size of the source replay."; + return false; + } + const uint64_t footer_block = static_cast(footer_size) + sizeof(uint32_t) + magic_size; + if (footer_block > layout.file_size || layout.file_size - footer_block < layout.header_end) { + error = "The source replay header and footer overlap; the file is corrupt."; + return false; + } + layout.footer_offset = layout.file_size - footer_block; + layout.footer_compressed = IsZstdAt(in, layout.footer_offset, footer_size); + return true; + } + + bool CopyRange(std::ifstream& in, std::ofstream& out, uint64_t offset, uint64_t bytes, std::string& error) { + constexpr size_t kBufferSize = 4 * 1024 * 1024; + std::vector buffer( + static_cast(std::min(std::max(bytes, 1), kBufferSize))); + in.clear(); + in.seekg(static_cast(offset)); + uint64_t remaining = bytes; + while (remaining > 0) { + const size_t step = static_cast(std::min(remaining, buffer.size())); + if (!in.read(buffer.data(), static_cast(step))) { + error = "Read failed at offset " + std::to_string(offset + (bytes - remaining)) + "."; + return false; + } + if (!out.write(buffer.data(), static_cast(step))) { + error = "Write failed while copying the header block."; + return false; + } + remaining -= step; + } + return true; + } + + std::string CompressIfBeneficial(std::string payload) { + if (payload.size() < kCompressMinBytes) { + return payload; + } + const size_t max_size = ZSTD_compressBound(payload.size()); + std::string compressed(max_size, '\0'); + const size_t compressed_size = + ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), kCompressionLevel); + if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) { + return payload; + } + compressed.resize(compressed_size); + return compressed; + } + + template + void RunFilter(const std::string& source_path, const std::string& dest_path, IVtxReaderFacade& reader, + const FileFooter& footer, const ReplayFilterMatcher& matcher, bool prune_events, + const ReplayFilterProgress& progress, ReplayFilterResult& result) { + namespace fs = std::filesystem; + std::string error; + const auto fail = [&](std::string message) { + result.error = std::move(message); + }; + + std::ifstream in(source_path, std::ios::binary); + if (!in) { + fail("Could not open the source replay: " + source_path); + return; + } + SourceLayout layout; + if (!ProbeSource(in, Policy::GetMagicBytes(), layout, error)) { + fail(error); + return; + } + + // Chunks in frame order. Frame ranges and per-chunk compression are kept + // exactly; only the bytes (and therefore offsets/sizes/checksums) change. + std::vector chunks = footer.chunk_index; + std::sort(chunks.begin(), chunks.end(), + [](const ChunkIndexEntry& a, const ChunkIndexEntry& b) { return a.start_frame < b.start_frame; }); + for (const ChunkIndexEntry& chunk : chunks) { + if (chunk.end_frame < chunk.start_frame || chunk.chunk_size_bytes <= sizeof(uint32_t) || + chunk.file_offset < layout.header_end || + chunk.file_offset + chunk.chunk_size_bytes > layout.footer_offset) { + fail("The source replay seek table is inconsistent with the file layout (chunk " + + std::to_string(chunk.chunk_index) + ")."); + return; + } + } + if (!chunks.empty() && chunks.front().file_offset != layout.header_end) { + fail("The source replay has an unexpected layout between its header and first chunk."); + return; + } + + std::ofstream out(dest_path, std::ios::binary | std::ios::trunc); + if (!out) { + fail("Could not create the output file: " + dest_path); + return; + } + const auto abort_output = [&](std::string message) { + out.close(); + std::error_code ec; + fs::remove(dest_path, ec); + fail(std::move(message)); + }; + + // 1) Header block, verbatim (schema, uuid, name, recording timestamp, + // metadata). Nothing in it depends on the entity payload. + if (!CopyRange(in, out, 0, layout.header_end, error)) { + abort_output(error); + return; + } + + const int32_t chunk_total = static_cast(chunks.size()); + if (progress && !progress(0, chunk_total)) { + abort_output("Filter cancelled."); + return; + } + + const bool track_survivors = prune_events && !footer.events.empty(); + std::unordered_set survivors; + + // 2) Every chunk re-serialized with the surviving entities. + std::vector seek_table; + seek_table.reserve(chunks.size()); + for (int32_t i = 0; i < chunk_total; ++i) { + const ChunkIndexEntry& chunk = chunks[static_cast(i)]; + std::vector> frames; + frames.reserve(static_cast(chunk.end_frame - chunk.start_frame + 1)); + + for (int32_t frame_index = chunk.start_frame; frame_index <= chunk.end_frame; ++frame_index) { + const Frame* source_frame = reader.GetFrameSync(frame_index); + if (!source_frame) { + abort_output("Could not read frame " + std::to_string(frame_index) + + " from the source replay."); + return; + } + Frame frame(*source_frame); + for (Bucket& bucket : frame.GetMutableBuckets()) { + const size_t before = bucket.entities.size(); + const size_t dropped = matcher.Apply(bucket, [&](const Bucket& b, size_t index) { + ++result.dropped_by_struct[matcher.TypeName(b.entities[index].entity_type_id)]; + }); + result.entities_seen += before; + result.entities_dropped += dropped; + result.entities_kept += before - dropped; + if (track_survivors) { + for (const std::string& id : bucket.unique_ids) { + survivors.insert(id); + } + } + } + frames.push_back(Policy::FromNative(std::move(frame))); + } + + const bool compress = + IsZstdAt(in, chunk.file_offset + sizeof(uint32_t), chunk.chunk_size_bytes - sizeof(uint32_t)); + std::string payload = Policy::SerializeChunk(frames, i, compress); + if (compress) { + payload = CompressIfBeneficial(std::move(payload)); + } + + ChunkIndexData entry; + entry.chunk_index = i; + entry.file_offset = static_cast(out.tellp()); + entry.chunk_size_bytes = static_cast(payload.size() + sizeof(uint32_t)); + entry.start_frame = chunk.start_frame; + entry.end_frame = chunk.end_frame; + entry.checksum = XXH3_64bits(payload.data(), payload.size()); + + const uint32_t payload_size = static_cast(payload.size()); + if (!out.write(reinterpret_cast(&payload_size), sizeof(payload_size)) || + !out.write(payload.data(), static_cast(payload.size()))) { + abort_output("Write failed for chunk " + std::to_string(i) + "."); + return; + } + seek_table.push_back(entry); + ++result.chunks_rewritten; + + if (progress && !progress(i + 1, chunk_total)) { + abort_output("Filter cancelled."); + return; + } + } + + // 3) Footer: same frames, same duration, same per-frame time table; + // fresh seek table; events carried over (pruned to surviving entities). + std::vector game_times(footer.times.game_time.begin(), footer.times.game_time.end()); + std::vector created_utc(footer.times.created_utc.begin(), footer.times.created_utc.end()); + std::vector gaps(footer.times.gaps.begin(), footer.times.gaps.end()); + std::vector segments(footer.times.segments.begin(), footer.times.segments.end()); + + std::vector events; + events.reserve(footer.events.size()); + for (const TimelineEvent& event : footer.events) { + if (track_survivors && !event.entity_unique_id.empty() && !survivors.contains(event.entity_unique_id)) { + ++result.events_dropped; + continue; + } + events.push_back(event); + ++result.events_kept; + } + + SessionFooter session_footer; + session_footer.total_frames = footer.total_frames; + session_footer.duration_seconds = footer.duration_seconds; + const bool has_times = !game_times.empty() || !created_utc.empty() || !gaps.empty() || !segments.empty(); + if (has_times) { + session_footer.game_times = &game_times; + session_footer.created_utc = &created_utc; + session_footer.gaps = &gaps; + session_footer.segments = &segments; + } + session_footer.events = &events; + + std::string footer_payload = Policy::SerializeFooter(seek_table, session_footer); + if (layout.footer_compressed) { + footer_payload = CompressIfBeneficial(std::move(footer_payload)); + } + const uint32_t footer_size = static_cast(footer_payload.size()); + const std::string magic = Policy::GetMagicBytes(); + if (!out.write(footer_payload.data(), static_cast(footer_payload.size())) || + !out.write(reinterpret_cast(&footer_size), sizeof(footer_size)) || + !out.write(magic.data(), static_cast(magic.size()))) { + abort_output("Write failed while finishing the footer."); + return; + } + out.flush(); + if (!out) { + abort_output("Flush failed for the output file."); + return; + } + out.close(); + + result.total_frames = footer.total_frames; + std::error_code ec; + result.output_bytes = fs::file_size(dest_path, ec); + } + + } // namespace + + // ---------------------------------------------------------------------- + // Rules + // ---------------------------------------------------------------------- + + ReplayFilterRule ReplayFilterRule::UniqueId(std::string pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::UniqueId; + rule.pattern = std::move(pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + ReplayFilterRule ReplayFilterRule::StructName(std::string pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::StructName; + rule.pattern = std::move(pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + ReplayFilterRule ReplayFilterRule::Property(std::string struct_name, std::string field_name, + std::string value_pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::Property; + rule.struct_name = std::move(struct_name); + rule.field_name = std::move(field_name); + rule.pattern = std::move(value_pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + + bool GlobMatch(std::string_view pattern, std::string_view text, bool case_insensitive) { + size_t p = 0; + size_t t = 0; + size_t star = std::string_view::npos; + size_t star_text = 0; + while (t < text.size()) { + if (p < pattern.size() && (pattern[p] == '?' || CharEquals(pattern[p], text[t], case_insensitive))) { + ++p; + ++t; + } else if (p < pattern.size() && pattern[p] == '*') { + star = p++; + star_text = t; + } else if (star != std::string_view::npos) { + p = star + 1; + t = ++star_text; + } else { + return false; + } + } + while (p < pattern.size() && pattern[p] == '*') { + ++p; + } + return p == pattern.size(); + } + + // ---------------------------------------------------------------------- + // Matcher + // ---------------------------------------------------------------------- + + ReplayFilterMatcher::ReplayFilterMatcher(const ReplayFilterSpec& spec, const PropertyAddressCache& cache) + : mode_(spec.mode) { + for (const auto& [type_id, struct_cache] : cache.structs) { + type_names_[type_id] = struct_cache.name; + } + if (spec.rules.empty()) { + error_ = "The filter has no rules; add at least one unique id, struct or property rule."; + return; + } + for (const ReplayFilterRule& rule : spec.rules) { + if (rule.pattern.empty()) { + error_ = std::string("A ") + KindName(rule.kind) + " rule has an empty pattern."; + return; + } + ResolvedRule resolved; + resolved.kind = rule.kind; + resolved.pattern = rule.pattern; + resolved.case_insensitive = rule.case_insensitive; + if (rule.kind == ReplayFilterRuleKind::Property) { + if (rule.field_name.empty()) { + error_ = "A property rule has no field name."; + return; + } + if (!rule.struct_name.empty()) { + const auto id_it = cache.name_to_id.find(rule.struct_name); + const auto struct_it = + id_it == cache.name_to_id.end() ? cache.structs.end() : cache.structs.find(id_it->second); + if (struct_it == cache.structs.end()) { + error_ = "Property rule: struct '" + rule.struct_name + "' is not in the schema."; + return; + } + const auto field_it = struct_it->second.properties.find(rule.field_name); + if (field_it == struct_it->second.properties.end()) { + error_ = + "Property rule: struct '" + rule.struct_name + "' has no field '" + rule.field_name + "'."; + return; + } + if (!IsMatchableField(field_it->second)) { + error_ = "Property rule: '" + rule.struct_name + "." + rule.field_name + + "' is not a scalar bool/int/float/double/string field."; + return; + } + resolved.slots[id_it->second] = field_it->second; + } else { + for (const auto& [type_id, struct_cache] : cache.structs) { + const auto field_it = struct_cache.properties.find(rule.field_name); + if (field_it != struct_cache.properties.end() && IsMatchableField(field_it->second)) { + resolved.slots[type_id] = field_it->second; + } + } + if (resolved.slots.empty()) { + error_ = "Property rule: no struct declares a scalar field named '" + rule.field_name + "'."; + return; + } + } + } + rules_.push_back(std::move(resolved)); + } + } + + std::string ReplayFilterMatcher::TypeName(int32_t type_id) const { + const auto it = type_names_.find(type_id); + if (it != type_names_.end()) { + return it->second; + } + return "type#" + std::to_string(type_id); + } + + bool ReplayFilterMatcher::RuleMatches(const ResolvedRule& rule, const Bucket& bucket, size_t index) const { + switch (rule.kind) { + case ReplayFilterRuleKind::UniqueId: { + const std::string_view id = + index < bucket.unique_ids.size() ? std::string_view(bucket.unique_ids[index]) : std::string_view(); + return GlobMatch(rule.pattern, id, rule.case_insensitive); + } + case ReplayFilterRuleKind::StructName: { + const auto it = type_names_.find(bucket.entities[index].entity_type_id); + if (it == type_names_.end()) { + return false; + } + return GlobMatch(rule.pattern, it->second, rule.case_insensitive); + } + case ReplayFilterRuleKind::Property: { + const PropertyContainer& entity = bucket.entities[index]; + const auto slot = rule.slots.find(entity.entity_type_id); + if (slot == rule.slots.end()) { + return false; + } + std::string value; + if (!PropertyValueToString(entity, slot->second, value)) { + return false; + } + return GlobMatch(rule.pattern, value, rule.case_insensitive); + } + } + return false; + } + + bool ReplayFilterMatcher::Matches(const Bucket& bucket, size_t index) const { + if (index >= bucket.entities.size()) { + return false; + } + for (const ResolvedRule& rule : rules_) { + if (RuleMatches(rule, bucket, index)) { + return true; + } + } + return false; + } + + size_t ReplayFilterMatcher::Apply(Bucket& bucket, const std::function& on_drop) const { + const size_t count = bucket.entities.size(); + size_t write = 0; + for (size_t read = 0; read < count; ++read) { + if (!Keeps(bucket, read)) { + if (on_drop) { + on_drop(bucket, read); + } + continue; + } + if (write != read) { + bucket.entities[write] = std::move(bucket.entities[read]); + if (read < bucket.unique_ids.size() && write < bucket.unique_ids.size()) { + bucket.unique_ids[write] = std::move(bucket.unique_ids[read]); + } + } + ++write; + } + if (write == count) { + return 0; + } + bucket.entities.resize(write); + if (bucket.unique_ids.size() > write) { + bucket.unique_ids.resize(write); + } + bucket.type_ranges.clear(); + return count - write; + } + + // ---------------------------------------------------------------------- + // Entry point + // ---------------------------------------------------------------------- + + ReplayFilterResult FilterReplayFile(const std::string& source_path, const std::string& dest_path, + const ReplayFilterSpec& spec, const ReplayFilterProgress& progress) { + namespace fs = std::filesystem; + const auto started = std::chrono::steady_clock::now(); + ReplayFilterResult result; + const auto finish = [&]() -> ReplayFilterResult& { + result.elapsed_seconds = std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + return result; + }; + const auto fail = [&](std::string message) -> ReplayFilterResult& { + result.error = std::move(message); + return finish(); + }; + + if (spec.rules.empty()) { + return fail("The filter has no rules; add at least one unique id, struct or property rule."); + } + if (source_path.empty() || dest_path.empty()) { + return fail("Source and destination paths are required."); + } + std::error_code ec; + if (!fs::is_regular_file(source_path, ec)) { + return fail("Source replay does not exist: " + source_path); + } + const fs::path source_abs = fs::weakly_canonical(fs::path(source_path), ec); + const fs::path dest_abs = fs::weakly_canonical(fs::path(dest_path), ec); + if (source_abs == dest_abs || (fs::exists(dest_path, ec) && fs::equivalent(source_path, dest_path, ec))) { + return fail("Destination must be a different file than the source."); + } + + ReaderContext context = OpenReplayFile(source_path); + if (!context.Loaded()) { + return fail("Could not open the source replay: " + context.GetError().message); + } + if (!context.WaitUntilReady()) { + return fail("The source replay failed to load: " + context.GetReadyError().message); + } + + const FileFooter footer = context->GetFooter(); + const PropertyAddressCache cache = context->GetPropertyAddressCache(); + const ReplayFilterMatcher matcher(spec, cache); + if (!matcher.valid()) { + return fail(matcher.error()); + } + result.source_bytes = fs::file_size(source_path, ec); + + switch (context.format) { + case VtxFormat::FlatBuffers: + RunFilter(source_path, dest_path, *context.reader, footer, matcher, spec.prune_events, + progress, result); + break; + case VtxFormat::Protobuf: + RunFilter(source_path, dest_path, *context.reader, footer, matcher, spec.prune_events, + progress, result); + break; + default: + return fail("Unknown replay format."); + } + return finish(); + } + +} // namespace VTX diff --git a/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp b/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp index e0bab92..0f875d8 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp +++ b/sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp @@ -118,7 +118,20 @@ std::string VTX::FlatBuffersVtxPolicy::SerializeFooter(const std::vector> {}); + std::vector> event_offsets; + if (footer_data.events) { + event_offsets.reserve(footer_data.events->size()); + for (const auto& ev : *footer_data.events) { + // Strings are always created (the reader dereferences them unconditionally). + const auto type_off = builder.CreateString(ev.event_type); + const auto label_off = builder.CreateString(ev.label); + const auto id_off = builder.CreateString(ev.entity_unique_id); + const fbsvtx::Vector location(ev.location.x, ev.location.y, ev.location.z); + event_offsets.push_back( + fbsvtx::CreateTimelineEvent(builder, ev.game_time, type_off, label_off, &location, id_off)); + } + } + auto events_vec = builder.CreateVector(event_offsets); auto footer_offset = fbsvtx::CreateFileFooter(builder, footer_data.total_frames, static_cast(footer_data.duration_seconds), diff --git a/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp b/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp index 7a038eb..f199ea7 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp +++ b/sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp @@ -126,6 +126,21 @@ std::string VTX::ProtobufVtxPolicy::SerializeFooter(const std::vectorReserve(static_cast(footerData.events->size())); + for (const auto& ev : *footerData.events) { + auto* pb_ev = footer_msg.add_events(); + pb_ev->set_game_time(ev.game_time); + pb_ev->set_event_type(ev.event_type); + pb_ev->set_label(ev.label); + auto* location = pb_ev->mutable_location(); + location->set_x(ev.location.x); + location->set_y(ev.location.y); + location->set_z(ev.location.z); + pb_ev->set_entity_unique_id(ev.entity_unique_id); + } + } + footer_msg.mutable_chunk_index()->Reserve(static_cast(seekTable.size())); for (const auto& entry : seekTable) { diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 328ec0f..302324c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -77,6 +77,7 @@ add_executable(vtx_tests writer/test_frame_post_processor.cpp writer/test_frame_finalization.cpp writer/test_crash_recovery.cpp + writer/test_replay_filter.cpp writer/test_network_sink.cpp writer/test_pipe_source.cpp writer/test_websocket_source.cpp diff --git a/tests/writer/test_replay_filter.cpp b/tests/writer/test_replay_filter.cpp new file mode 100644 index 0000000..f8f9f09 --- /dev/null +++ b/tests/writer/test_replay_filter.cpp @@ -0,0 +1,775 @@ +// Replay entity filter tests: FilterReplayFile() writes a copy of a replay with +// entities dropped (blacklist) or exclusively kept (whitelist) by unique id glob, +// struct name or property value. Every frame survives; the header is byte-identical; +// the footer keeps the frame count, duration and per-frame time table; the seek +// table is rebuilt and its checksums match the bytes on disk; timeline events are +// carried over and pruned to the entities that still exist. +// +// Runs as a TEST_P over both backends (FlatBuffers + Protobuf). + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers +#include "vtx_schema.pb.h" + +#include "vtx/common/readers/schema_reader/schema_registry.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" +#include "vtx/writer/core/vtx_replay_filter.h" +#include "vtx/writer/core/vtx_writer_facade.h" +#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" +#include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" + +#include "util/test_fixtures.h" + +namespace { + + constexpr int kFrames = 100; // 3 chunks: [0,39] [40,79] [80,99] + constexpr int kChunkFrames = 40; + constexpr float kFps = 60.0f; + constexpr int64_t kBaseUtc = 17'000'000'000'000'000LL; // unix-relative 100 ns ticks + constexpr int64_t kFrameTicks = 166'666; + + const char* FormatName(VTX::VtxFormat f) { + return f == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; + } + + // Fixture schema (tests/fixtures/test_schema.json), one bucket "entity": + // Player (0): strings [UniqueID, Name] int32 [Team, Score, Deaths] float [Health, Armor] + // vector [Position, Velocity] quat [Rotation] bool [IsAlive] + // Projectile (1): strings [UniqueID, OwnerID, Type] vector [Position, Velocity] float [Damage] + // MatchState (2): strings [UniqueID, Phase] int32 [ScoreTeam1, ScoreTeam2, Round] float [TimeRemaining] + constexpr int32_t kPlayer = 0; + constexpr int32_t kProjectile = 1; + constexpr int32_t kMatchState = 2; + + VTX::PropertyContainer MakePlayer(const std::string& id, const std::string& name, int team, int frame) { + VTX::PropertyContainer pc; + pc.entity_type_id = kPlayer; + pc.string_properties = {id, name}; + pc.int32_properties = {team, frame, frame / 10}; + pc.float_properties = {100.0f - float(frame % 50), 25.0f + float(team)}; + pc.vector_properties = {VTX::Vector {double(frame), double(team), 0.0}, VTX::Vector {1.0, 0.0, 0.0}}; + pc.quat_properties = {VTX::Quat {0.0f, 0.0f, 0.0f, 1.0f}}; + pc.bool_properties = {frame % 7 != 0}; + return pc; + } + + VTX::PropertyContainer MakeProjectile(const std::string& id, const std::string& owner, const std::string& type, + int frame) { + VTX::PropertyContainer pc; + pc.entity_type_id = kProjectile; + pc.string_properties = {id, owner, type}; + pc.vector_properties = {VTX::Vector {double(frame) * 2.0, 1.0, 2.0}, VTX::Vector {0.0, 5.0, 0.0}}; + pc.float_properties = {10.0f + float(frame % 3)}; + return pc; + } + + VTX::PropertyContainer MakeMatch(int frame) { + VTX::PropertyContainer pc; + pc.entity_type_id = kMatchState; + pc.string_properties = {"match", frame < 50 ? "warmup" : "live"}; + pc.int32_properties = {frame / 20, frame / 30, 1 + frame / 50}; + pc.float_properties = {300.0f - float(frame)}; + return pc; + } + + void Add(VTX::Bucket& bucket, const std::string& id, VTX::PropertyContainer pc) { + bucket.unique_ids.push_back(id); + bucket.entities.push_back(std::move(pc)); + } + + std::string ProjectileId(int frame) { + return "proj_" + std::to_string(frame % 3); + } + + // Two players, one projectile (id cycles proj_0..proj_2) and the match state. + VTX::Frame BuildFrame(int frame) { + VTX::Frame f; + auto& bucket = f.CreateBucket("entity"); + Add(bucket, "player_0", MakePlayer("player_0", "Alpha", 1, frame)); + Add(bucket, "player_1", MakePlayer("player_1", "Bravo", 2, frame)); + Add(bucket, ProjectileId(frame), + MakeProjectile(ProjectileId(frame), frame % 2 ? "player_1" : "player_0", frame % 2 ? "rocket" : "bullet", + frame)); + Add(bucket, "match", MakeMatch(frame)); + return f; + } + + struct Snapshot { + bool ok = false; + VTX::VtxFormat format = VTX::VtxFormat::Unknown; + VTX::FileHeader header; + VTX::ContextualSchema schema; + VTX::FileFooter footer; + std::vector frames; + }; + + Snapshot ReadReplay(const std::string& path) { + Snapshot s; + VTX::ReaderContext ctx = VTX::OpenReplayFile(path); + if (!ctx.Loaded()) { + ADD_FAILURE() << "open failed for " << path << ": " << ctx.GetError().message; + return s; + } + if (!ctx.WaitUntilReady()) { + ADD_FAILURE() << "load failed for " << path << ": " << ctx.GetReadyError().message; + return s; + } + s.format = ctx.format; + s.header = ctx->GetHeader(); + s.schema = ctx->GetContextualSchema(); + s.footer = ctx->GetFooter(); + const int32_t total = ctx->GetTotalFrames(); + s.frames.reserve(static_cast(total)); + for (int32_t i = 0; i < total; ++i) { + const VTX::Frame* frame = ctx->GetFrameSync(i); + if (!frame) { + ADD_FAILURE() << "frame " << i << " unreadable in " << path; + return s; + } + s.frames.push_back(*frame); + } + s.ok = true; + return s; + } + + void ExpectEntityEqual(const VTX::PropertyContainer& a, const VTX::PropertyContainer& b, const std::string& what) { + EXPECT_EQ(a.entity_type_id, b.entity_type_id) << what; + EXPECT_EQ(a.content_hash, b.content_hash) << what; + EXPECT_EQ(a.bool_properties, b.bool_properties) << what; + EXPECT_EQ(a.int32_properties, b.int32_properties) << what; + EXPECT_EQ(a.int64_properties, b.int64_properties) << what; + EXPECT_EQ(a.float_properties, b.float_properties) << what; + EXPECT_EQ(a.double_properties, b.double_properties) << what; + EXPECT_EQ(a.string_properties, b.string_properties) << what; + EXPECT_EQ(a.transform_properties, b.transform_properties) << what; + EXPECT_EQ(a.vector_properties, b.vector_properties) << what; + EXPECT_EQ(a.quat_properties, b.quat_properties) << what; + EXPECT_EQ(a.range_properties, b.range_properties) << what; + EXPECT_EQ(a.int32_arrays.data, b.int32_arrays.data) << what; + EXPECT_EQ(a.int32_arrays.offsets, b.int32_arrays.offsets) << what; + EXPECT_EQ(a.float_arrays.data, b.float_arrays.data) << what; + EXPECT_EQ(a.float_arrays.offsets, b.float_arrays.offsets) << what; + EXPECT_EQ(a.string_arrays.data, b.string_arrays.data) << what; + EXPECT_EQ(a.string_arrays.offsets, b.string_arrays.offsets) << what; + EXPECT_EQ(a.vector_arrays.data, b.vector_arrays.data) << what; + EXPECT_EQ(a.any_struct_properties.size(), b.any_struct_properties.size()) << what; + EXPECT_EQ(a.map_properties.size(), b.map_properties.size()) << what; + } + + // The output bucket must be exactly the source bucket minus the entities `dropped` + // rejects, in the original order, with every survivor unchanged. + template + void ExpectBucketFiltered(const VTX::Bucket& source, const VTX::Bucket& output, DropPredicate dropped, + const std::string& what) { + std::vector expected; + for (size_t i = 0; i < source.entities.size(); ++i) { + if (!dropped(source, i)) { + expected.push_back(i); + } + } + ASSERT_EQ(output.entities.size(), expected.size()) << what; + ASSERT_EQ(output.unique_ids.size(), expected.size()) << what; + for (size_t o = 0; o < expected.size(); ++o) { + const size_t s = expected[o]; + EXPECT_EQ(output.unique_ids[o], source.unique_ids[s]) << what << " entity " << o; + ExpectEntityEqual(source.entities[s], output.entities[o], what + " entity " + source.unique_ids[s]); + } + } + + template + void ExpectFramesFiltered(const Snapshot& source, const Snapshot& output, DropPredicate dropped) { + ASSERT_EQ(output.frames.size(), source.frames.size()); + for (size_t f = 0; f < source.frames.size(); ++f) { + const auto& sb = source.frames[f].GetBuckets(); + const auto& ob = output.frames[f].GetBuckets(); + ASSERT_EQ(ob.size(), sb.size()) << "frame " << f; + for (size_t b = 0; b < sb.size(); ++b) { + ExpectBucketFiltered(sb[b], ob[b], dropped, + "frame " + std::to_string(f) + " bucket " + std::to_string(b)); + } + } + } + + // Header bit-identical fields, same frame count/duration/time table, same chunk + // frame ranges. + void ExpectSameHeaderFooterFraming(const Snapshot& source, const Snapshot& output) { + EXPECT_EQ(output.format, source.format); + EXPECT_EQ(output.header.replay_uuid, source.header.replay_uuid); + EXPECT_EQ(output.header.replay_name, source.header.replay_name); + EXPECT_EQ(output.header.recorded_utc_timestamp, source.header.recorded_utc_timestamp); + EXPECT_EQ(output.header.custom_json_metadata, source.header.custom_json_metadata); + EXPECT_EQ(output.header.version.format_major, source.header.version.format_major); + EXPECT_EQ(output.header.version.format_minor, source.header.version.format_minor); + EXPECT_EQ(output.header.version.schema_version, source.header.version.schema_version); + EXPECT_EQ(output.schema.property_mapping, source.schema.property_mapping); + EXPECT_EQ(output.schema.data_identifier, source.schema.data_identifier); + + EXPECT_EQ(output.footer.total_frames, source.footer.total_frames); + EXPECT_FLOAT_EQ(output.footer.duration_seconds, source.footer.duration_seconds); + EXPECT_EQ(output.footer.times.game_time, source.footer.times.game_time); + EXPECT_EQ(output.footer.times.created_utc, source.footer.times.created_utc); + EXPECT_EQ(output.footer.times.gaps, source.footer.times.gaps); + EXPECT_EQ(output.footer.times.segments, source.footer.times.segments); + EXPECT_FALSE(source.footer.times.created_utc.empty()) << "fixture should stamp created_utc"; + + ASSERT_EQ(output.footer.chunk_index.size(), source.footer.chunk_index.size()); + for (size_t i = 0; i < source.footer.chunk_index.size(); ++i) { + EXPECT_EQ(output.footer.chunk_index[i].chunk_index, source.footer.chunk_index[i].chunk_index) << i; + EXPECT_EQ(output.footer.chunk_index[i].start_frame, source.footer.chunk_index[i].start_frame) << i; + EXPECT_EQ(output.footer.chunk_index[i].end_frame, source.footer.chunk_index[i].end_frame) << i; + } + } + + // Every seek-table entry must describe the bytes actually on disk: the length + // prefix equals chunk_size_bytes - 4 and the xxHash64 of the payload matches. + void ExpectSeekTableMatchesFile(const std::string& path, const VTX::FileFooter& footer) { + std::ifstream in(path, std::ios::binary); + ASSERT_TRUE(in) << path; + in.seekg(0, std::ios::end); + const uint64_t file_size = static_cast(in.tellg()); + ASSERT_FALSE(footer.chunk_index.empty()); + for (const VTX::ChunkIndexEntry& entry : footer.chunk_index) { + ASSERT_LE(entry.file_offset + entry.chunk_size_bytes, file_size) << "chunk " << entry.chunk_index; + in.seekg(static_cast(entry.file_offset)); + uint32_t payload_size = 0; + ASSERT_TRUE(in.read(reinterpret_cast(&payload_size), sizeof(payload_size))); + ASSERT_EQ(payload_size + sizeof(uint32_t), entry.chunk_size_bytes) << "chunk " << entry.chunk_index; + std::string payload(payload_size, '\0'); + ASSERT_TRUE(in.read(payload.data(), payload_size)); + EXPECT_EQ(XXH3_64bits(payload.data(), payload.size()), entry.checksum) << "chunk " << entry.chunk_index; + } + // Chunks are contiguous from the first chunk to the footer block. + for (size_t i = 1; i < footer.chunk_index.size(); ++i) { + EXPECT_EQ(footer.chunk_index[i].file_offset, + footer.chunk_index[i - 1].file_offset + footer.chunk_index[i - 1].chunk_size_bytes); + } + } + + VTX::TimelineEvent MakeEvent(float game_time, const std::string& type, const std::string& label, + const std::string& entity_id) { + VTX::TimelineEvent ev; + ev.game_time = game_time; + ev.event_type = type; + ev.label = label; + ev.location = VTX::Vector {1.0, 2.0, 3.0}; + ev.entity_unique_id = entity_id; + return ev; + } + + // The public writer never emits timeline events, so fabricate a source that has + // them: replace the file's footer block with one serialized through the same + // policy the sink uses, carrying the original seek table and time table plus the + // given events. + void RewriteFooterWithEvents(const std::string& path, VTX::VtxFormat format, + const std::vector& events) { + const Snapshot s = ReadReplay(path); + ASSERT_TRUE(s.ok); + + std::vector seek_table; + for (const VTX::ChunkIndexEntry& e : s.footer.chunk_index) { + VTX::ChunkIndexData d; + d.chunk_index = e.chunk_index; + d.file_offset = static_cast(e.file_offset); + d.chunk_size_bytes = e.chunk_size_bytes; + d.start_frame = e.start_frame; + d.end_frame = e.end_frame; + d.checksum = e.checksum; + seek_table.push_back(d); + } + const std::vector game_times(s.footer.times.game_time.begin(), s.footer.times.game_time.end()); + const std::vector created_utc(s.footer.times.created_utc.begin(), s.footer.times.created_utc.end()); + const std::vector gaps(s.footer.times.gaps.begin(), s.footer.times.gaps.end()); + const std::vector segments(s.footer.times.segments.begin(), s.footer.times.segments.end()); + + VTX::SessionFooter sf; + sf.total_frames = s.footer.total_frames; + sf.duration_seconds = s.footer.duration_seconds; + sf.game_times = &game_times; + sf.created_utc = &created_utc; + sf.gaps = &gaps; + sf.segments = &segments; + sf.events = &events; + + const bool flat = format == VTX::VtxFormat::FlatBuffers; + const std::string payload = flat ? VTX::FlatBuffersVtxPolicy::SerializeFooter(seek_table, sf) + : VTX::ProtobufVtxPolicy::SerializeFooter(seek_table, sf); + const std::string magic = + flat ? VTX::FlatBuffersVtxPolicy::GetMagicBytes() : VTX::ProtobufVtxPolicy::GetMagicBytes(); + + // Locate the existing footer block: [payload][u32 size][magic] at the tail. + uint64_t file_size = 0; + uint32_t old_footer_size = 0; + { + std::ifstream in(path, std::ios::binary); + ASSERT_TRUE(in); + in.seekg(0, std::ios::end); + file_size = static_cast(in.tellg()); + in.seekg(static_cast(file_size - magic.size() - sizeof(uint32_t))); + ASSERT_TRUE(in.read(reinterpret_cast(&old_footer_size), sizeof(old_footer_size))); + } + const uint64_t footer_offset = file_size - magic.size() - sizeof(uint32_t) - old_footer_size; + std::filesystem::resize_file(path, footer_offset); + + std::ofstream out(path, std::ios::binary | std::ios::app); + ASSERT_TRUE(out); + const uint32_t footer_size = static_cast(payload.size()); + out.write(payload.data(), static_cast(payload.size())); + out.write(reinterpret_cast(&footer_size), sizeof(footer_size)); + out.write(magic.data(), static_cast(magic.size())); + ASSERT_TRUE(out.good()); + } + +} // namespace + +class ReplayFilterTest : public ::testing::TestWithParam { +protected: + std::string Path(const std::string& suffix) const { + return VtxTest::OutputPath(std::string("replay_filter_") + FormatName(GetParam()) + "_" + suffix + ".vtx"); + } + + std::string WriteSource(const std::string& suffix) const { + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = Path(suffix + "_src"); + cfg.schema_json_path = VtxTest::FixturePath("test_schema.json"); + cfg.replay_name = "ReplayFilterTest"; + cfg.replay_uuid = "uuid-filter-" + suffix; + cfg.default_fps = kFps; + cfg.chunk_max_frames = kChunkFrames; + cfg.use_compression = true; + + auto writer = GetParam() == VTX::VtxFormat::FlatBuffers ? VTX::CreateFlatBuffersWriterFacade(cfg) + : VTX::CreateProtobufWriterFacade(cfg); + EXPECT_TRUE(writer); + if (!writer) { + return {}; + } + for (int i = 0; i < kFrames; ++i) { + auto frame = BuildFrame(i); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / kFps; + t.created_utc_time = kBaseUtc + int64_t(i) * kFrameTicks; + writer->RecordFrame(frame, t); + } + writer->Stop(); + return cfg.output_filepath; + } +}; + +// --------------------------------------------------------------------------- +// Blacklist by struct name: every Projectile goes, everything else is untouched, +// header/footer framing survives, the seek table describes the bytes on disk. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, DropByStructName) { + const std::string src = WriteSource("dropstruct"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropstruct_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Drop; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + + std::vector> ticks; + const auto r = VTX::FilterReplayFile(src, dst, spec, [&](int32_t done, int32_t total) { + ticks.emplace_back(done, total); + return true; + }); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, kFrames); + EXPECT_EQ(r.chunks_rewritten, 3); + EXPECT_EQ(r.entities_seen, 4u * kFrames); + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.entities_kept, 3u * kFrames); + ASSERT_EQ(r.dropped_by_struct.size(), 1u); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), 1u * kFrames); + EXPECT_EQ(r.events_kept, 0u); + EXPECT_EQ(r.events_dropped, 0u); + EXPECT_GT(r.output_bytes, 0u); + EXPECT_LE(r.output_bytes, r.source_bytes); + EXPECT_GE(r.elapsed_seconds, 0.0); + ASSERT_EQ(ticks.size(), 4u); // (0,3) then one per chunk + EXPECT_EQ(ticks.front(), std::make_pair(0, 3)); + EXPECT_EQ(ticks.back(), std::make_pair(3, 3)); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFiltered(source, output, + [](const VTX::Bucket& b, size_t i) { return b.entities[i].entity_type_id == kProjectile; }); + ExpectSeekTableMatchesFile(dst, output.footer); + // The reader exposes the seek-table checksums through GetFooter() for the + // source too (they used to be dropped in the footer conversion). + ExpectSeekTableMatchesFile(src, source.footer); +} + +// --------------------------------------------------------------------------- +// Whitelist by unique id glob: only the players remain. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, KeepByUniqueIdGlob) { + const std::string src = WriteSource("keepid"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("keepid_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Keep; + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("player_*")); + + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_kept, 2u * kFrames); + EXPECT_EQ(r.entities_dropped, 2u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), 1u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("MatchState"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFiltered(source, output, + [](const VTX::Bucket& b, size_t i) { return b.unique_ids[i].rfind("player_", 0) != 0; }); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 1u); + EXPECT_EQ(frame.GetBuckets()[0].unique_ids, (std::vector {"player_0", "player_1"})); + } + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// Property rules: on one struct, and on "any struct with that field". +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, DropByPropertyValue) { + const std::string src = WriteSource("dropprop"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropprop_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Name", "Bra*")); + // Empty struct = every struct that declares a scalar "UniqueID" (all three do). + spec.rules.push_back(VTX::ReplayFilterRule::Property("", "UniqueID", "proj_1")); + + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + int proj1_frames = 0; + for (int i = 0; i < kFrames; ++i) { + proj1_frames += (i % 3 == 1) ? 1 : 0; + } + EXPECT_EQ(r.entities_dropped, uint64_t(kFrames + proj1_frames)); + EXPECT_EQ(r.dropped_by_struct.at("Player"), uint64_t(kFrames)); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), uint64_t(proj1_frames)); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFiltered(source, output, [](const VTX::Bucket& b, size_t i) { + return b.unique_ids[i] == "player_1" || b.unique_ids[i] == "proj_1"; + }); + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// A numeric property, matched through its decimal string form. +TEST_P(ReplayFilterTest, DropByNumericProperty) { + const std::string src = WriteSource("dropnum"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropnum_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Team", "2")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectFramesFiltered(source, output, [](const VTX::Bucket& b, size_t i) { return b.unique_ids[i] == "player_1"; }); +} + +// --------------------------------------------------------------------------- +// Case-insensitive glob on the struct name. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, CaseInsensitiveStructGlob) { + const std::string src = WriteSource("cistruct"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("cistruct_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("MATCH*", /*case_insensitive=*/true)); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("MatchState"), 1u * kFrames); + + // Case-sensitive by default: the same pattern matches nothing. + VTX::ReplayFilterSpec strict; + strict.rules.push_back(VTX::ReplayFilterRule::StructName("MATCH*")); + const auto r2 = VTX::FilterReplayFile(src, Path("cistruct_strict_out"), strict); + ASSERT_TRUE(r2.ok()) << r2.error; + EXPECT_EQ(r2.entities_dropped, 0u); + EXPECT_EQ(r2.entities_kept, 4u * kFrames); +} + +// --------------------------------------------------------------------------- +// A whitelist that matches nothing still writes every frame, with empty buckets. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, KeepNothingLeavesEmptyBuckets) { + const std::string src = WriteSource("keepnone"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("keepnone_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Keep; + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("nothing_has_this_id")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_kept, 0u); + EXPECT_EQ(r.entities_dropped, 4u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ASSERT_EQ(output.frames.size(), size_t(kFrames)); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 1u); + EXPECT_TRUE(frame.GetBuckets()[0].entities.empty()); + EXPECT_TRUE(frame.GetBuckets()[0].unique_ids.empty()); + } + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// Timeline events ride along and are pruned to the entities that still exist. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, EventsCarriedAndPruned) { + const std::string src = WriteSource("events"); + ASSERT_FALSE(src.empty()); + const std::vector events = { + MakeEvent(0.5f, "Fire", "proj_0 launched", "proj_0"), + MakeEvent(1.0f, "Spawn", "player_0 spawned", "player_0"), + MakeEvent(1.5f, "Round", "round started", ""), + }; + RewriteFooterWithEvents(src, GetParam(), events); + { + const Snapshot s = ReadReplay(src); + ASSERT_TRUE(s.ok); + ASSERT_EQ(s.footer.events.size(), 3u) << "fixture should now carry events"; + EXPECT_EQ(s.footer.total_frames, kFrames); + } + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + spec.prune_events = true; + const std::string pruned = Path("events_pruned"); + const auto r = VTX::FilterReplayFile(src, pruned, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.events_kept, 2u); + EXPECT_EQ(r.events_dropped, 1u); + { + const Snapshot out = ReadReplay(pruned); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.footer.events.size(), 2u); + EXPECT_EQ(out.footer.events[0].entity_unique_id, "player_0"); + EXPECT_EQ(out.footer.events[0].event_type, "Spawn"); + EXPECT_EQ(out.footer.events[0].label, "player_0 spawned"); + EXPECT_FLOAT_EQ(out.footer.events[0].game_time, 1.0f); + EXPECT_EQ(out.footer.events[0].location, (VTX::Vector {1.0, 2.0, 3.0})); + EXPECT_EQ(out.footer.events[1].entity_unique_id, ""); + EXPECT_EQ(out.footer.events[1].label, "round started"); + EXPECT_EQ(out.footer.total_frames, kFrames); + } + + spec.prune_events = false; + const std::string carried = Path("events_carried"); + const auto r2 = VTX::FilterReplayFile(src, carried, spec); + ASSERT_TRUE(r2.ok()) << r2.error; + EXPECT_EQ(r2.events_kept, 3u); + EXPECT_EQ(r2.events_dropped, 0u); + const Snapshot out = ReadReplay(carried); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.footer.events.size(), 3u); + EXPECT_EQ(out.footer.events[0].entity_unique_id, "proj_0"); +} + +// --------------------------------------------------------------------------- +// Failure paths report an error and never leave a destination behind. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, ErrorsAreReported) { + const std::string src = WriteSource("errors"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("errors_out"); + namespace fs = std::filesystem; + + { + VTX::ReplayFilterSpec empty; + const auto r = VTX::FilterReplayFile(src, dst, empty); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("no rules"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Nope", "X", "*")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("not in the schema"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Nope", "*")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("has no field"), std::string::npos) << r.error; + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Property("Player", "Position", "*")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("not a scalar"), std::string::npos) << r.error; + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("empty pattern"), std::string::npos) << r.error; + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + const auto r = VTX::FilterReplayFile(src, src, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("different file"), std::string::npos) << r.error; + EXPECT_TRUE(fs::exists(src)); + } + { + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + const auto r = VTX::FilterReplayFile(Path("does_not_exist"), dst, spec); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("does not exist"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } +} + +TEST_P(ReplayFilterTest, CancelRemovesDestination) { + const std::string src = WriteSource("cancel"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("cancel_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + const auto r = VTX::FilterReplayFile(src, dst, spec, [](int32_t done, int32_t) { return done < 1; }); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("cancel"), std::string::npos) << r.error; + EXPECT_EQ(r.chunks_rewritten, 1); + EXPECT_FALSE(std::filesystem::exists(dst)); +} + +INSTANTIATE_TEST_SUITE_P(Formats, ReplayFilterTest, + ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), + [](const ::testing::TestParamInfo& info) { return FormatName(info.param); }); + +// =========================================================================== +// Rule primitives, no file involved. +// =========================================================================== +TEST(ReplayFilterGlob, Semantics) { + EXPECT_TRUE(VTX::GlobMatch("*", "")); + EXPECT_TRUE(VTX::GlobMatch("*", "anything")); + EXPECT_TRUE(VTX::GlobMatch("", "")); + EXPECT_FALSE(VTX::GlobMatch("", "a")); + EXPECT_TRUE(VTX::GlobMatch("abc", "abc")); + EXPECT_FALSE(VTX::GlobMatch("abc", "abcd")); + EXPECT_TRUE(VTX::GlobMatch("a*", "abc")); + EXPECT_TRUE(VTX::GlobMatch("*c", "abc")); + EXPECT_TRUE(VTX::GlobMatch("a?c", "abc")); + EXPECT_FALSE(VTX::GlobMatch("a?c", "ac")); + EXPECT_TRUE(VTX::GlobMatch("*.vtx", "capture.vtx")); + EXPECT_TRUE(VTX::GlobMatch("*x*y*", "axbyc")); + EXPECT_TRUE(VTX::GlobMatch("a*b*c", "axxbxxc")); + EXPECT_FALSE(VTX::GlobMatch("a*b*c", "acb")); + EXPECT_TRUE(VTX::GlobMatch("**a", "ba")); + EXPECT_FALSE(VTX::GlobMatch("abc", "ABC")); + EXPECT_TRUE(VTX::GlobMatch("abc", "ABC", /*case_insensitive=*/true)); + EXPECT_TRUE(VTX::GlobMatch("SystemHealth*", "SystemHealthProcess")); + EXPECT_FALSE(VTX::GlobMatch("SystemHealth*", "PlayerState")); +} + +TEST(ReplayFilterMatcher, ApplyRemovesRejectedEntitiesInPlace) { + VTX::SchemaRegistry registry; + ASSERT_TRUE(registry.LoadFromJson(VtxTest::FixturePath("test_schema.json"))); + const VTX::PropertyAddressCache& cache = registry.GetPropertyCache(); + ASSERT_FALSE(cache.structs.empty()); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Drop; + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Projectile")); + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("player_1")); + const VTX::ReplayFilterMatcher matcher(spec, cache); + ASSERT_TRUE(matcher.valid()) << matcher.error(); + EXPECT_EQ(matcher.TypeName(kProjectile), "Projectile"); + EXPECT_EQ(matcher.TypeName(999), "type#999"); + + VTX::Frame frame = BuildFrame(5); // player_0, player_1, proj_2, match + VTX::Bucket& bucket = frame.GetBucket("entity"); + ASSERT_EQ(bucket.entities.size(), 4u); + EXPECT_TRUE(matcher.Keeps(bucket, 0)); + EXPECT_FALSE(matcher.Keeps(bucket, 1)); + EXPECT_FALSE(matcher.Keeps(bucket, 2)); + EXPECT_TRUE(matcher.Keeps(bucket, 3)); + EXPECT_FALSE(matcher.Matches(bucket, 42)); // out of range never matches + + std::vector dropped_ids; + const size_t dropped = + matcher.Apply(bucket, [&](const VTX::Bucket& b, size_t index) { dropped_ids.push_back(b.unique_ids[index]); }); + EXPECT_EQ(dropped, 2u); + EXPECT_EQ(dropped_ids, (std::vector {"player_1", "proj_2"})); + EXPECT_EQ(bucket.unique_ids, (std::vector {"player_0", "match"})); + ASSERT_EQ(bucket.entities.size(), 2u); + EXPECT_EQ(bucket.entities[0].entity_type_id, kPlayer); + EXPECT_EQ(bucket.entities[1].entity_type_id, kMatchState); + EXPECT_TRUE(bucket.type_ranges.empty()); + + // Nothing left to remove: the bucket is untouched and 0 is reported. + EXPECT_EQ(matcher.Apply(bucket), 0u); + EXPECT_EQ(bucket.entities.size(), 2u); + + // Keep mode with a property rule resolved against the schema. + VTX::ReplayFilterSpec keep; + keep.mode = VTX::ReplayFilterMode::Keep; + keep.rules.push_back(VTX::ReplayFilterRule::Property("MatchState", "Phase", "warm*")); + const VTX::ReplayFilterMatcher keeper(keep, cache); + ASSERT_TRUE(keeper.valid()) << keeper.error(); + VTX::Frame early = BuildFrame(5); // Phase = warmup + VTX::Frame late = BuildFrame(75); // Phase = live + EXPECT_EQ(keeper.Apply(early.GetBucket("entity")), 3u); + EXPECT_EQ(early.GetBucket("entity").unique_ids, (std::vector {"match"})); + EXPECT_EQ(keeper.Apply(late.GetBucket("entity")), 4u); + EXPECT_TRUE(late.GetBucket("entity").entities.empty()); + + // Unresolvable rules are reported, not silently ignored. + VTX::ReplayFilterSpec bad; + bad.rules.push_back(VTX::ReplayFilterRule::Property("", "NoSuchField", "*")); + const VTX::ReplayFilterMatcher invalid(bad, cache); + EXPECT_FALSE(invalid.valid()); + EXPECT_NE(invalid.error().find("NoSuchField"), std::string::npos) << invalid.error(); +} diff --git a/tools/inspector/include/inspector_layout.h b/tools/inspector/include/inspector_layout.h index 19405dd..31fc384 100644 --- a/tools/inspector/include/inspector_layout.h +++ b/tools/inspector/include/inspector_layout.h @@ -4,6 +4,7 @@ #include "gui/gui_layer.h" class CutReplayWindow; +class FilterEntitiesWindow; class GuiScaleController; class InspectorSession; class RepairReplayWindow; @@ -32,4 +33,7 @@ class InspectorLayout : public IGuiLayer { // File > Cut Replay: floating, operates on the loaded replay. At most one. std::shared_ptr cut_window_; + + // File > Filter Entities: floating, operates on the loaded replay. At most one. + std::shared_ptr filter_window_; }; diff --git a/tools/inspector/include/windows/filter_entities_window.h b/tools/inspector/include/windows/filter_entities_window.h new file mode 100644 index 0000000..537af16 --- /dev/null +++ b/tools/inspector/include/windows/filter_entities_window.h @@ -0,0 +1,101 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include "gui/gui_layer.h" +#include "vtx/common/vtx_property_cache.h" +#include "vtx/writer/core/vtx_replay_filter.h" + +class InspectorSession; + +// Floating window (File > Filter Entities...) that writes a new .vtx with entities +// of the loaded replay dropped (blacklist) or exclusively kept (whitelist) by +// struct name, unique id glob or property value. Every frame survives; only the +// entity payload changes. The rewrite is VTX::FilterReplayFile on a worker thread +// with live chunk progress and cancel. +// +// Implements IGuiLayer directly (rather than the shared ImGuiWindow base) for the +// same reason as CutReplayWindow: gui/gui_window.h's `class ImGuiWindow` clashes +// with Dear ImGui's `struct ImGuiWindow` in translation units that include +// imgui_internal.h (inspector_layout.cpp). +class FilterEntitiesWindow : public IGuiLayer { +public: + explicit FilterEntitiesWindow(std::shared_ptr session); + ~FilterEntitiesWindow() override; + + void OnUpdate() override {} + void OnRender() override; + + void SetOpen(bool open) { is_open_ = open; } + bool IsOpen() const { return is_open_; } + +private: + enum class Phase { Idle, Running, Done }; + + struct StructRow { + std::string name; + int32_t type_id = -1; + bool selected = false; + }; + + struct PropertyRow { + char struct_name[128] = {}; + char field_name[128] = {}; + char pattern[256] = {}; + }; + + // Shared with the worker thread. + struct Progress { + std::atomic done {0}; + std::atomic total {0}; + std::atomic cancel {false}; + }; + + // What the current rules do to the current frame (recomputed when either changes). + struct Preview { + std::string key; + std::string error; ///< Rule resolution error from the matcher, if any. + bool has_matcher = false; + int32_t frame_index = -1; + uint64_t seen = 0; + uint64_t kept = 0; + std::vector>> buckets; ///< name -> (kept, seen) + std::vector per_struct_seen; ///< indexed like structs_ + }; + + void DrawContent(); + void DrawRules(); + void DrawPreview(const VTX::ReplayFilterSpec& spec); + void DrawRunControls(const VTX::ReplayFilterSpec& spec); + void DrawOutcome(); + VTX::ReplayFilterSpec BuildSpec() const; + static std::string SpecSignature(const VTX::ReplayFilterSpec& spec); + void RefreshPreview(const VTX::ReplayFilterSpec& spec, const std::string& key); + void StartFilter(const VTX::ReplayFilterSpec& spec, const std::string& dest_path); + void JoinWorker(); + + std::shared_ptr session_; + VTX::PropertyAddressCache schema_cache_; + std::vector structs_; + std::vector property_rules_; + char unique_id_patterns_[4096] = {}; + int mode_ = 0; // 0 = drop matching (blacklist), 1 = keep only matching (whitelist) + bool case_insensitive_ = false; + bool prune_events_ = true; + + Preview preview_; + + Phase phase_ = Phase::Idle; + std::shared_ptr progress_; + std::future future_; + VTX::ReplayFilterResult outcome_; + std::string outcome_dest_; + bool outcome_logged_ = false; + bool is_open_ = true; +}; diff --git a/tools/inspector/src/inspector_layout.cpp b/tools/inspector/src/inspector_layout.cpp index a6b2754..3b08406 100644 --- a/tools/inspector/src/inspector_layout.cpp +++ b/tools/inspector/src/inspector_layout.cpp @@ -12,6 +12,7 @@ #include "inspector_session.h" #include "windows/analysis_window_factory.h" #include "windows/cut_replay_window.h" +#include "windows/filter_entities_window.h" #include "windows/repair_replay_window.h" namespace { @@ -270,6 +271,15 @@ void InspectorLayout::OnRender() { } } + // Rewrite the loaded replay into a new .vtx without the selected entities. + if (ImGui::MenuItem("Filter Entities...", nullptr, false, session_->HasLoadedReplay())) { + if (!filter_window_) { + filter_window_ = std::make_shared(session_); + } else { + filter_window_->SetOpen(true); + } + } + ImGui::Separator(); if (ImGui::MenuItem("Exit")) { @@ -350,4 +360,12 @@ void InspectorLayout::OnRender() { cut_window_.reset(); } } + + // Filter-entities window (floating, single instance), same lifecycle as cut. + if (filter_window_) { + filter_window_->OnRender(); + if (!filter_window_->IsOpen()) { + filter_window_.reset(); + } + } } diff --git a/tools/inspector/src/windows/filter_entities_window.cpp b/tools/inspector/src/windows/filter_entities_window.cpp new file mode 100644 index 0000000..4d6c445 --- /dev/null +++ b/tools/inspector/src/windows/filter_entities_window.cpp @@ -0,0 +1,442 @@ +#include "windows/filter_entities_window.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "gui/portable-file-dialogs.h" +#include "inspector_session.h" +#include "vtx/reader/core/vtx_reader_facade.h" + +namespace { + + const ImVec4 kOk {0.40f, 0.85f, 0.45f, 1.0f}; + const ImVec4 kWarn {0.95f, 0.80f, 0.30f, 1.0f}; + const ImVec4 kErr {0.95f, 0.45f, 0.45f, 1.0f}; + const ImVec4 kDim {0.65f, 0.65f, 0.65f, 1.0f}; + + std::string Trim(std::string s) { + const auto not_space = [](unsigned char c) { + return !std::isspace(c); + }; + s.erase(s.begin(), std::find_if(s.begin(), s.end(), not_space)); + s.erase(std::find_if(s.rbegin(), s.rend(), not_space).base(), s.end()); + return s; + } + + // Non-empty, trimmed lines of a text buffer. + std::vector Lines(const char* text) { + std::vector out; + std::istringstream in(text); + std::string line; + while (std::getline(in, line)) { + line = Trim(line); + if (!line.empty()) { + out.push_back(line); + } + } + return out; + } + + std::string FormatBytes(uint64_t bytes) { + char buffer[64]; + if (bytes >= 1024ull * 1024ull) { + std::snprintf(buffer, sizeof(buffer), "%.2f MB", static_cast(bytes) / (1024.0 * 1024.0)); + } else if (bytes >= 1024ull) { + std::snprintf(buffer, sizeof(buffer), "%.1f KB", static_cast(bytes) / 1024.0); + } else { + std::snprintf(buffer, sizeof(buffer), "%llu B", static_cast(bytes)); + } + return buffer; + } + +} // namespace + +FilterEntitiesWindow::FilterEntitiesWindow(std::shared_ptr session) + : session_(std::move(session)) { + if (session_ && session_->HasLoadedReplay()) { + if (VTX::IVtxReaderFacade* reader = session_->GetReader()) { + schema_cache_ = reader->GetPropertyAddressCache(); + for (const auto& [type_id, struct_cache] : schema_cache_.structs) { + structs_.push_back(StructRow {struct_cache.name, type_id, false}); + } + std::sort(structs_.begin(), structs_.end(), + [](const StructRow& a, const StructRow& b) { return a.name < b.name; }); + } + } +} + +FilterEntitiesWindow::~FilterEntitiesWindow() { + JoinWorker(); +} + +void FilterEntitiesWindow::JoinWorker() { + if (future_.valid()) { + if (progress_) { + progress_->cancel.store(true); + } + future_.wait(); + } +} + +void FilterEntitiesWindow::OnRender() { + if (!is_open_) { + return; + } + ImGui::SetNextWindowSize(ImVec2(640, 700), ImGuiCond_FirstUseEver); + if (ImGui::Begin("Filter Entities", &is_open_)) { + DrawContent(); + } + ImGui::End(); +} + +VTX::ReplayFilterSpec FilterEntitiesWindow::BuildSpec() const { + VTX::ReplayFilterSpec spec; + spec.mode = mode_ == 1 ? VTX::ReplayFilterMode::Keep : VTX::ReplayFilterMode::Drop; + spec.prune_events = prune_events_; + for (const StructRow& row : structs_) { + if (row.selected) { + spec.rules.push_back(VTX::ReplayFilterRule::StructName(row.name, case_insensitive_)); + } + } + for (const std::string& pattern : Lines(unique_id_patterns_)) { + spec.rules.push_back(VTX::ReplayFilterRule::UniqueId(pattern, case_insensitive_)); + } + for (const PropertyRow& row : property_rules_) { + const std::string field = Trim(row.field_name); + const std::string pattern = Trim(row.pattern); + if (field.empty() && pattern.empty()) { + continue; // untouched row + } + spec.rules.push_back(VTX::ReplayFilterRule::Property(Trim(row.struct_name), field, pattern, case_insensitive_)); + } + return spec; +} + +std::string FilterEntitiesWindow::SpecSignature(const VTX::ReplayFilterSpec& spec) { + std::string key = spec.mode == VTX::ReplayFilterMode::Keep ? "keep" : "drop"; + key += spec.prune_events ? "|prune" : "|carry"; + for (const VTX::ReplayFilterRule& rule : spec.rules) { + key += '|'; + key += std::to_string(static_cast(rule.kind)); + key += rule.case_insensitive ? "i:" : "c:"; + key += rule.struct_name; + key += '.'; + key += rule.field_name; + key += '='; + key += rule.pattern; + } + return key; +} + +void FilterEntitiesWindow::RefreshPreview(const VTX::ReplayFilterSpec& spec, const std::string& key) { + preview_ = Preview {}; + preview_.key = key; + preview_.per_struct_seen.assign(structs_.size(), 0); + + VTX::IVtxReaderFacade* reader = session_ ? session_->GetReader() : nullptr; + if (!reader) { + return; + } + preview_.frame_index = std::max(0, session_->GetCurrentFrame()); + const VTX::Frame* frame = reader->GetFrameSync(preview_.frame_index); + if (!frame) { + return; + } + + std::unordered_map struct_rows; + for (size_t i = 0; i < structs_.size(); ++i) { + struct_rows[structs_[i].type_id] = i; + } + + std::unique_ptr matcher; + if (!spec.rules.empty()) { + matcher = std::make_unique(spec, schema_cache_); + if (!matcher->valid()) { + preview_.error = matcher->error(); + matcher.reset(); + } + } + preview_.has_matcher = matcher != nullptr; + + const auto& buckets = frame->GetBuckets(); + for (size_t b = 0; b < buckets.size(); ++b) { + const VTX::Bucket& bucket = buckets[b]; + uint64_t kept = 0; + for (size_t i = 0; i < bucket.entities.size(); ++i) { + const auto row = struct_rows.find(bucket.entities[i].entity_type_id); + if (row != struct_rows.end()) { + ++preview_.per_struct_seen[row->second]; + } + if (matcher && matcher->Keeps(bucket, i)) { + ++kept; + } + } + std::string name = b < schema_cache_.bucket_names.size() ? schema_cache_.bucket_names[b] : std::string(); + if (name.empty()) { + name = "bucket " + std::to_string(b); + } + preview_.buckets.emplace_back(std::move(name), + std::make_pair(kept, static_cast(bucket.entities.size()))); + preview_.seen += bucket.entities.size(); + preview_.kept += kept; + } +} + +void FilterEntitiesWindow::StartFilter(const VTX::ReplayFilterSpec& spec, const std::string& dest_path) { + JoinWorker(); + phase_ = Phase::Running; + outcome_ = VTX::ReplayFilterResult {}; + outcome_dest_ = dest_path; + outcome_logged_ = false; + progress_ = std::make_shared(); + + // The worker touches only its by-value captures and the shared progress block. + const std::shared_ptr progress = progress_; + const std::string source = session_->current_file_path_; + future_ = std::async(std::launch::async, [source, dest_path, spec, progress]() { + return VTX::FilterReplayFile(source, dest_path, spec, [progress](int32_t done, int32_t total) { + progress->done.store(done); + progress->total.store(total); + return !progress->cancel.load(); + }); + }); +} + +void FilterEntitiesWindow::DrawContent() { + if (phase_ == Phase::Running && future_.valid() && + future_.wait_for(std::chrono::seconds(0)) == std::future_status::ready) { + outcome_ = future_.get(); + phase_ = Phase::Done; + if (!outcome_logged_ && session_) { + outcome_logged_ = true; + if (outcome_.ok()) { + session_->AddGuiInfoLog("Filter Entities: wrote " + outcome_dest_ + " (" + + std::to_string(outcome_.entities_kept) + " entities kept, " + + std::to_string(outcome_.entities_dropped) + " dropped)"); + } else { + session_->AddGuiErrorLog("Filter Entities failed: " + outcome_.error); + } + } + } + + if (!session_ || !session_->HasLoadedReplay()) { + ImGui::TextColored(kWarn, "Open a replay first."); + return; + } + + const std::filesystem::path source(session_->current_file_path_); + ImGui::TextWrapped("Source: %s", source.filename().string().c_str()); + ImGui::TextColored(kDim, + "%d frames, %d structs in the schema. Every frame is kept; only the entity payload changes.", + session_->GetTotalFrames(), static_cast(structs_.size())); + ImGui::Separator(); + + const bool busy = phase_ == Phase::Running; + ImGui::BeginDisabled(busy); + ImGui::TextUnformatted("Mode"); + ImGui::SameLine(); + ImGui::RadioButton("Drop matching (blacklist)", &mode_, 0); + ImGui::SameLine(); + ImGui::RadioButton("Keep only matching (whitelist)", &mode_, 1); + ImGui::Spacing(); + DrawRules(); + ImGui::EndDisabled(); + + const VTX::ReplayFilterSpec spec = BuildSpec(); + ImGui::Separator(); + DrawPreview(spec); + ImGui::Separator(); + DrawRunControls(spec); + DrawOutcome(); +} + +void FilterEntitiesWindow::DrawRules() { + if (ImGui::CollapsingHeader("Structs", ImGuiTreeNodeFlags_DefaultOpen)) { + if (ImGui::SmallButton("Select all")) { + for (StructRow& row : structs_) { + row.selected = true; + } + } + ImGui::SameLine(); + if (ImGui::SmallButton("Clear")) { + for (StructRow& row : structs_) { + row.selected = false; + } + } + ImGui::SameLine(); + ImGui::TextColored(kDim, "(count = entities of that struct in the current frame)"); + + const float row_height = ImGui::GetFrameHeightWithSpacing(); + const float height = + std::min(220.0f, row_height * static_cast(std::max(structs_.size(), 1)) + 8.0f); + if (ImGui::BeginChild("##structs", ImVec2(0, height), ImGuiChildFlags_Border)) { + for (size_t i = 0; i < structs_.size(); ++i) { + StructRow& row = structs_[i]; + ImGui::PushID(static_cast(i)); + const uint64_t count = i < preview_.per_struct_seen.size() ? preview_.per_struct_seen[i] : 0; + char label[256]; + std::snprintf(label, sizeof(label), "%s (%llu)", row.name.c_str(), + static_cast(count)); + ImGui::Checkbox(label, &row.selected); + ImGui::PopID(); + } + } + ImGui::EndChild(); + } + + if (ImGui::CollapsingHeader("Unique id patterns", ImGuiTreeNodeFlags_DefaultOpen)) { + ImGui::TextColored(kDim, "One glob per line: * matches any run of characters, ? exactly one."); + ImGui::InputTextMultiline("##unique_ids", unique_id_patterns_, sizeof(unique_id_patterns_), + ImVec2(-FLT_MIN, 70)); + } + + if (ImGui::CollapsingHeader("Property rules", ImGuiTreeNodeFlags_DefaultOpen)) { + ImGui::TextColored(kDim, "Match one scalar field (bool/int/float/double/string) by its text value. " + "Leave the struct empty for any struct with that field."); + int remove_index = -1; + for (size_t i = 0; i < property_rules_.size(); ++i) { + PropertyRow& row = property_rules_[i]; + ImGui::PushID(static_cast(i) + 1000); + ImGui::SetNextItemWidth(170); + ImGui::InputTextWithHint("##struct", "struct (any)", row.struct_name, sizeof(row.struct_name)); + ImGui::SameLine(); + ImGui::SetNextItemWidth(150); + ImGui::InputTextWithHint("##field", "field", row.field_name, sizeof(row.field_name)); + ImGui::SameLine(); + ImGui::SetNextItemWidth(-40); + ImGui::InputTextWithHint("##value", "value glob", row.pattern, sizeof(row.pattern)); + ImGui::SameLine(); + if (ImGui::SmallButton("X")) { + remove_index = static_cast(i); + } + ImGui::PopID(); + } + if (remove_index >= 0) { + property_rules_.erase(property_rules_.begin() + remove_index); + } + if (ImGui::SmallButton("Add property rule")) { + property_rules_.emplace_back(); + } + } + + ImGui::Spacing(); + ImGui::Checkbox("Case-insensitive patterns", &case_insensitive_); + ImGui::SameLine(); + ImGui::Checkbox("Prune timeline events of dropped entities", &prune_events_); +} + +void FilterEntitiesWindow::DrawPreview(const VTX::ReplayFilterSpec& spec) { + const std::string key = SpecSignature(spec) + "#" + std::to_string(session_->GetCurrentFrame()); + if (key != preview_.key) { + RefreshPreview(spec, key); + } + + if (spec.rules.empty()) { + ImGui::TextColored(kWarn, + "Add at least one rule: tick a struct, enter a unique id pattern or add a property rule."); + return; + } + if (!preview_.error.empty()) { + ImGui::TextColored(kErr, "Rules cannot be resolved:"); + ImGui::TextWrapped("%s", preview_.error.c_str()); + return; + } + ImGui::Text("Preview on frame %d: keeps %llu of %llu entities, drops %llu", preview_.frame_index, + static_cast(preview_.kept), static_cast(preview_.seen), + static_cast(preview_.seen - preview_.kept)); + for (const auto& [name, counts] : preview_.buckets) { + ImGui::TextColored(kDim, " %s: %llu of %llu kept", name.c_str(), static_cast(counts.first), + static_cast(counts.second)); + } + ImGui::TextColored(kDim, "The rewrite applies the same rules to every frame."); +} + +void FilterEntitiesWindow::DrawRunControls(const VTX::ReplayFilterSpec& spec) { + const bool busy = phase_ == Phase::Running; + const bool can_run = !busy && !spec.rules.empty() && preview_.error.empty(); + + ImGui::BeginDisabled(!can_run); + if (ImGui::Button("Filter && Save As...", ImVec2(170, 0))) { + namespace fs = std::filesystem; + const fs::path source(session_->current_file_path_); + const std::string default_name = source.stem().string() + "_filtered.vtx"; + auto dialog = pfd::save_file("Save filtered replay", (source.parent_path() / default_name).string(), + {"VTX Files (.vtx)", "*.vtx", "All Files", "*"}); + std::string dest = dialog.result(); + if (!dest.empty()) { + if (fs::path(dest).extension().empty()) { + dest += ".vtx"; + } + StartFilter(spec, dest); + } + } + ImGui::EndDisabled(); + + if (busy) { + ImGui::SameLine(); + if (ImGui::Button("Cancel") && progress_) { + progress_->cancel.store(true); + } + } + if (phase_ == Phase::Done) { + ImGui::SameLine(); + if (ImGui::Button("Reset")) { + phase_ = Phase::Idle; + outcome_ = VTX::ReplayFilterResult {}; + outcome_dest_.clear(); + outcome_logged_ = false; + } + } + + if (busy && progress_) { + const int32_t done = progress_->done.load(); + const int32_t total = progress_->total.load(); + const float fraction = total > 0 ? static_cast(done) / static_cast(total) : 0.0f; + char overlay[64]; + std::snprintf(overlay, sizeof(overlay), "%d / %d chunks", done, total); + ImGui::ProgressBar(fraction, ImVec2(-FLT_MIN, 0), overlay); + ImGui::TextColored(kDim, "Reading every chunk, dropping entities and rewriting it..."); + } +} + +void FilterEntitiesWindow::DrawOutcome() { + if (phase_ != Phase::Done) { + return; + } + ImGui::Spacing(); + if (!outcome_.ok()) { + ImGui::TextColored(kErr, "Filter FAILED"); + ImGui::TextWrapped("%s", outcome_.error.c_str()); + return; + } + ImGui::TextColored(kOk, "Filter SUCCEEDED"); + ImGui::TextWrapped("Wrote %s", outcome_dest_.c_str()); + ImGui::Text("%d frames in %d chunks, %.2f s", outcome_.total_frames, outcome_.chunks_rewritten, + outcome_.elapsed_seconds); + ImGui::Text("Entities: %llu kept, %llu dropped (of %llu)", static_cast(outcome_.entities_kept), + static_cast(outcome_.entities_dropped), + static_cast(outcome_.entities_seen)); + if (outcome_.events_kept > 0 || outcome_.events_dropped > 0) { + ImGui::Text("Timeline events: %llu kept, %llu dropped", static_cast(outcome_.events_kept), + static_cast(outcome_.events_dropped)); + } + ImGui::Text("Size: %s -> %s", FormatBytes(outcome_.source_bytes).c_str(), + FormatBytes(outcome_.output_bytes).c_str()); + if (!outcome_.dropped_by_struct.empty()) { + ImGui::TextColored(kDim, "Dropped by struct:"); + for (const auto& [name, count] : outcome_.dropped_by_struct) { + ImGui::TextColored(kDim, " %s: %llu", name.c_str(), static_cast(count)); + } + } + ImGui::TextColored(kDim, "The new file opens like any other replay: same frames, same timestamps, same schema."); +} From 69d4d5bb53dc1acfeba525d5358321a6c8b2199c Mon Sep 17 00:00:00 2001 From: onur-zenos Date: Wed, 9 Sep 2026 21:48:23 +0200 Subject: [PATCH 2/4] feat(writer): bucket rules for FilterReplayFile; bucket checklist in the inspector - ReplayFilterRule::Bucket(pattern): selects every entity of the buckets whose schema name matches the glob, or of the bucket at that index when the pattern is digits only (also for buckets the schema does not name). Dropping a bucket empties it in every frame; the slot stays because bucket index is positional. A bucket rule that matches no schema bucket is an error listing the available names. ReplayFilterMatcher queries now take the bucket index (Matches / Keeps / Apply) and expose BucketMatches / BucketName; ReplayFilterResult adds dropped_by_bucket. - tools/inspector: Filter Entities gets a Buckets checklist (index, name, current-frame count) above the struct list; the outcome lists drops per bucket. - tests: DropByBucketName, KeepBucketCombinesWithOtherRules, BucketByIndexAndGlob (both backends, three-bucket schema) and BucketRulesResolveAgainstSchemaBuckets. Docs and CHANGELOG updated. --- CHANGELOG.md | 4 +- docs/SDK_API.md | 9 +- .../vtx/writer/core/vtx_replay_filter.h | 62 +++-- .../src/vtx/writer/core/vtx_replay_filter.cpp | 101 +++++++- tests/writer/test_replay_filter.cpp | 234 +++++++++++++++++- .../include/windows/filter_entities_window.h | 8 + .../src/windows/filter_entities_window.cpp | 59 ++++- 7 files changed, 428 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b8e1759..0d4a5f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,9 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **writer**: **`FilterReplayFile`** (`vtx/writer/core/vtx_replay_filter.h`) -- writes a copy of a replay with entities dropped (blacklist) or exclusively kept (whitelist) by **unique id glob**, **struct name glob** or **scalar property value** (`struct.field` resolved through the file's schema; empty struct = any struct with that field); rules combine as a union. Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums), and the footer is rebuilt with the same `total_frames`, duration and per-frame time table -- only the seek table changes. Timeline events are carried over and (by default) pruned to entities that still exist. Progress callback with cancel (the partial destination is removed); `ReplayFilterResult` reports kept/dropped counts, per-struct drops, event counts and sizes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for previews on in-memory frames. Both backends, output keeps the source format. `vtx_writer` now links `vtx_reader` privately; the entry point is compiled only when `VTX_BUILD_READER` is on. Covered by `ReplayFilterTest.*` (both backends: struct / unique id / property / numeric / case-insensitive rules, keep-nothing, events carried + pruned, error paths, cancel) plus `ReplayFilterGlob` / `ReplayFilterMatcher` unit tests +- **writer**: **`FilterReplayFile`** (`vtx/writer/core/vtx_replay_filter.h`) -- writes a copy of a replay with entities dropped (blacklist) or exclusively kept (whitelist) by **bucket** (schema bucket name glob or bucket index; a dropped bucket is emptied in every frame and keeps its positional slot), **unique id glob**, **struct name glob** or **scalar property value** (`struct.field` resolved through the file's schema; empty struct = any struct with that field); rules combine as a union. Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums), and the footer is rebuilt with the same `total_frames`, duration and per-frame time table -- only the seek table changes. Timeline events are carried over and (by default) pruned to entities that still exist. Progress callback with cancel (the partial destination is removed); `ReplayFilterResult` reports kept/dropped counts, per-struct and per-bucket drops, event counts and sizes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for previews on in-memory frames. Both backends, output keeps the source format. `vtx_writer` now links `vtx_reader` privately; the entry point is compiled only when `VTX_BUILD_READER` is on. Covered by `ReplayFilterTest.*` (both backends: bucket / struct / unique id / property / numeric / case-insensitive rules, keep-nothing, events carried + pruned, error paths, cancel) plus `ReplayFilterGlob` / `ReplayFilterMatcher` unit tests - **writer/serialization**: `SessionFooter::events` -- both footer serializers now write timeline events when the pointer is set. The writer facade still writes none; the replay filter uses it to carry a source file's events into the rewritten footer -- **tools/inspector**: **File > Filter Entities** -- rewrite the loaded replay into a new `.vtx` without the selected entities (or with only them). Struct checklist with per-struct counts for the current frame, unique id globs (one per line), property rules (struct / field / value glob), case-insensitive and event-pruning toggles; a live preview shows how many entities of the current frame the rules keep per bucket and reports unresolvable rules before anything runs. The rewrite runs on a worker thread (`VTX::FilterReplayFile`) with a chunk progress bar and Cancel, then reports frames, kept/dropped entities, per-struct drops, event counts and sizes +- **tools/inspector**: **File > Filter Entities** -- rewrite the loaded replay into a new `.vtx` without the selected entities (or with only them). Bucket and struct checklists with per-item counts for the current frame, unique id globs (one per line), property rules (struct / field / value glob), case-insensitive and event-pruning toggles; a live preview shows how many entities of the current frame the rules keep per bucket and reports unresolvable rules before anything runs. The rewrite runs on a worker thread (`VTX::FilterReplayFile`) with a chunk progress bar and Cancel, then reports frames, kept/dropped entities, per-struct drops, event counts and sizes ### Fixed diff --git a/docs/SDK_API.md b/docs/SDK_API.md index 3777aa8..856ce7d 100644 --- a/docs/SDK_API.md +++ b/docs/SDK_API.md @@ -300,14 +300,15 @@ Durability knobs on the file sink (`ChunkedFileSink::Config`; facade users curre ### Filtering entities into a new replay -`FilterReplayFile` writes a copy of a replay with some entities removed -- a blacklist (`Drop`: entities matching any rule go) or a whitelist (`Keep`: only matching entities stay). Rules select by **unique id glob**, **schema struct name glob**, or the **string form of one scalar property**; several rules combine as a union. Every frame survives: the header block is copied byte for byte, each chunk is re-serialized with the surviving entities (same frame range, same per-chunk compression, fresh xxHash64 checksums), and the footer keeps `total_frames`, `duration_seconds` and the per-frame time table -- only the seek table is rebuilt. Timeline events ride along and are, by default, pruned to the entities that still appear somewhere in the output. +`FilterReplayFile` writes a copy of a replay with some entities removed -- a blacklist (`Drop`: entities matching any rule go) or a whitelist (`Keep`: only matching entities stay). Rules select **whole buckets** (schema bucket name glob, or the bucket index as digits), **unique id globs**, **schema struct name globs**, or the **string form of one scalar property**; several rules combine as a union. Every frame survives: the header block is copied byte for byte, each chunk is re-serialized with the surviving entities (same frame range, same per-chunk compression, fresh xxHash64 checksums), and the footer keeps `total_frames`, `duration_seconds` and the per-frame time table -- only the seek table is rebuilt. Timeline events ride along and are, by default, pruned to the entities that still appear somewhere in the output. ```cpp #include "vtx/writer/core/vtx_replay_filter.h" VTX::ReplayFilterSpec spec; spec.mode = VTX::ReplayFilterMode::Drop; // or Keep (whitelist) -spec.rules.push_back(VTX::ReplayFilterRule::StructName("SystemHealth*")); // glob on the struct name +spec.rules.push_back(VTX::ReplayFilterRule::Bucket("SystemHealth")); // every entity of that bucket +spec.rules.push_back(VTX::ReplayFilterRule::StructName("Vehicle*")); // glob on the struct name spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); // glob on the unique id spec.rules.push_back(VTX::ReplayFilterRule::Property("Vehicles", "Team", "2")); // one scalar field, as text @@ -315,13 +316,13 @@ VTX::ReplayFilterResult r = VTX::FilterReplayFile("capture.vtx", "capture_filter [](int32_t done, int32_t total) { return true; }); // optional; false cancels if (r.ok()) { // r.total_frames, r.chunks_rewritten, r.entities_kept / r.entities_dropped, - // r.dropped_by_struct (struct name -> count), r.events_kept / r.events_dropped, r.output_bytes + // r.dropped_by_struct / r.dropped_by_bucket (name -> count), r.events_kept / r.events_dropped, r.output_bytes } else { // r.error -- the destination is removed on any failure or cancel } ``` -A `Property` rule with an empty struct name means "every struct declaring that field". Matchable field types are Bool (`true`/`false`), Int32, Int64, Float, Double (shortest round-trip decimal) and String; arrays, maps, nested structs and the compound value types cannot be matched, and unresolvable rules (unknown struct or field, empty pattern) fail before anything is written. Only top-level bucket entities are filtered -- nested containers inside a kept entity are never touched -- and a bucket that ends up empty stays in place, since bucket index is positional. The schema is never pruned: dropping every instance of a struct leaves the struct declared. Both backends are supported and the output keeps the source format. `ReplayFilterMatcher` exposes the same rule resolution for previews on in-memory frames (`Keeps(bucket, index)`, `Apply(bucket)`), and `GlobMatch` is the pattern primitive (`*` any run, `?` one character). Requires the reader module (`VTX_BUILD_READER`). +A `Property` rule with an empty struct name means "every struct declaring that field". Matchable field types are Bool (`true`/`false`), Int32, Int64, Float, Double (shortest round-trip decimal) and String; arrays, maps, nested structs and the compound value types cannot be matched, and unresolvable rules (unknown struct or field, empty pattern) fail before anything is written. A `Bucket` rule that matches no schema bucket is an error too, and it lists the available names. Only top-level bucket entities are filtered -- nested containers inside a kept entity are never touched -- and a bucket that ends up empty (including a dropped bucket) stays in place, since bucket index is positional: readers still see the bucket, with no entities. The schema is never pruned: dropping every instance of a struct leaves the struct declared. Both backends are supported and the output keeps the source format. `ReplayFilterMatcher` exposes the same rule resolution for previews on in-memory frames (`Keeps(bucket_index, bucket, entity_index)`, `Apply(bucket_index, bucket)`, `BucketMatches(bucket_index)`), and `GlobMatch` is the pattern primitive (`*` any run, `?` one character). Requires the reader module (`VTX_BUILD_READER`). ### Sink performance observer diff --git a/sdk/include/vtx/writer/core/vtx_replay_filter.h b/sdk/include/vtx/writer/core/vtx_replay_filter.h index 5a23b71..7ff8935 100644 --- a/sdk/include/vtx/writer/core/vtx_replay_filter.h +++ b/sdk/include/vtx/writer/core/vtx_replay_filter.h @@ -4,20 +4,26 @@ * * @details FilterReplayFile() rewrites an existing replay into a new file, keeping * every frame but dropping (or keeping only) the entities selected by a small rule - * set: unique id globs, schema struct names, or the value of one scalar property. - * The header block (schema, uuid, name, recording timestamp, metadata) is copied - * byte for byte, every chunk is re-serialized with the surviving entities, and the - * footer is rebuilt with the same frame count, duration and per-frame time table; - * only the seek table (offsets, sizes, checksums) changes. Timeline events are - * carried over, optionally pruned to the entities that still exist. + * set: whole buckets, unique id globs, schema struct names, or the value of one + * scalar property. The header block (schema, uuid, name, recording timestamp, + * metadata) is copied byte for byte, every chunk is re-serialized with the + * surviving entities, and the footer is rebuilt with the same frame count, duration + * and per-frame time table; only the seek table (offsets, sizes, checksums) + * changes. Timeline events are carried over, optionally pruned to the entities + * that still exist. * * VTX::ReplayFilterSpec spec; * spec.mode = VTX::ReplayFilterMode::Drop; - * spec.rules.push_back(VTX::ReplayFilterRule::StructName("SystemHealth*")); + * spec.rules.push_back(VTX::ReplayFilterRule::Bucket("SystemHealth")); + * spec.rules.push_back(VTX::ReplayFilterRule::StructName("Vehicle*")); * spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); * const auto r = VTX::FilterReplayFile("in.vtx", "out.vtx", spec); * if (!r.ok()) { ... r.error ... } * + * Buckets are positional (the schema names them by index), so a dropped bucket is + * emptied in every frame rather than removed: readers still see it, with no + * entities. + * * Requires the reader module (VTX_BUILD_READER): the source replay is read through * the standard reader, so the function is only compiled when vtx_reader is built. * @@ -49,6 +55,7 @@ namespace VTX { UniqueId, ///< Glob against the entity's unique id (Bucket::unique_ids). StructName, ///< Glob against the schema struct name of the entity's type id. Property, ///< Glob against the string form of one scalar property value. + Bucket, ///< Every entity of the buckets whose schema name matches the glob (or whose index equals it). }; /** @@ -59,10 +66,14 @@ namespace VTX { * field". Supported field types: Bool ("true"/"false"), Int32, Int64, * Float, Double (shortest round-trip decimal) and String. Arrays, maps, * nested structs and the compound value types are not matchable. + * Bucket rules resolve against the schema's bucket names; a pattern made + * only of digits selects the bucket at that index instead (also for + * buckets the schema does not name). A bucket rule matching no bucket + * is an error. */ struct ReplayFilterRule { ReplayFilterRuleKind kind = ReplayFilterRuleKind::UniqueId; - std::string pattern; ///< Glob matched against the id / struct name / value. + std::string pattern; ///< Glob matched against the id / struct name / value / bucket name. std::string struct_name; ///< Property only. Empty = any struct with that field. std::string field_name; ///< Property only. bool case_insensitive = false; ///< ASCII case-insensitive match. @@ -71,6 +82,7 @@ namespace VTX { static ReplayFilterRule StructName(std::string pattern, bool case_insensitive = false); static ReplayFilterRule Property(std::string struct_name, std::string field_name, std::string value_pattern, bool case_insensitive = false); + static ReplayFilterRule Bucket(std::string pattern, bool case_insensitive = false); }; struct ReplayFilterSpec { @@ -95,6 +107,9 @@ namespace VTX { /// Dropped entity count per struct name (entities of an unknown type id are /// listed as "type#"). std::map dropped_by_struct; + /// Dropped entity count per bucket (schema name, or "bucket#" when the + /// schema does not name it). + std::map dropped_by_bucket; bool ok() const { return error.empty(); } }; @@ -110,7 +125,9 @@ namespace VTX { /** * @brief Resolves a rule set against a file's schema and evaluates entities. * @details Built once per file; used by FilterReplayFile() and available to - * callers that want to preview a spec on in-memory frames. + * callers that want to preview a spec on in-memory frames. Every query + * takes the bucket's index within the frame, which is what bucket rules + * and the schema's bucket names key on. */ class ReplayFilterMatcher { public: @@ -120,25 +137,32 @@ namespace VTX { const std::string& error() const { return error_; } ReplayFilterMode mode() const { return mode_; } - /// True if entity @p index of @p bucket matches at least one rule. - bool Matches(const Bucket& bucket, size_t index) const; + /// True if entity @p index of @p bucket (the frame's bucket number + /// @p bucket_index) matches at least one rule. + bool Matches(int32_t bucket_index, const Bucket& bucket, size_t index) const; /// Applies the mode: true if the entity survives the filter. - bool Keeps(const Bucket& bucket, size_t index) const { - return Matches(bucket, index) == (mode_ == ReplayFilterMode::Keep); + bool Keeps(int32_t bucket_index, const Bucket& bucket, size_t index) const { + return Matches(bucket_index, bucket, index) == (mode_ == ReplayFilterMode::Keep); } + /// True if a bucket rule selects the whole bucket @p bucket_index. + bool BucketMatches(int32_t bucket_index) const; /// Struct name for a type id ("type#" when the schema does not know it). std::string TypeName(int32_t type_id) const; + /// Schema name of a bucket ("bucket#" when the schema does not name it). + std::string BucketName(int32_t bucket_index) const; /** * @brief Removes every entity Keeps() rejects, in place. + * @param bucket_index The bucket's number within its frame. * @param on_drop Optional; called with the bucket and the index of each * dropped entity before it is removed. * @return Number of removed entities. When anything was removed the bucket's * type_ranges are cleared (serialization rebuilds them from * entity_type_id). */ - size_t Apply(Bucket& bucket, const std::function& on_drop = nullptr) const; + size_t Apply(int32_t bucket_index, Bucket& bucket, + const std::function& on_drop = nullptr) const; private: struct ResolvedRule { @@ -146,13 +170,16 @@ namespace VTX { std::string pattern; bool case_insensitive = false; std::unordered_map slots; ///< Property: type id -> address. + std::vector bucket_indices; ///< Bucket: schema buckets matched by name. + int32_t bucket_ordinal = -1; ///< Bucket: numeric pattern, matched by index. }; - bool RuleMatches(const ResolvedRule& rule, const Bucket& bucket, size_t index) const; + bool RuleMatches(const ResolvedRule& rule, int32_t bucket_index, const Bucket& bucket, size_t index) const; ReplayFilterMode mode_ = ReplayFilterMode::Drop; std::vector rules_; std::unordered_map type_names_; + std::vector bucket_names_; std::string error_; }; @@ -164,8 +191,9 @@ namespace VTX { * Header bytes are copied verbatim, chunks keep their frame ranges and * per-chunk compression, the footer keeps total_frames, duration and the * per-frame time table. Nested containers inside a kept entity are never - * touched; buckets that end up empty stay in place (bucket index is - * positional). dest_path must differ from source_path. + * touched; buckets that end up empty (including dropped buckets) stay in + * place, since bucket index is positional. dest_path must differ from + * source_path. */ ReplayFilterResult FilterReplayFile(const std::string& source_path, const std::string& dest_path, const ReplayFilterSpec& spec, const ReplayFilterProgress& progress = nullptr); diff --git a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp b/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp index 4c0d4ad..4531dcc 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp +++ b/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp @@ -119,6 +119,8 @@ namespace VTX { return "struct"; case ReplayFilterRuleKind::Property: return "property"; + case ReplayFilterRuleKind::Bucket: + return "bucket"; } return "rule"; } @@ -320,14 +322,22 @@ namespace VTX { return; } Frame frame(*source_frame); - for (Bucket& bucket : frame.GetMutableBuckets()) { + std::vector& buckets = frame.GetMutableBuckets(); + for (size_t b = 0; b < buckets.size(); ++b) { + Bucket& bucket = buckets[b]; + const int32_t bucket_index = static_cast(b); const size_t before = bucket.entities.size(); - const size_t dropped = matcher.Apply(bucket, [&](const Bucket& b, size_t index) { - ++result.dropped_by_struct[matcher.TypeName(b.entities[index].entity_type_id)]; - }); + const size_t dropped = + matcher.Apply(bucket_index, bucket, [&](const Bucket& dropped_from, size_t index) { + ++result + .dropped_by_struct[matcher.TypeName(dropped_from.entities[index].entity_type_id)]; + }); result.entities_seen += before; result.entities_dropped += dropped; result.entities_kept += before - dropped; + if (dropped > 0) { + result.dropped_by_bucket[matcher.BucketName(bucket_index)] += dropped; + } if (track_survivors) { for (const std::string& id : bucket.unique_ids) { survivors.insert(id); @@ -454,6 +464,14 @@ namespace VTX { return rule; } + ReplayFilterRule ReplayFilterRule::Bucket(std::string pattern, bool case_insensitive) { + ReplayFilterRule rule; + rule.kind = ReplayFilterRuleKind::Bucket; + rule.pattern = std::move(pattern); + rule.case_insensitive = case_insensitive; + return rule; + } + bool GlobMatch(std::string_view pattern, std::string_view text, bool case_insensitive) { size_t p = 0; size_t t = 0; @@ -488,8 +506,9 @@ namespace VTX { for (const auto& [type_id, struct_cache] : cache.structs) { type_names_[type_id] = struct_cache.name; } + bucket_names_ = cache.bucket_names; if (spec.rules.empty()) { - error_ = "The filter has no rules; add at least one unique id, struct or property rule."; + error_ = "The filter has no rules; add at least one bucket, unique id, struct or property rule."; return; } for (const ReplayFilterRule& rule : spec.rules) { @@ -538,6 +557,36 @@ namespace VTX { return; } } + } else if (rule.kind == ReplayFilterRuleKind::Bucket) { + const bool numeric = std::all_of(rule.pattern.begin(), rule.pattern.end(), + [](unsigned char c) { return std::isdigit(c) != 0; }); + if (numeric) { + // A digits-only pattern addresses the bucket by index, named or not. + int32_t ordinal = -1; + const auto parsed = + std::from_chars(rule.pattern.data(), rule.pattern.data() + rule.pattern.size(), ordinal); + if (parsed.ec != std::errc {} || ordinal < 0) { + error_ = "Bucket rule: '" + rule.pattern + "' is not a valid bucket index."; + return; + } + resolved.bucket_ordinal = ordinal; + } else { + for (size_t b = 0; b < bucket_names_.size(); ++b) { + if (GlobMatch(rule.pattern, bucket_names_[b], rule.case_insensitive)) { + resolved.bucket_indices.push_back(static_cast(b)); + } + } + if (resolved.bucket_indices.empty()) { + std::string names; + for (const std::string& name : bucket_names_) { + names += (names.empty() ? "'" : ", '") + name + "'"; + } + error_ = "Bucket rule: no schema bucket matches '" + rule.pattern + "'" + + (names.empty() ? std::string(" (the schema names no buckets).") + : " (buckets: " + names + ")."); + return; + } + } } rules_.push_back(std::move(resolved)); } @@ -551,8 +600,37 @@ namespace VTX { return "type#" + std::to_string(type_id); } - bool ReplayFilterMatcher::RuleMatches(const ResolvedRule& rule, const Bucket& bucket, size_t index) const { + std::string ReplayFilterMatcher::BucketName(int32_t bucket_index) const { + if (bucket_index >= 0 && static_cast(bucket_index) < bucket_names_.size() && + !bucket_names_[static_cast(bucket_index)].empty()) { + return bucket_names_[static_cast(bucket_index)]; + } + return "bucket#" + std::to_string(bucket_index); + } + + bool ReplayFilterMatcher::BucketMatches(int32_t bucket_index) const { + if (bucket_index < 0) { + return false; + } + for (const ResolvedRule& rule : rules_) { + if (rule.kind != ReplayFilterRuleKind::Bucket) { + continue; + } + if (bucket_index == rule.bucket_ordinal || std::find(rule.bucket_indices.begin(), rule.bucket_indices.end(), + bucket_index) != rule.bucket_indices.end()) { + return true; + } + } + return false; + } + + bool ReplayFilterMatcher::RuleMatches(const ResolvedRule& rule, int32_t bucket_index, const Bucket& bucket, + size_t index) const { switch (rule.kind) { + case ReplayFilterRuleKind::Bucket: + return bucket_index >= 0 && (bucket_index == rule.bucket_ordinal || + std::find(rule.bucket_indices.begin(), rule.bucket_indices.end(), + bucket_index) != rule.bucket_indices.end()); case ReplayFilterRuleKind::UniqueId: { const std::string_view id = index < bucket.unique_ids.size() ? std::string_view(bucket.unique_ids[index]) : std::string_view(); @@ -581,23 +659,24 @@ namespace VTX { return false; } - bool ReplayFilterMatcher::Matches(const Bucket& bucket, size_t index) const { + bool ReplayFilterMatcher::Matches(int32_t bucket_index, const Bucket& bucket, size_t index) const { if (index >= bucket.entities.size()) { return false; } for (const ResolvedRule& rule : rules_) { - if (RuleMatches(rule, bucket, index)) { + if (RuleMatches(rule, bucket_index, bucket, index)) { return true; } } return false; } - size_t ReplayFilterMatcher::Apply(Bucket& bucket, const std::function& on_drop) const { + size_t ReplayFilterMatcher::Apply(int32_t bucket_index, Bucket& bucket, + const std::function& on_drop) const { const size_t count = bucket.entities.size(); size_t write = 0; for (size_t read = 0; read < count; ++read) { - if (!Keeps(bucket, read)) { + if (!Keeps(bucket_index, bucket, read)) { if (on_drop) { on_drop(bucket, read); } @@ -641,7 +720,7 @@ namespace VTX { }; if (spec.rules.empty()) { - return fail("The filter has no rules; add at least one unique id, struct or property rule."); + return fail("The filter has no rules; add at least one bucket, unique id, struct or property rule."); } if (source_path.empty() || dest_path.empty()) { return fail("Source and destination paths are required."); diff --git a/tests/writer/test_replay_filter.cpp b/tests/writer/test_replay_filter.cpp index f8f9f09..46683bb 100644 --- a/tests/writer/test_replay_filter.cpp +++ b/tests/writer/test_replay_filter.cpp @@ -40,6 +40,23 @@ namespace { constexpr int64_t kBaseUtc = 17'000'000'000'000'000LL; // unix-relative 100 ns ticks constexpr int64_t kFrameTicks = 166'666; + std::string ReadFileText(const std::string& path) { + std::ifstream in(path, std::ios::binary); + return std::string((std::istreambuf_iterator(in)), std::istreambuf_iterator()); + } + + // The fixture schema with three buckets instead of one; the struct definitions + // are unchanged so the entity builders below stay valid. + std::string MultiBucketSchemaJson() { + std::string json = ReadFileText(VtxTest::FixturePath("test_schema.json")); + const std::string single = "\"buckets\": [\"entity\"]"; + const size_t at = json.find(single); + if (at != std::string::npos) { + json.replace(at, single.size(), "\"buckets\": [\"entity\", \"fx\", \"telemetry\"]"); + } + return json; + } + const char* FormatName(VTX::VtxFormat f) { return f == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; } @@ -106,6 +123,20 @@ namespace { return f; } + // Same entities spread over three buckets: players in "entity" (0), the + // projectile in "fx" (1), the match state in "telemetry" (2). + VTX::Frame BuildMultiBucketFrame(int frame) { + VTX::Frame f; + auto& entity = f.CreateBucket("entity"); + Add(entity, "player_0", MakePlayer("player_0", "Alpha", 1, frame)); + Add(entity, "player_1", MakePlayer("player_1", "Bravo", 2, frame)); + auto& fx = f.CreateBucket("fx"); + Add(fx, ProjectileId(frame), MakeProjectile(ProjectileId(frame), "player_0", "bullet", frame)); + auto& telemetry = f.CreateBucket("telemetry"); + Add(telemetry, "match", MakeMatch(frame)); + return f; + } + struct Snapshot { bool ok = false; VTX::VtxFormat format = VTX::VtxFormat::Unknown; @@ -202,6 +233,22 @@ namespace { } } + // Like ExpectFramesFiltered, with the predicate also given the bucket index. + template + void ExpectFramesFilteredByBucket(const Snapshot& source, const Snapshot& output, DropPredicate dropped) { + ASSERT_EQ(output.frames.size(), source.frames.size()); + for (size_t f = 0; f < source.frames.size(); ++f) { + const auto& sb = source.frames[f].GetBuckets(); + const auto& ob = output.frames[f].GetBuckets(); + ASSERT_EQ(ob.size(), sb.size()) << "frame " << f; + for (size_t b = 0; b < sb.size(); ++b) { + ExpectBucketFiltered( + sb[b], ob[b], [&](const VTX::Bucket& bucket, size_t i) { return dropped(b, bucket, i); }, + "frame " + std::to_string(f) + " bucket " + std::to_string(b)); + } + } + } + // Header bit-identical fields, same frame count/duration/time table, same chunk // frame ranges. void ExpectSameHeaderFooterFraming(const Snapshot& source, const Snapshot& output) { @@ -365,6 +412,33 @@ class ReplayFilterTest : public ::testing::TestWithParam { writer->Stop(); return cfg.output_filepath; } + + std::string WriteMultiBucketSource(const std::string& suffix) const { + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = Path(suffix + "_src"); + cfg.schema_json_content = MultiBucketSchemaJson(); + cfg.replay_name = "ReplayFilterTest"; + cfg.replay_uuid = "uuid-filter-" + suffix; + cfg.default_fps = kFps; + cfg.chunk_max_frames = kChunkFrames; + cfg.use_compression = true; + + auto writer = GetParam() == VTX::VtxFormat::FlatBuffers ? VTX::CreateFlatBuffersWriterFacade(cfg) + : VTX::CreateProtobufWriterFacade(cfg); + EXPECT_TRUE(writer); + if (!writer) { + return {}; + } + for (int i = 0; i < kFrames; ++i) { + auto frame = BuildMultiBucketFrame(i); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / kFps; + t.created_utc_time = kBaseUtc + int64_t(i) * kFrameTicks; + writer->RecordFrame(frame, t); + } + writer->Stop(); + return cfg.output_filepath; + } }; // --------------------------------------------------------------------------- @@ -685,6 +759,109 @@ TEST_P(ReplayFilterTest, CancelRemovesDestination) { EXPECT_FALSE(std::filesystem::exists(dst)); } +// --------------------------------------------------------------------------- +// Bucket rules: a dropped bucket is emptied in every frame and keeps its slot. +// --------------------------------------------------------------------------- +TEST_P(ReplayFilterTest, DropByBucketName) { + const std::string src = WriteMultiBucketSource("dropbucket"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("dropbucket_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("fx")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_seen, 4u * kFrames); + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.entities_kept, 3u * kFrames); + ASSERT_EQ(r.dropped_by_bucket.size(), 1u); + EXPECT_EQ(r.dropped_by_bucket.at("fx"), 1u * kFrames); + EXPECT_EQ(r.dropped_by_struct.at("Projectile"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFilteredByBucket(source, output, [](size_t bucket, const VTX::Bucket&, size_t) { return bucket == 1; }); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 3u); // the emptied bucket keeps its slot + EXPECT_TRUE(frame.GetBuckets()[1].entities.empty()); + EXPECT_TRUE(frame.GetBuckets()[1].unique_ids.empty()); + EXPECT_EQ(frame.GetBuckets()[0].unique_ids, (std::vector {"player_0", "player_1"})); + EXPECT_EQ(frame.GetBuckets()[2].unique_ids, (std::vector {"match"})); + } + ExpectSeekTableMatchesFile(dst, output.footer); +} + +// A whitelisted bucket keeps all of its entities; other rules still apply elsewhere. +TEST_P(ReplayFilterTest, KeepBucketCombinesWithOtherRules) { + const std::string src = WriteMultiBucketSource("keepbucket"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("keepbucket_out"); + + VTX::ReplayFilterSpec spec; + spec.mode = VTX::ReplayFilterMode::Keep; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("telemetry")); + spec.rules.push_back(VTX::ReplayFilterRule::StructName("Player")); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_kept, 3u * kFrames); + EXPECT_EQ(r.entities_dropped, 1u * kFrames); + EXPECT_EQ(r.dropped_by_bucket.at("fx"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectSameHeaderFooterFraming(source, output); + ExpectFramesFilteredByBucket(source, output, [](size_t bucket, const VTX::Bucket& b, size_t i) { + return bucket != 2 && b.entities[i].entity_type_id != kPlayer; + }); + for (const VTX::Frame& frame : output.frames) { + ASSERT_EQ(frame.GetBuckets().size(), 3u); + EXPECT_EQ(frame.GetBuckets()[0].entities.size(), 2u); + EXPECT_TRUE(frame.GetBuckets()[1].entities.empty()); + EXPECT_EQ(frame.GetBuckets()[2].unique_ids, (std::vector {"match"})); + } +} + +// Buckets can be addressed by index or by (case-insensitive) name glob; a rule +// that matches no schema bucket is an error rather than a silent no-op. +TEST_P(ReplayFilterTest, BucketByIndexAndGlob) { + const std::string src = WriteMultiBucketSource("bucketidx"); + ASSERT_FALSE(src.empty()); + const std::string dst = Path("bucketidx_out"); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("2")); + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("F?", /*case_insensitive=*/true)); + const auto r = VTX::FilterReplayFile(src, dst, spec); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.entities_dropped, 2u * kFrames); + EXPECT_EQ(r.dropped_by_bucket.at("fx"), 1u * kFrames); + EXPECT_EQ(r.dropped_by_bucket.at("telemetry"), 1u * kFrames); + + const Snapshot source = ReadReplay(src); + const Snapshot output = ReadReplay(dst); + ASSERT_TRUE(source.ok); + ASSERT_TRUE(output.ok); + ExpectFramesFilteredByBucket(source, output, [](size_t bucket, const VTX::Bucket&, size_t) { return bucket >= 1; }); + + VTX::ReplayFilterSpec strict; + strict.rules.push_back(VTX::ReplayFilterRule::Bucket("F?")); // case-sensitive: matches nothing + const auto r2 = VTX::FilterReplayFile(src, Path("bucketidx_strict_out"), strict); + EXPECT_FALSE(r2.ok()); + EXPECT_NE(r2.error.find("no schema bucket"), std::string::npos) << r2.error; + EXPECT_NE(r2.error.find("'telemetry'"), std::string::npos) << r2.error; // lists the available names + + VTX::ReplayFilterSpec bad_index; + bad_index.rules.push_back(VTX::ReplayFilterRule::Bucket("99999999999")); + const auto r3 = VTX::FilterReplayFile(src, Path("bucketidx_bad_out"), bad_index); + EXPECT_FALSE(r3.ok()); + EXPECT_NE(r3.error.find("not a valid bucket index"), std::string::npos) << r3.error; +} + INSTANTIATE_TEST_SUITE_P(Formats, ReplayFilterTest, ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), [](const ::testing::TestParamInfo& info) { return FormatName(info.param); }); @@ -732,15 +909,15 @@ TEST(ReplayFilterMatcher, ApplyRemovesRejectedEntitiesInPlace) { VTX::Frame frame = BuildFrame(5); // player_0, player_1, proj_2, match VTX::Bucket& bucket = frame.GetBucket("entity"); ASSERT_EQ(bucket.entities.size(), 4u); - EXPECT_TRUE(matcher.Keeps(bucket, 0)); - EXPECT_FALSE(matcher.Keeps(bucket, 1)); - EXPECT_FALSE(matcher.Keeps(bucket, 2)); - EXPECT_TRUE(matcher.Keeps(bucket, 3)); - EXPECT_FALSE(matcher.Matches(bucket, 42)); // out of range never matches + EXPECT_TRUE(matcher.Keeps(0, bucket, 0)); + EXPECT_FALSE(matcher.Keeps(0, bucket, 1)); + EXPECT_FALSE(matcher.Keeps(0, bucket, 2)); + EXPECT_TRUE(matcher.Keeps(0, bucket, 3)); + EXPECT_FALSE(matcher.Matches(0, bucket, 42)); // out of range never matches std::vector dropped_ids; - const size_t dropped = - matcher.Apply(bucket, [&](const VTX::Bucket& b, size_t index) { dropped_ids.push_back(b.unique_ids[index]); }); + const size_t dropped = matcher.Apply( + 0, bucket, [&](const VTX::Bucket& b, size_t index) { dropped_ids.push_back(b.unique_ids[index]); }); EXPECT_EQ(dropped, 2u); EXPECT_EQ(dropped_ids, (std::vector {"player_1", "proj_2"})); EXPECT_EQ(bucket.unique_ids, (std::vector {"player_0", "match"})); @@ -750,7 +927,7 @@ TEST(ReplayFilterMatcher, ApplyRemovesRejectedEntitiesInPlace) { EXPECT_TRUE(bucket.type_ranges.empty()); // Nothing left to remove: the bucket is untouched and 0 is reported. - EXPECT_EQ(matcher.Apply(bucket), 0u); + EXPECT_EQ(matcher.Apply(0, bucket), 0u); EXPECT_EQ(bucket.entities.size(), 2u); // Keep mode with a property rule resolved against the schema. @@ -761,9 +938,9 @@ TEST(ReplayFilterMatcher, ApplyRemovesRejectedEntitiesInPlace) { ASSERT_TRUE(keeper.valid()) << keeper.error(); VTX::Frame early = BuildFrame(5); // Phase = warmup VTX::Frame late = BuildFrame(75); // Phase = live - EXPECT_EQ(keeper.Apply(early.GetBucket("entity")), 3u); + EXPECT_EQ(keeper.Apply(0, early.GetBucket("entity")), 3u); EXPECT_EQ(early.GetBucket("entity").unique_ids, (std::vector {"match"})); - EXPECT_EQ(keeper.Apply(late.GetBucket("entity")), 4u); + EXPECT_EQ(keeper.Apply(0, late.GetBucket("entity")), 4u); EXPECT_TRUE(late.GetBucket("entity").entities.empty()); // Unresolvable rules are reported, not silently ignored. @@ -773,3 +950,40 @@ TEST(ReplayFilterMatcher, ApplyRemovesRejectedEntitiesInPlace) { EXPECT_FALSE(invalid.valid()); EXPECT_NE(invalid.error().find("NoSuchField"), std::string::npos) << invalid.error(); } + +TEST(ReplayFilterMatcher, BucketRulesResolveAgainstSchemaBuckets) { + VTX::SchemaRegistry registry; + ASSERT_TRUE(registry.LoadFromRawString(MultiBucketSchemaJson())); + const VTX::PropertyAddressCache& cache = registry.GetPropertyCache(); + ASSERT_EQ(cache.bucket_names, (std::vector {"entity", "fx", "telemetry"})); + + VTX::ReplayFilterSpec spec; + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("tele*")); + spec.rules.push_back(VTX::ReplayFilterRule::Bucket("7")); // by index, beyond the named ones + const VTX::ReplayFilterMatcher matcher(spec, cache); + ASSERT_TRUE(matcher.valid()) << matcher.error(); + EXPECT_FALSE(matcher.BucketMatches(0)); + EXPECT_FALSE(matcher.BucketMatches(1)); + EXPECT_TRUE(matcher.BucketMatches(2)); + EXPECT_TRUE(matcher.BucketMatches(7)); + EXPECT_FALSE(matcher.BucketMatches(-1)); + EXPECT_EQ(matcher.BucketName(1), "fx"); + EXPECT_EQ(matcher.BucketName(7), "bucket#7"); + + VTX::Frame frame = BuildMultiBucketFrame(3); + VTX::Bucket& telemetry = frame.GetBucket("telemetry"); + VTX::Bucket& fx = frame.GetBucket("fx"); + EXPECT_TRUE(matcher.Matches(2, telemetry, 0)); + EXPECT_FALSE(matcher.Matches(1, fx, 0)); + EXPECT_FALSE(matcher.Keeps(2, telemetry, 0)); // Drop mode + EXPECT_TRUE(matcher.Keeps(1, fx, 0)); + EXPECT_EQ(matcher.Apply(2, telemetry), 1u); + EXPECT_TRUE(telemetry.entities.empty()); + EXPECT_EQ(matcher.Apply(1, fx), 0u); + + VTX::ReplayFilterSpec unknown; + unknown.rules.push_back(VTX::ReplayFilterRule::Bucket("nope")); + const VTX::ReplayFilterMatcher invalid(unknown, cache); + EXPECT_FALSE(invalid.valid()); + EXPECT_NE(invalid.error().find("no schema bucket"), std::string::npos) << invalid.error(); +} diff --git a/tools/inspector/include/windows/filter_entities_window.h b/tools/inspector/include/windows/filter_entities_window.h index 537af16..01553bc 100644 --- a/tools/inspector/include/windows/filter_entities_window.h +++ b/tools/inspector/include/windows/filter_entities_window.h @@ -38,6 +38,12 @@ class FilterEntitiesWindow : public IGuiLayer { private: enum class Phase { Idle, Running, Done }; + struct BucketRow { + std::string name; + int32_t index = -1; ///< Bucket number within a frame (schema order). + bool selected = false; + }; + struct StructRow { std::string name; int32_t type_id = -1; @@ -67,6 +73,7 @@ class FilterEntitiesWindow : public IGuiLayer { uint64_t kept = 0; std::vector>> buckets; ///< name -> (kept, seen) std::vector per_struct_seen; ///< indexed like structs_ + std::vector per_bucket_seen; ///< indexed by bucket number }; void DrawContent(); @@ -82,6 +89,7 @@ class FilterEntitiesWindow : public IGuiLayer { std::shared_ptr session_; VTX::PropertyAddressCache schema_cache_; + std::vector buckets_; std::vector structs_; std::vector property_rules_; char unique_id_patterns_[4096] = {}; diff --git a/tools/inspector/src/windows/filter_entities_window.cpp b/tools/inspector/src/windows/filter_entities_window.cpp index 4d6c445..8ba8bfe 100644 --- a/tools/inspector/src/windows/filter_entities_window.cpp +++ b/tools/inspector/src/windows/filter_entities_window.cpp @@ -71,6 +71,9 @@ FilterEntitiesWindow::FilterEntitiesWindow(std::shared_ptr ses } std::sort(structs_.begin(), structs_.end(), [](const StructRow& a, const StructRow& b) { return a.name < b.name; }); + for (size_t b = 0; b < schema_cache_.bucket_names.size(); ++b) { + buckets_.push_back(BucketRow {schema_cache_.bucket_names[b], static_cast(b), false}); + } } } } @@ -103,6 +106,11 @@ VTX::ReplayFilterSpec FilterEntitiesWindow::BuildSpec() const { VTX::ReplayFilterSpec spec; spec.mode = mode_ == 1 ? VTX::ReplayFilterMode::Keep : VTX::ReplayFilterMode::Drop; spec.prune_events = prune_events_; + for (const BucketRow& row : buckets_) { + if (row.selected) { + spec.rules.push_back(VTX::ReplayFilterRule::Bucket(std::to_string(row.index))); + } + } for (const StructRow& row : structs_) { if (row.selected) { spec.rules.push_back(VTX::ReplayFilterRule::StructName(row.name, case_insensitive_)); @@ -142,6 +150,7 @@ void FilterEntitiesWindow::RefreshPreview(const VTX::ReplayFilterSpec& spec, con preview_ = Preview {}; preview_.key = key; preview_.per_struct_seen.assign(structs_.size(), 0); + preview_.per_bucket_seen.assign(buckets_.size(), 0); VTX::IVtxReaderFacade* reader = session_ ? session_->GetReader() : nullptr; if (!reader) { @@ -171,13 +180,16 @@ void FilterEntitiesWindow::RefreshPreview(const VTX::ReplayFilterSpec& spec, con const auto& buckets = frame->GetBuckets(); for (size_t b = 0; b < buckets.size(); ++b) { const VTX::Bucket& bucket = buckets[b]; + if (b < preview_.per_bucket_seen.size()) { + preview_.per_bucket_seen[b] = bucket.entities.size(); + } uint64_t kept = 0; for (size_t i = 0; i < bucket.entities.size(); ++i) { const auto row = struct_rows.find(bucket.entities[i].entity_type_id); if (row != struct_rows.end()) { ++preview_.per_struct_seen[row->second]; } - if (matcher && matcher->Keeps(bucket, i)) { + if (matcher && matcher->Keeps(static_cast(b), bucket, i)) { ++kept; } } @@ -237,8 +249,10 @@ void FilterEntitiesWindow::DrawContent() { const std::filesystem::path source(session_->current_file_path_); ImGui::TextWrapped("Source: %s", source.filename().string().c_str()); ImGui::TextColored(kDim, - "%d frames, %d structs in the schema. Every frame is kept; only the entity payload changes.", - session_->GetTotalFrames(), static_cast(structs_.size())); + "%d frames, %d buckets, %d structs in the schema. Every frame is kept; only the entity " + "payload changes.", + session_->GetTotalFrames(), static_cast(buckets_.size()), + static_cast(structs_.size())); ImGui::Separator(); const bool busy = phase_ == Phase::Running; @@ -261,6 +275,35 @@ void FilterEntitiesWindow::DrawContent() { } void FilterEntitiesWindow::DrawRules() { + if (ImGui::CollapsingHeader("Buckets", ImGuiTreeNodeFlags_DefaultOpen)) { + if (ImGui::SmallButton("Select all##buckets")) { + for (BucketRow& row : buckets_) { + row.selected = true; + } + } + ImGui::SameLine(); + if (ImGui::SmallButton("Clear##buckets")) { + for (BucketRow& row : buckets_) { + row.selected = false; + } + } + ImGui::SameLine(); + ImGui::TextColored(kDim, "(a dropped bucket is emptied in every frame; its slot stays)"); + if (buckets_.empty()) { + ImGui::TextColored(kDim, "The schema names no buckets."); + } + for (size_t i = 0; i < buckets_.size(); ++i) { + BucketRow& row = buckets_[i]; + ImGui::PushID(static_cast(i) + 2000); + const uint64_t count = i < preview_.per_bucket_seen.size() ? preview_.per_bucket_seen[i] : 0; + char label[256]; + std::snprintf(label, sizeof(label), "%d: %s (%llu)", row.index, row.name.c_str(), + static_cast(count)); + ImGui::Checkbox(label, &row.selected); + ImGui::PopID(); + } + } + if (ImGui::CollapsingHeader("Structs", ImGuiTreeNodeFlags_DefaultOpen)) { if (ImGui::SmallButton("Select all")) { for (StructRow& row : structs_) { @@ -342,8 +385,8 @@ void FilterEntitiesWindow::DrawPreview(const VTX::ReplayFilterSpec& spec) { } if (spec.rules.empty()) { - ImGui::TextColored(kWarn, - "Add at least one rule: tick a struct, enter a unique id pattern or add a property rule."); + ImGui::TextColored(kWarn, "Add at least one rule: tick a bucket or struct, enter a unique id pattern or add a " + "property rule."); return; } if (!preview_.error.empty()) { @@ -432,6 +475,12 @@ void FilterEntitiesWindow::DrawOutcome() { } ImGui::Text("Size: %s -> %s", FormatBytes(outcome_.source_bytes).c_str(), FormatBytes(outcome_.output_bytes).c_str()); + if (!outcome_.dropped_by_bucket.empty()) { + ImGui::TextColored(kDim, "Dropped by bucket:"); + for (const auto& [name, count] : outcome_.dropped_by_bucket) { + ImGui::TextColored(kDim, " %s: %llu", name.c_str(), static_cast(count)); + } + } if (!outcome_.dropped_by_struct.empty()) { ImGui::TextColored(kDim, "Dropped by struct:"); for (const auto& [name, count] : outcome_.dropped_by_struct) { From afb287a4fa3e21389eaf3153dbffddafd34e1829 Mon Sep 17 00:00:00 2001 From: Alejandro Canela Date: Thu, 10 Sep 2026 14:20:57 +0200 Subject: [PATCH 3/4] refactor(sdk): add vtx_transform and define the .vtx framing once New module above vtx_reader + vtx_writer (VTX_BUILD_TRANSFORM, on by default) for every tool that modifies an existing .vtx. It is the only module linking both, so the writer no longer links the reader. - FilterReplayFile moves from vtx/writer/core to vtx/transform - RepairReplayFile / ReplayNeedsRecovery / RecoveryJournalPath move from vtx_writer to vtx_transform (RecoveryJournal / DurableFile stay in the writer: they are part of recording); the CrashRecovery* suite moves with them - The inspector's ReplayCutService is promoted to CutReplayFile / PlanCutFrames / PlanCutChunks, now on both backends and tested; the inspector links vtx_transform instead of vtx_writer vtx/common/vtx_replay_framing.h is the single definition of the on-disk framing (size prefixes, footer trailer, zstd-if-beneficial, xxHash64 checksums, layout probe), replacing four copies across both sinks, repair and filter. ChunkedNetworkSink now fills the seek-table checksum (was 0). --- CHANGELOG.md | 12 +- CMakeLists.txt | 6 +- README.md | 12 +- benchmarks/CMakeLists.txt | 1 + benchmarks/bench_crash_recovery.cpp | 2 +- docs/ARCHITECTURE.md | 28 +- docs/BUILD.md | 3 + docs/FILE_FORMAT.md | 8 +- docs/SDK_API.md | 87 +++-- sdk/include/vtx/common/vtx_replay_framing.h | 300 +++++++++++++++ sdk/include/vtx/reader/core/vtx_reader.h | 5 +- sdk/include/vtx/transform/vtx_replay_cut.h | 88 +++++ .../core => transform}/vtx_replay_filter.h | 6 +- .../core => transform}/vtx_replay_recovery.h | 6 + .../vtx/writer/policies/sinks/file_sink.h | 48 +-- .../vtx/writer/policies/sinks/network_sink.h | 34 +- sdk/src/CMakeLists.txt | 8 + .../src/vtx/common/vtx_replay_framing.cpp | 25 ++ .../reader/core/vtx_deserializer_service.cpp | 14 +- sdk/src/vtx_transform/CMakeLists.txt | 67 ++++ .../src/vtx/transform/vtx_replay_cut.cpp | 364 ++++++++++++++++++ .../src/vtx/transform}/vtx_replay_filter.cpp | 166 +------- .../vtx/transform}/vtx_replay_recovery.cpp | 93 ++--- sdk/src/vtx_writer/CMakeLists.txt | 10 - tests/CMakeLists.txt | 17 +- tests/common/test_replay_framing.cpp | 265 +++++++++++++ .../test_crash_recovery.cpp | 2 +- tests/transform/test_replay_cut.cpp | 335 ++++++++++++++++ .../test_replay_filter.cpp | 98 +---- tests/util/replay_snapshot.h | 109 ++++++ tools/inspector/CMakeLists.txt | 11 +- .../include/services/replay_cut_service.h | 47 --- .../include/windows/cut_replay_window.h | 7 +- .../include/windows/filter_entities_window.h | 2 +- .../include/windows/repair_replay_window.h | 2 +- .../src/services/replay_cut_service.cpp | 328 ---------------- .../src/windows/cut_replay_window.cpp | 18 +- 37 files changed, 1819 insertions(+), 815 deletions(-) create mode 100644 sdk/include/vtx/common/vtx_replay_framing.h create mode 100644 sdk/include/vtx/transform/vtx_replay_cut.h rename sdk/include/vtx/{writer/core => transform}/vtx_replay_filter.h (97%) rename sdk/include/vtx/{writer/core => transform}/vtx_replay_recovery.h (88%) create mode 100644 sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp create mode 100644 sdk/src/vtx_transform/CMakeLists.txt create mode 100644 sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp rename sdk/src/{vtx_writer/src/vtx/writer/core => vtx_transform/src/vtx/transform}/vtx_replay_filter.cpp (78%) rename sdk/src/{vtx_writer/src/vtx/writer/core => vtx_transform/src/vtx/transform}/vtx_replay_recovery.cpp (79%) create mode 100644 tests/common/test_replay_framing.cpp rename tests/{writer => transform}/test_crash_recovery.cpp (99%) create mode 100644 tests/transform/test_replay_cut.cpp rename tests/{writer => transform}/test_replay_filter.cpp (90%) create mode 100644 tests/util/replay_snapshot.h delete mode 100644 tools/inspector/include/services/replay_cut_service.h delete mode 100644 tools/inspector/src/services/replay_cut_service.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d4a5f3..bd207f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,12 +9,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **writer**: **`FilterReplayFile`** (`vtx/writer/core/vtx_replay_filter.h`) -- writes a copy of a replay with entities dropped (blacklist) or exclusively kept (whitelist) by **bucket** (schema bucket name glob or bucket index; a dropped bucket is emptied in every frame and keeps its positional slot), **unique id glob**, **struct name glob** or **scalar property value** (`struct.field` resolved through the file's schema; empty struct = any struct with that field); rules combine as a union. Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums), and the footer is rebuilt with the same `total_frames`, duration and per-frame time table -- only the seek table changes. Timeline events are carried over and (by default) pruned to entities that still exist. Progress callback with cancel (the partial destination is removed); `ReplayFilterResult` reports kept/dropped counts, per-struct and per-bucket drops, event counts and sizes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for previews on in-memory frames. Both backends, output keeps the source format. `vtx_writer` now links `vtx_reader` privately; the entry point is compiled only when `VTX_BUILD_READER` is on. Covered by `ReplayFilterTest.*` (both backends: bucket / struct / unique id / property / numeric / case-insensitive rules, keep-nothing, events carried + pruned, error paths, cancel) plus `ReplayFilterGlob` / `ReplayFilterMatcher` unit tests +- **transform**: new module **`vtx_transform`** (`VTX_BUILD_TRANSFORM`, on by default) -- the home for every tool that modifies an existing replay (repair, cut, entity filter). Cut and filter read the source through `vtx_reader` and re-serialize through `vtx_writer`'s formatter policies, so it is the one module that links both; `vtx_reader`, `vtx_writer` and `vtx_differ` stay independent siblings on `vtx_common` (the writer does **not** link the reader). Skipped automatically when reader or writer is off; `vtx_inspector` requires it. Public headers under `vtx/transform/`, exported as `VTX::vtx_transform` +- **transform**: **`FilterReplayFile`** (`vtx/transform/vtx_replay_filter.h`) -- writes a copy of a replay with entities dropped (blacklist) or exclusively kept (whitelist) by **bucket** (schema bucket name glob or bucket index; a dropped bucket is emptied in every frame and keeps its positional slot), **unique id glob**, **struct name glob** or **scalar property value** (`struct.field` resolved through the file's schema; empty struct = any struct with that field); rules combine as a union. Every frame survives: the header block is copied verbatim, every chunk is re-serialized with the surviving entities (same frame ranges, same per-chunk compression, fresh xxHash64 checksums), and the footer is rebuilt with the same `total_frames`, duration and per-frame time table -- only the seek table changes. Timeline events are carried over and (by default) pruned to entities that still exist. Progress callback with cancel (the partial destination is removed); `ReplayFilterResult` reports kept/dropped counts, per-struct and per-bucket drops, event counts and sizes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for previews on in-memory frames. Both backends, output keeps the source format. Covered by `ReplayFilterTest.*` (both backends: bucket / struct / unique id / property / numeric / case-insensitive rules, keep-nothing, events carried + pruned, error paths, cancel) plus `ReplayFilterGlob` / `ReplayFilterMatcher` unit tests +- **transform**: **`CutReplayFile`** / **`PlanCutFrames`** / **`PlanCutChunks`** (`vtx/transform/vtx_replay_cut.h`) -- the inspector's `ReplayCutService` promoted to an SDK entry point: header copied verbatim, whole chunks copied verbatim with rebased seek entries, edge chunks re-serialized with only the kept frames, footer rebuilt (frames renumbered from 0, time table sliced with absolute ticks, gaps/segments rebased, duration recomputed). Now **both backends** (the inspector cut was FlatBuffers-only) and the per-chunk zstd decision is mirrored from the source instead of forced on. `ReplayCutResult` reports frames, chunks written / rewritten and size; the destination is removed on failure. First tests for the cut -- it was previously untested and never compiled by CI (GUI tools are off there): `ReplayCutTest.*` (both backends: whole-chunk verbatim copy, exact-frame edge rewrite, bound clamping, error paths) and `ReplayCutPlan.*` +- **common/framing**: **`vtx/common/vtx_replay_framing.h`** -- the `.vtx` on-disk framing defined once: size prefixes, header block, footer trailer, the zstd-if-beneficial rule (`ShouldCompress` / `Compress` / `CompressIfBeneficial`), xxHash64 chunk checksums, the zstd sniff, and a layout probe (`ProbeHeader` / `ProbeFooterTrailer` / `ProbeLayout`) plus `CopyRange` for rewriters. `ChunkedFileSink`, `ChunkedNetworkSink`, `RepairReplayFile`, `CutReplayFile`, `FilterReplayFile` and the reader's zstd sniff all go through it -- four independent copies of the framing collapsed into one, so their outputs are byte-identical by construction (the crash-recovery byte-identity suite is the regression net). Covered by `ReplayFraming.*` (the byte layout of every helper, probes on synthetic files) and `ReplayFramingOnDisk.*` (a file recorded by the real sink re-derives its entire seek table through the shared helpers, both backends) - **writer/serialization**: `SessionFooter::events` -- both footer serializers now write timeline events when the pointer is set. The writer facade still writes none; the replay filter uses it to carry a source file's events into the rewritten footer +- **writer/sinks**: `ChunkedNetworkSink` now fills `checksum` in the seek table it streams (was always 0), the same contract as the file sink -- a receiver that concatenates the stream gets a footer whose chunks verify - **tools/inspector**: **File > Filter Entities** -- rewrite the loaded replay into a new `.vtx` without the selected entities (or with only them). Bucket and struct checklists with per-item counts for the current frame, unique id globs (one per line), property rules (struct / field / value glob), case-insensitive and event-pruning toggles; a live preview shows how many entities of the current frame the rules keep per bucket and reports unresolvable rules before anything runs. The rewrite runs on a worker thread (`VTX::FilterReplayFile`) with a chunk progress bar and Cancel, then reports frames, kept/dropped entities, per-struct drops, event counts and sizes +- **tools/inspector**: **File > Cut Replay** now works on Protobuf replays too (it runs through `VTX::CutReplayFile`) + +### Changed + +- **transform**: `RepairReplayFile` / `ReplayNeedsRecovery` / `RecoveryJournalPath` move from `vtx_writer` to `vtx_transform` -- include **`vtx/transform/vtx_replay_recovery.h`** (was `vtx/writer/core/vtx_replay_recovery.h`) and link `VTX::vtx_transform`. The rule is now uniform: the writer only produces new recordings, and every tool that modifies an existing `.vtx` (repair, cut, entity filter) lives in the transform module. `RecoveryJournal` / `DurableFile` stay in the writer (they are part of recording). Behaviour and on-disk output are unchanged; the `CrashRecovery*` byte-identity suite moves with it to `tests/transform/`. `vtx_inspector` links `vtx_transform` (not `vtx_writer`) for **File > Repair Replay**; `vtx_benchmarks` now requires `VTX_BUILD_TRANSFORM` ### Fixed +- **docs**: `FILE_FORMAT.md` described the 4 bytes before each chunk as a "marker / padding" and the trailing 4 bytes of the file as a "sentinel"; they are the chunk's size prefix and the repeated format magic - **reader**: `GetFooter().chunk_index[i].checksum` was always 0 on both backends -- the footer-to-native conversions copied every seek-table field except the xxHash64 checksum (the internal seek table used for chunk loading did carry it). Visible as `checksum: 0` for every chunk in the CLI's `chunks` command, and the inspector's cut copied 0 into the seek-table entries of verbatim chunks. The FlatBuffers footer conversion also dropped each timeline event's `entity_unique_id` and `location`. All three conversions now carry the fields; `ReplayFilterTest.DropByStructName` asserts the footer checksums of a freshly written file match the bytes on disk and `ReplayFilterTest.EventsCarriedAndPruned` reads events back with their entity id and location on both backends ## [0.5.0] - 2026-07-27 diff --git a/CMakeLists.txt b/CMakeLists.txt index 92095c2..6658985 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -71,6 +71,7 @@ set(VTX_INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/sdk/include") option(VTX_BUILD_WRITER "Build vtx_writer module" ON) option(VTX_BUILD_READER "Build vtx_reader module" ON) option(VTX_BUILD_DIFFER "Build vtx_differ module" ON) +option(VTX_BUILD_TRANSFORM "Build vtx_transform module (repair / cut / entity-filter of existing replays; needs reader + writer)" ON) option(BUILD_VTX_TOOL "Build tool suite (tools/)" ON) option(BUILD_VTX_SAMPLES "Build sample programs (samples/)" ON) option(VTX_BUILD_TESTS "Build unit test suite (tests/)" ON) @@ -118,6 +119,9 @@ endif() if(VTX_BUILD_DIFFER) add_compile_definitions(VTX_HAS_DIFFER) endif() +if(VTX_BUILD_TRANSFORM AND VTX_BUILD_READER AND VTX_BUILD_WRITER) + add_compile_definitions(VTX_HAS_TRANSFORM) +endif() # --- Tools --- if(BUILD_VTX_TOOL AND VTX_BUILD_READER AND VTX_BUILD_WRITER) @@ -145,7 +149,7 @@ else() endif() # --- Benchmarks --- -if(VTX_BUILD_BENCHMARKS AND VTX_BUILD_READER AND VTX_BUILD_WRITER AND VTX_BUILD_DIFFER) +if(VTX_BUILD_BENCHMARKS AND VTX_BUILD_READER AND VTX_BUILD_WRITER AND VTX_BUILD_DIFFER AND VTX_BUILD_TRANSFORM) message(STATUS "VTX: Building benchmarks...") add_subdirectory(benchmarks) else() diff --git a/README.md b/README.md index 69dd5f8..2496acf 100644 --- a/README.md +++ b/README.md @@ -140,16 +140,18 @@ The reader's chunk cache is the single sharpest performance edge in the SDK. A w ## Architecture ``` -vtx_common Core types, schemas, serialization adapters, compression (zstd) +vtx_common Core types, schemas, serialization adapters, compression (zstd), on-disk framing | - +--- vtx_writer Record live frame data into .vtx replay files + +--- vtx_writer Record live frame data into .vtx replay files | - +--- vtx_reader Stream and random-access .vtx replay files with chunk-based caching + +--- vtx_reader Stream and random-access .vtx replay files with chunk-based caching | - +--- vtx_differ Compute structural diffs between frames or replay trees + +--- vtx_differ Compute structural diffs between frames or replay trees + | + +--- vtx_transform Modify existing replays: repair a crashed one, cut a frame range, filter entities (uses reader + writer) ``` -Four static libraries. Enable or disable each with a CMake option. Full module breakdown and extension model in [SDK Architecture](https://github.com/ZenosInteractive/VTX/wiki/SDK-Architecture). +Five static libraries. Enable or disable each with a CMake option. Full module breakdown and extension model in [SDK Architecture](https://github.com/ZenosInteractive/VTX/wiki/SDK-Architecture). ## Tools diff --git a/benchmarks/CMakeLists.txt b/benchmarks/CMakeLists.txt index 1d03045..e89c3ee 100644 --- a/benchmarks/CMakeLists.txt +++ b/benchmarks/CMakeLists.txt @@ -74,6 +74,7 @@ target_link_libraries(vtx_benchmarks PRIVATE vtx_reader vtx_writer vtx_differ + vtx_transform # bench_crash_recovery.cpp: VTX::RepairReplayFile ) target_include_directories(vtx_benchmarks PRIVATE diff --git a/benchmarks/bench_crash_recovery.cpp b/benchmarks/bench_crash_recovery.cpp index 087bcc9..3dff19a 100644 --- a/benchmarks/bench_crash_recovery.cpp +++ b/benchmarks/bench_crash_recovery.cpp @@ -18,7 +18,7 @@ #include "vtx/common/vtx_logger.h" #include "vtx/common/vtx_types.h" #include "vtx/reader/core/vtx_reader_facade.h" -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" #include "vtx/writer/core/vtx_writer_facade.h" // brings SchemaRegistry/SchemaSanitizer for writer.h #include "vtx/writer/core/writer.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ba62576..ffd5420 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -3,16 +3,19 @@ ## Module Dependency Graph ``` -vtx_common Core types, compression, schema, serialization adapters +vtx_common Core types, compression, on-disk framing, schema, serialization adapters | - +--- vtx_writer Writes .vtx replay files (frame recording, chunking, flushing) + +--- vtx_writer Writes .vtx replay files (frame recording, chunking, flushing) | - +--- vtx_reader Reads .vtx replay files (chunk-based caching, random access) + +--- vtx_reader Reads .vtx replay files (chunk-based caching, random access) | - +--- vtx_differ Structural diff between frames (binary tree-diff engine) + +--- vtx_differ Structural diff between frames (binary tree-diff engine) + | + +--- vtx_transform Modifies existing replays (repair a crashed one, cut a frame range, filter entities) + -- the one module that links both vtx_reader and vtx_writer ``` -All four modules are built as static libraries. Each can be toggled independently via CMake options (`VTX_BUILD_WRITER`, `VTX_BUILD_READER`, `VTX_BUILD_DIFFER`). `vtx_common` is always built as the shared foundation. +All five modules are built as static libraries. Each can be toggled via CMake options (`VTX_BUILD_WRITER`, `VTX_BUILD_READER`, `VTX_BUILD_DIFFER`, `VTX_BUILD_TRANSFORM`); writer, reader and differ are independent siblings on `vtx_common`, while `vtx_transform` requires reader and writer. `vtx_common` is always built as the shared foundation. ## Module Overview @@ -30,6 +33,7 @@ Provides the shared type system, serialization infrastructure, and utilities use | Diagnostics | `vtx_diagnostics.h` | One SDK-wide error model: `Severity`, `VtxErrorCode`, `VtxDiagnostic` (`VtxError`/`VtxWarning`), `VtxResult`, `ValidationReport` | | Validation | `vtx_validation.h`, `vtx_frame_accessor.h` | `ValidateSchema` / `ValidateEntity` / `ValidateFrame`; strict accessors `FrameAccessor::TryResolve`, `EntityView::TryGet`, `EntityMutator::TrySet` | | Compression | `vtx_deserializer_service.h` | zstd-based chunk compression/decompression | +| On-disk framing | `vtx_replay_framing.h` | Single definition of the `.vtx` block layout: size prefixes, footer trailer, the zstd-if-beneficial rule, xxHash64 chunk checksums, and a layout probe for rewriters. Shared by both sinks and the `vtx_transform` tools (`RepairReplayFile`, `CutReplayFile`, `FilterReplayFile`), so their output is byte-identical by construction | | Logger | `vtx_logger.h` | Thread-safe singleton logger with `VTX_INFO`, `VTX_WARN`, `VTX_ERROR`, `VTX_DEBUG` macros using `std::format` syntax | | Hashing | `vtx_types_helpers.h` | xxHash64 content hashing for fast entity comparison | | Generated code | `src/generated/` | Protobuf (`.pb.h/.cc`) and FlatBuffers (`_generated.h`) schemas, auto-generated from `schemas/` | @@ -45,7 +49,7 @@ Records live frame data into `.vtx` replay files (or streams the same bytes over - **Config**: `WriterFacadeConfig` (file path) or `NetworkWriterFacadeConfig` (host + port) — chunk size, compression, and the schema supplied as a JSON path, an in-memory JSON string (`schema_json_content`), or a pre-built `SchemaRegistry` (`schema_registry`); `create_output_dirs` (default on) auto-creates missing parent directories of the output file - **Policy**: Template-parameterized writer policies select FlatBuffers or Protobuf serialization at compile time - **Sinks**: `ChunkedFileSink` (file), `ChunkedNetworkSink` (TCP stream) — both produce the **same `.vtx` byte sequence**, so a socket receiver only has to concatenate incoming bytes into a file to get a valid replay -- **Crash durability & recovery** (file sink only): `DurableFile` (FILE* wrapper with real `fsync`; `durable_writes` on by default) makes every chunk power-loss durable, each chunk carries an xxHash64 checksum in the seek table, and `RecoveryJournal` maintains an append-only `.vtx.recovery` write-ahead sidecar (chunk commits, exact per-frame times, in-flight frames) that is deleted on clean `Stop()`. `RepairReplayFile()` (`vtx_replay_recovery.h`) reconstructs a valid file from a crashed recording — footer, seek table, and derived time data byte-exact — with `ReplayNeedsRecovery()` / `RecoveryJournalPath()` as the user-driven detection helpers (never automatic on open). `IFileSinkPerfObserver` (`Config::perf_observer` / `WriterFacadeConfig::perf_observer`) exposes per-stage sink timings (serialize / compress / disk write incl. the durability flush) +- **Crash durability & recovery** (file sink only): `DurableFile` (FILE* wrapper with real `fsync`; `durable_writes` on by default) makes every chunk power-loss durable, each chunk carries an xxHash64 checksum in the seek table, and `RecoveryJournal` maintains an append-only `.vtx.recovery` write-ahead sidecar (chunk commits, exact per-frame times, in-flight frames) that is deleted on clean `Stop()`. Reconstructing a crashed recording from that sidecar is `RepairReplayFile()` in `vtx_transform` (below) — the writer only produces new recordings. `IFileSinkPerfObserver` (`Config::perf_observer` / `WriterFacadeConfig::perf_observer`) exposes per-stage sink timings (serialize / compress / disk write incl. the durability flush) - **Data-source interface**: `IFrameDataSource` (`Initialize()` / `GetNextFrame()` / `GetExpectedTotalFrames()`). Concrete implementations: `samples/advance_write.cpp` (JSON / Protobuf / FlatBuffers from disk); `PipeFrameDataSource` (stdin, Windows named pipes, POSIX FIFOs — including a **server mode** that creates the pipe and waits, for independent game-injector-style external producers); `WebSocketFrameDataSource` (`ws://` + `wss://` with TLS). Both streaming sources share the `IFramePayloadAdapter` concept, so a single adapter plugs into either transport. A `SharedMemoryFrameDataSource` (zero-copy SPSC ring, pluggable `ISharedMemoryTransport`) is present in-tree but **experimental / WIP and not yet functional** — landed unintentionally, not a supported input path - **Dependencies**: protobuf + flatbuffers (serialization), zstd (chunk compression), xxHash/xxh3 (chunk + journal-record checksums), IXWebSocket + mbedTLS (WebSocket transport; hidden behind a PIMPL boundary in `websocket_client.cpp` — never leak into the public SDK headers) @@ -72,6 +76,15 @@ Computes structural diffs between two serialized frames. - **Adapters**: `FlatbufferViewAdapter`, `FProtobufViewAdapter` — zero-copy binary node views into serialized buffers - **Output**: `PatchIndex` — list of `DiffIndexOp` operations (Add, Remove, Replace, ReplaceRange) with binary paths +### vtx_transform + +Every tool that modifies an existing `.vtx` lives here — the writer only produces new recordings, the reader only consumes them. It sits above `vtx_reader` and `vtx_writer`: cut and filter read the source through the reader facade and re-serialize chunks through the writer's formatter policies, repair rebuilds a crashed file from its journal through the writer's `RecoveryJournal`, and every byte is framed via `vtx_replay_framing.h` — which is what lets reader and writer stay independent siblings. Requires both (`VTX_BUILD_TRANSFORM`, on by default; skipped automatically when either is off). + +- **Repair** (`vtx_replay_recovery.h`): `RepairReplayFile` reconstructs a recording that died before `Stop()` from its `.vtx.recovery` sidecar — drops a torn tail chunk, verifies each surviving chunk's checksum, re-appends the in-flight frames and synthesizes the footer with the exact per-frame times (at a chunk boundary the result is byte-identical to a clean `Stop()`). `ReplayNeedsRecovery` / `RecoveryJournalPath` are the user-driven detection helpers — never automatic on open. Needs no reader: it works from the journal and the raw bytes +- **Cut** (`vtx_replay_cut.h`): `PlanCutFrames` / `PlanCutChunks` resolve a frame or chunk range against a footer (cheap, for previews); `CutReplayFile` executes the plan — header copied verbatim, whole chunks copied verbatim with rebased seek entries, edge chunks re-serialized with only the kept frames, footer rebuilt with frames renumbered from 0 and the time table sliced (tick values stay absolute). Both backends; the output keeps the source format +- **Entity filter** (`vtx_replay_filter.h`): `FilterReplayFile` rewrites every chunk with entities dropped (blacklist) or exclusively kept (whitelist) by bucket, unique-id glob, struct-name glob or scalar property value. Every frame survives; header, frame count and time table are preserved and only the seek table changes. `ReplayFilterMatcher` / `GlobMatch` expose the rule engine for in-memory previews +- These are the SDK entry points behind the inspector's **File > Repair Replay**, **File > Cut Replay** and **File > Filter Entities** (`vtx_inspector` links `vtx_reader` + `vtx_transform`, not `vtx_writer`) + ## Design Patterns ### Facade Pattern @@ -107,7 +120,8 @@ Each module exposes a single abstract interface (`IVtxReaderFacade`, `IVtxWriter | Namespace | Module | Contents | |---|---|---| -| `VTX` | vtx_common, vtx_reader, vtx_writer | Core types, reader/writer facades, format detection, integration primitives (`JsonMapping`, `ProtoBinding`, `FlatBufferBinding`, `IFrameDataSource`) | +| `VTX` | vtx_common, vtx_reader, vtx_writer, vtx_transform | Core types, reader/writer facades, format detection, replay tools (`RepairReplayFile`, `CutReplayFile`, `FilterReplayFile`), integration primitives (`JsonMapping`, `ProtoBinding`, `FlatBufferBinding`, `IFrameDataSource`) | +| `VTX::Framing` | vtx_common | The `.vtx` on-disk framing helpers (`vtx_replay_framing.h`) | | `VtxDiff` | vtx_differ | Diff engine, patch types, binary view adapters | | `VtxDiff::Flatbuffers` | vtx_differ | FlatBuffers binary view adapter | | `VtxDiff::Protobuf` | vtx_differ | Protobuf binary view adapter | diff --git a/docs/BUILD.md b/docs/BUILD.md index f453ca9..b34cf19 100644 --- a/docs/BUILD.md +++ b/docs/BUILD.md @@ -204,6 +204,7 @@ cmake --build --preset windows-release | `VTX_BUILD_WRITER` | `ON` | Build the writer module | | `VTX_BUILD_READER` | `ON` | Build the reader module | | `VTX_BUILD_DIFFER` | `ON` | Build the differ module | +| `VTX_BUILD_TRANSFORM` | `ON` | Build the transform module (repair / cut / entity filter of existing replays); needs reader + writer, skipped otherwise | | `BUILD_VTX_TOOL` | `ON` | Build tools | | `BUILD_VTX_SAMPLES` | `ON` | Build sample programs | | `VTX_BUILD_TESTS` | `ON` | Build the unit test suite | @@ -255,6 +256,7 @@ dist/ vtx_writer.lib vtx_reader.lib vtx_differ.lib + vtx_transform.lib cmake/VTX/ bin/ vtx_inspector.exe @@ -303,6 +305,7 @@ Imported targets: | `VTX::vtx_writer` | Replay writer | | `VTX::vtx_reader` | Replay reader | | `VTX::vtx_differ` | Frame differ | +| `VTX::vtx_transform` | Tools that modify an existing replay (repair, cut, entity filter) | ## Project Structure diff --git a/docs/FILE_FORMAT.md b/docs/FILE_FORMAT.md index 61f66fa..cc2f96d 100644 --- a/docs/FILE_FORMAT.md +++ b/docs/FILE_FORMAT.md @@ -23,10 +23,12 @@ VTX files store frame-based replay data in a chunked binary format. Each file co +------------------+ | Footer Size (4B) | uint32_t — compressed footer size +------------------+ -| Sentinel (4B) | Padding / alignment +| Magic (4 bytes) | "VTXF" or "VTXP" again — the footer trailer +------------------+ ``` +The framing itself — the size prefixes, the footer trailer, the zstd-if-beneficial rule and the chunk checksums — is defined once in `sdk/include/vtx/common/vtx_replay_framing.h`. Every producer goes through it: the file and network sinks while recording, and the `vtx_transform` tools that modify an existing file (`RepairReplayFile`, `CutReplayFile`, `FilterReplayFile`). Readers locate the blocks the same way (`Framing::ProbeLayout`). + ## Magic Bytes The first 4 bytes identify the serialization format: @@ -66,8 +68,8 @@ The **property schema** embedded in the header defines the layout of `PropertyCo Each chunk contains a batch of serialized frames. Chunks are the unit of I/O and caching. **On disk:** -1. `uint32_t` — chunk marker / padding (4 bytes, skipped on read) -2. Compressed blob — zstd-compressed serialized chunk data +1. `uint32_t` — size in bytes of the payload that follows +2. Payload — serialized chunk data, zstd-compressed when that shrinks it and left raw otherwise (readers sniff the zstd magic rather than trusting a flag) **Chunk contents (after decompression):** - Array of serialized frames, each containing: diff --git a/docs/SDK_API.md b/docs/SDK_API.md index 856ce7d..74d0a60 100644 --- a/docs/SDK_API.md +++ b/docs/SDK_API.md @@ -267,10 +267,10 @@ If the facade is destroyed without an explicit `Stop()`, its destructor finalize A recording that dies before `Stop()` -- process crash or power loss, mid-chunk or mid-frame -- is **recoverable up to the last recorded frame**. While recording, the file sink maintains a write-ahead sidecar **`.vtx.recovery`**: every recorded frame is journaled before it joins the pending batch, and every flushed chunk is committed to the journal *after* its bytes are durable in the `.vtx` (data-before-journal ordering, append-only log, per-record checksums). On a clean `Stop()` the sidecar is deleted -- its presence at open time is the signal of an unclean shutdown. -Recovery is **never automatic**; the flow is user-driven: +Recovery is **never automatic**; the flow is user-driven. The repair entry point lives in `vtx_transform` (link `VTX::vtx_transform`): the writer only produces new recordings, and every tool that modifies an existing `.vtx` sits in that module (see "Rewriting Replays" below): ```cpp -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" if (VTX::ReplayNeedsRecovery(path)) { // cheap: does ".recovery" exist? VTX::RepairResult r = VTX::RepairReplayFile(path); @@ -298,32 +298,6 @@ Durability knobs on the file sink (`ChunkedFileSink::Config`; facade users curre | `enable_recovery_journal` | `true` | Maintain the `.recovery` sidecar. Opting out removes any stale sidecar at session start; a crash then leaves an unrecoverable (footerless) file. | | `journal_compact_threshold_bytes` | `0` (64 MB) | How many superseded journal bytes accrue before the sidecar is compacted (rewritten via an atomic rename). | -### Filtering entities into a new replay - -`FilterReplayFile` writes a copy of a replay with some entities removed -- a blacklist (`Drop`: entities matching any rule go) or a whitelist (`Keep`: only matching entities stay). Rules select **whole buckets** (schema bucket name glob, or the bucket index as digits), **unique id globs**, **schema struct name globs**, or the **string form of one scalar property**; several rules combine as a union. Every frame survives: the header block is copied byte for byte, each chunk is re-serialized with the surviving entities (same frame range, same per-chunk compression, fresh xxHash64 checksums), and the footer keeps `total_frames`, `duration_seconds` and the per-frame time table -- only the seek table is rebuilt. Timeline events ride along and are, by default, pruned to the entities that still appear somewhere in the output. - -```cpp -#include "vtx/writer/core/vtx_replay_filter.h" - -VTX::ReplayFilterSpec spec; -spec.mode = VTX::ReplayFilterMode::Drop; // or Keep (whitelist) -spec.rules.push_back(VTX::ReplayFilterRule::Bucket("SystemHealth")); // every entity of that bucket -spec.rules.push_back(VTX::ReplayFilterRule::StructName("Vehicle*")); // glob on the struct name -spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); // glob on the unique id -spec.rules.push_back(VTX::ReplayFilterRule::Property("Vehicles", "Team", "2")); // one scalar field, as text - -VTX::ReplayFilterResult r = VTX::FilterReplayFile("capture.vtx", "capture_filtered.vtx", spec, - [](int32_t done, int32_t total) { return true; }); // optional; false cancels -if (r.ok()) { - // r.total_frames, r.chunks_rewritten, r.entities_kept / r.entities_dropped, - // r.dropped_by_struct / r.dropped_by_bucket (name -> count), r.events_kept / r.events_dropped, r.output_bytes -} else { - // r.error -- the destination is removed on any failure or cancel -} -``` - -A `Property` rule with an empty struct name means "every struct declaring that field". Matchable field types are Bool (`true`/`false`), Int32, Int64, Float, Double (shortest round-trip decimal) and String; arrays, maps, nested structs and the compound value types cannot be matched, and unresolvable rules (unknown struct or field, empty pattern) fail before anything is written. A `Bucket` rule that matches no schema bucket is an error too, and it lists the available names. Only top-level bucket entities are filtered -- nested containers inside a kept entity are never touched -- and a bucket that ends up empty (including a dropped bucket) stays in place, since bucket index is positional: readers still see the bucket, with no entities. The schema is never pruned: dropping every instance of a struct leaves the struct declared. Both backends are supported and the output keeps the source format. `ReplayFilterMatcher` exposes the same rule resolution for previews on in-memory frames (`Keeps(bucket_index, bucket, entity_index)`, `Apply(bucket_index, bucket)`, `BucketMatches(bucket_index)`), and `GlobMatch` is the pattern primitive (`*` any run, `?` one character). Requires the reader module (`VTX_BUILD_READER`). - ### Sink performance observer To see where write time goes, attach a perf observer — it receives per-stage timings synchronously from the writer thread (implementations must be cheap; leave null for a zero-cost no-op): @@ -396,6 +370,63 @@ The writer behaves identically behind the `IVtxWriterFacade` abstraction -- the --- +## Rewriting Replays (`vtx_transform`) + +`vtx_transform` is where every tool that modifies an existing `.vtx` lives -- the writer only produces new recordings. Three entry points: **repair** a crashed recording (`RepairReplayFile`, documented above under *Crash recovery* next to the journal that feeds it), **cut** a frame range and **filter** entities. Cut and filter read the source through the reader and re-serialize through the writer's formatter policies, so it is the one module that links both (`VTX_BUILD_TRANSFORM`, on by default; skipped automatically when reader or writer is off). Both rewrites copy the header block byte for byte, frame every output block through the shared `vtx/common/vtx_replay_framing.h`, support both backends and keep the source format. Neither touches the source -- `dest_path` must be a different file, and a failed or cancelled rewrite removes its partial destination. + +### Cutting a frame range + +`CutReplayFile` writes the frames of a range into a new replay. Plan first -- the plan is cheap, computed from the footer, and is what a UI previews -- then execute it against the same footer: + +```cpp +#include "vtx/transform/vtx_replay_cut.h" + +const VTX::FileFooter footer = reader->GetFooter(); + +VTX::ReplayCutPlan plan = VTX::PlanCutFrames(footer, 1200, 1799); // exact frames, inclusive +// or: VTX::PlanCutChunks(footer, 3, 5); // whole chunks, copied verbatim +if (!plan.valid) { /* plan.error */ } +// plan.first_frame / last_frame, first_chunk / last_chunk, trims_head / trims_tail, +// chunk_bytes (source bytes involved), FrameCount() + +VTX::ReplayCutResult r = VTX::CutReplayFile("match.vtx", footer, plan, "match_round2.vtx"); +if (r.ok()) { + // r.total_frames, r.chunks_written, r.chunks_rewritten (edge chunks re-serialized: 0, 1 or 2), r.output_bytes +} else { + // r.error +} +``` + +Whole chunks inside the range are copied verbatim (same bytes, same checksum; only their seek entry is rebased). When a bound falls inside a chunk, that edge chunk is re-serialized with only the kept frames -- the per-chunk zstd decision is mirrored from the source and the entry gets a fresh xxHash64 -- so exact cuts never decode more than the two edge chunks and whole-chunk cuts never decode a frame. The footer is rebuilt: frames renumbered from 0, the per-frame time table sliced with **absolute** tick values, gaps and segments filtered and rebased, duration recomputed from the kept stamps. Timeline events are not carried over. Bounds are clamped to the replay; a start that falls between chunks snaps to the next covered frame. + +### Filtering entities + +`FilterReplayFile` writes a copy of a replay with some entities removed -- a blacklist (`Drop`: entities matching any rule go) or a whitelist (`Keep`: only matching entities stay). Rules select **whole buckets** (schema bucket name glob, or the bucket index as digits), **unique id globs**, **schema struct name globs**, or the **string form of one scalar property**; several rules combine as a union. Every frame survives: each chunk is re-serialized with the surviving entities (same frame range, same per-chunk compression, fresh xxHash64 checksums), and the footer keeps `total_frames`, `duration_seconds` and the per-frame time table -- only the seek table is rebuilt. Timeline events ride along and are, by default, pruned to the entities that still appear somewhere in the output. + +```cpp +#include "vtx/transform/vtx_replay_filter.h" + +VTX::ReplayFilterSpec spec; +spec.mode = VTX::ReplayFilterMode::Drop; // or Keep (whitelist) +spec.rules.push_back(VTX::ReplayFilterRule::Bucket("SystemHealth")); // every entity of that bucket +spec.rules.push_back(VTX::ReplayFilterRule::StructName("Vehicle*")); // glob on the struct name +spec.rules.push_back(VTX::ReplayFilterRule::UniqueId("keyboard")); // glob on the unique id +spec.rules.push_back(VTX::ReplayFilterRule::Property("Vehicles", "Team", "2")); // one scalar field, as text + +VTX::ReplayFilterResult r = VTX::FilterReplayFile("capture.vtx", "capture_filtered.vtx", spec, + [](int32_t done, int32_t total) { return true; }); // optional; false cancels +if (r.ok()) { + // r.total_frames, r.chunks_rewritten, r.entities_kept / r.entities_dropped, + // r.dropped_by_struct / r.dropped_by_bucket (name -> count), r.events_kept / r.events_dropped, r.output_bytes +} else { + // r.error -- the destination is removed on any failure or cancel +} +``` + +A `Property` rule with an empty struct name means "every struct declaring that field". Matchable field types are Bool (`true`/`false`), Int32, Int64, Float, Double (shortest round-trip decimal) and String; arrays, maps, nested structs and the compound value types cannot be matched, and unresolvable rules (unknown struct or field, empty pattern) fail before anything is written. A `Bucket` rule that matches no schema bucket is an error too, and it lists the available names. Only top-level bucket entities are filtered -- nested containers inside a kept entity are never touched -- and a bucket that ends up empty (including a dropped bucket) stays in place, since bucket index is positional: readers still see the bucket, with no entities. The schema is never pruned: dropping every instance of a struct leaves the struct declared. `ReplayFilterMatcher` exposes the same rule resolution for previews on in-memory frames (`Keeps(bucket_index, bucket, entity_index)`, `Apply(bucket_index, bucket)`, `BucketMatches(bucket_index)`), and `GlobMatch` is the pattern primitive (`*` any run, `?` one character). + +--- + ## Frame Post-Processor A writer-side hook that runs on every `RecordFrame()` call, **after** timer validation and **before** the serializer touches the native `Frame`. Whatever the processor mutates lands on disk -- callers who re-read the `.vtx` see the post-processed values. diff --git a/sdk/include/vtx/common/vtx_replay_framing.h b/sdk/include/vtx/common/vtx_replay_framing.h new file mode 100644 index 0000000..8fb951c --- /dev/null +++ b/sdk/include/vtx/common/vtx_replay_framing.h @@ -0,0 +1,300 @@ +/** + * @file vtx_replay_framing.h + * @brief The .vtx on-disk framing, defined once for every producer and consumer. + * + * @details A .vtx file is laid out as + * + * [magic][u32 header_size][header] [u32 chunk_size][chunk]* [footer][u32 footer_size][magic] + * + * Every blob (header, chunk, footer) is zstd-compressed when that is beneficial and + * left raw otherwise; readers sniff the zstd magic instead of trusting a flag. A + * chunk's seek-table checksum is xxHash64 over its on-disk payload (the bytes after + * the size prefix). + * + * This header is the single definition of those rules. ChunkedFileSink and + * ChunkedNetworkSink (recording) and the vtx_transform tools that modify an existing + * file (RepairReplayFile, CutReplayFile, FilterReplayFile) all frame bytes through it, + * so a format change is made in one place and their outputs stay byte-identical to + * each other by construction. The reader keeps its own header/footer parser (it owns + * the corrupt-file diagnostics) but shares the constants and the zstd sniff. + * + * Byte order: sizes are stored in host byte order, which is little-endian on every + * supported target -- exactly the bytes the format has always shipped. + * + * @author Zenos Interactive + */ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace VTX { + namespace Framing { + + inline constexpr size_t kMagicSize = 4; ///< "VTXF" / "VTXP" + inline constexpr size_t kSizePrefixSize = sizeof(uint32_t); ///< u32 before header / chunk payloads + inline constexpr size_t kFooterTrailerSize = kSizePrefixSize + kMagicSize; ///< [u32 footer_size][magic] + inline constexpr size_t kCompressMinBytes = 512; ///< Smaller payloads stay raw. + inline constexpr int8_t kDefaultCompressionLevel = 10; ///< zstd level of the writer defaults. + inline constexpr unsigned char kZstdMagic[4] = {0x28, 0xB5, 0x2F, 0xFD}; ///< zstd frame magic, as stored. + + // ------------------------------------------------------------------ + // Compression + // ------------------------------------------------------------------ + + /// True if @p data starts with the zstd frame magic. + inline bool LooksLikeZstd(const void* data, size_t size) { + return data != nullptr && size >= sizeof(kZstdMagic) && + std::memcmp(data, kZstdMagic, sizeof(kZstdMagic)) == 0; + } + + inline bool LooksLikeZstd(std::string_view bytes) { + return LooksLikeZstd(bytes.data(), bytes.size()); + } + + /// The writer's rule for attempting compression at all: enabled and not tiny. + inline bool ShouldCompress(size_t payload_size, bool enabled) { + return enabled && payload_size >= kCompressMinBytes; + } + + /// zstd at @p level; returns @p payload unchanged when zstd fails or does not + /// shrink it (readers accept either form). Defined in vtx_common + /// (vtx_replay_framing.cpp) so this header carries no zstd include. + std::string Compress(std::string payload, int8_t level); + + /// ShouldCompress + Compress: the complete zstd-if-beneficial rule. + inline std::string CompressIfBeneficial(std::string payload, bool enabled = true, + int8_t level = kDefaultCompressionLevel) { + if (!ShouldCompress(payload.size(), enabled)) { + return payload; + } + return Compress(std::move(payload), level); + } + + // ------------------------------------------------------------------ + // Checksums + // ------------------------------------------------------------------ + + /// Seek-table checksum of a chunk: xxHash64 over the on-disk payload (after the size prefix). + inline uint64_t PayloadChecksum(const void* data, size_t size) { + return XXH3_64bits(data, size); + } + + inline uint64_t PayloadChecksum(std::string_view payload) { + return PayloadChecksum(payload.data(), payload.size()); + } + + // ------------------------------------------------------------------ + // Size prefixes and framed blocks + // ------------------------------------------------------------------ + + using SizePrefix = std::array; + + /// The u32 prefix exactly as it is written (host byte order). + inline SizePrefix EncodeSizePrefix(uint32_t size) { + SizePrefix out {}; + std::memcpy(out.data(), &size, sizeof(size)); + return out; + } + + inline uint32_t DecodeSizePrefix(const char* bytes) { + uint32_t size = 0; + std::memcpy(&size, bytes, sizeof(size)); + return size; + } + + /// Bytes a chunk occupies on disk: prefix + payload (what ChunkIndexEntry::chunk_size_bytes stores). + inline uint32_t ChunkSizeOnDisk(size_t payload_size) { + return static_cast(payload_size + kSizePrefixSize); + } + + /// [u32 size][payload] + inline std::string ChunkBlock(std::string_view payload) { + std::string block; + block.reserve(kSizePrefixSize + payload.size()); + const SizePrefix prefix = EncodeSizePrefix(static_cast(payload.size())); + block.append(prefix.data(), prefix.size()); + block.append(payload.data(), payload.size()); + return block; + } + + /// [magic][u32 size][payload] + inline std::string HeaderBlock(std::string_view magic, std::string_view payload) { + std::string block; + block.reserve(magic.size() + kSizePrefixSize + payload.size()); + block.append(magic.data(), magic.size()); + const SizePrefix prefix = EncodeSizePrefix(static_cast(payload.size())); + block.append(prefix.data(), prefix.size()); + block.append(payload.data(), payload.size()); + return block; + } + + /// [u32 footer_size][magic] -- the bytes that follow the footer payload. + inline std::string FooterTrailer(uint32_t payload_size, std::string_view magic) { + std::string trailer; + trailer.reserve(kSizePrefixSize + magic.size()); + const SizePrefix prefix = EncodeSizePrefix(payload_size); + trailer.append(prefix.data(), prefix.size()); + trailer.append(magic.data(), magic.size()); + return trailer; + } + + /// [payload][u32 size][magic] + inline std::string FooterBlock(std::string_view payload, std::string_view magic) { + std::string block; + block.reserve(payload.size() + kSizePrefixSize + magic.size()); + block.append(payload.data(), payload.size()); + block += FooterTrailer(static_cast(payload.size()), magic); + return block; + } + + // ------------------------------------------------------------------ + // Locating the blocks of a finished file (rewriters copy bytes by range) + // ------------------------------------------------------------------ + + struct FileLayout { + uint64_t file_size = 0; + uint64_t header_end = 0; ///< First byte after [magic][u32][header]; chunks start here. + uint64_t footer_offset = 0; ///< First byte of the footer payload. + uint32_t footer_size = 0; ///< Footer payload bytes (before the trailer). + bool footer_compressed = false; + }; + + /// Size of the stream in bytes; restores the read position. + inline uint64_t StreamSize(std::istream& in) { + in.clear(); + const std::streampos prev = in.tellg(); + in.seekg(0, std::ios::end); + const std::streamoff end = in.tellg(); + in.clear(); + in.seekg(prev); + return end < 0 ? 0 : static_cast(end); + } + + inline bool ReadAt(std::istream& in, uint64_t offset, void* dst, size_t size) { + in.clear(); + in.seekg(static_cast(offset)); + return static_cast(in.read(static_cast(dst), static_cast(size))); + } + + /// True if the @p available bytes at @p offset start with the zstd magic. + inline bool IsZstdAt(std::istream& in, uint64_t offset, uint64_t available) { + unsigned char magic[sizeof(kZstdMagic)]; + if (available < sizeof(magic) || !ReadAt(in, offset, magic, sizeof(magic))) { + return false; + } + return LooksLikeZstd(magic, sizeof(magic)); + } + + /// Validates [magic][u32 header_size] at offset 0 and yields where the chunks begin. + inline bool ProbeHeader(std::istream& in, std::string_view expected_magic, uint64_t file_size, + uint64_t& header_end, std::string& error) { + if (file_size < expected_magic.size() + kSizePrefixSize) { + error = "The replay is too small to hold a header."; + return false; + } + std::string magic(expected_magic.size(), '\0'); + if (!ReadAt(in, 0, magic.data(), magic.size()) || magic != expected_magic) { + error = "The replay does not start with the expected magic bytes."; + return false; + } + char prefix[kSizePrefixSize]; + if (!ReadAt(in, expected_magic.size(), prefix, sizeof(prefix))) { + error = "Could not read the header size of the replay."; + return false; + } + const uint32_t header_size = DecodeSizePrefix(prefix); + const uint64_t end = expected_magic.size() + kSizePrefixSize + header_size; + if (header_size == 0 || end > file_size) { + error = "The replay header size is implausible; the file is corrupt or truncated."; + return false; + } + header_end = end; + return true; + } + + /// Validates the trailing [u32 footer_size][magic] and yields where the footer payload starts. + /// A footerless (crash-truncated) file fails here. + inline bool ProbeFooterTrailer(std::istream& in, std::string_view expected_magic, uint64_t file_size, + uint64_t& footer_offset, uint32_t& footer_size, std::string& error) { + const size_t trailer_size = kSizePrefixSize + expected_magic.size(); + if (file_size < trailer_size) { + error = "The replay is too small to hold a footer."; + return false; + } + std::string trailer(trailer_size, '\0'); + if (!ReadAt(in, file_size - trailer_size, trailer.data(), trailer.size())) { + error = "Could not read the footer trailer of the replay."; + return false; + } + if (std::string_view(trailer).substr(kSizePrefixSize) != expected_magic) { + error = "The replay has no footer (incomplete recording?). Repair it first."; + return false; + } + const uint32_t size = DecodeSizePrefix(trailer.data()); + if (size == 0 || static_cast(size) + trailer_size > file_size) { + error = "The replay footer size is implausible; the file is corrupt or truncated."; + return false; + } + footer_size = size; + footer_offset = file_size - trailer_size - size; + return true; + } + + /// Header + footer probe of a complete file, plus the footer's compression state. + inline bool ProbeLayout(std::istream& in, std::string_view expected_magic, FileLayout& out, + std::string& error) { + FileLayout layout; + layout.file_size = StreamSize(in); + if (!ProbeHeader(in, expected_magic, layout.file_size, layout.header_end, error)) { + return false; + } + if (!ProbeFooterTrailer(in, expected_magic, layout.file_size, layout.footer_offset, layout.footer_size, + error)) { + return false; + } + if (layout.footer_offset < layout.header_end) { + error = "The replay header and footer overlap; the file is corrupt."; + return false; + } + layout.footer_compressed = IsZstdAt(in, layout.footer_offset, layout.footer_size); + out = layout; + return true; + } + + /// Streams [offset, offset + bytes) from @p in to @p out through a bounded buffer. + inline bool CopyRange(std::istream& in, std::ostream& out, uint64_t offset, uint64_t bytes, + std::string& error) { + constexpr size_t kBufferSize = 4 * 1024 * 1024; + std::vector buffer( + static_cast(std::min(std::max(bytes, 1), kBufferSize))); + in.clear(); + in.seekg(static_cast(offset)); + uint64_t remaining = bytes; + while (remaining > 0) { + const size_t step = static_cast(std::min(remaining, buffer.size())); + if (!in.read(buffer.data(), static_cast(step))) { + error = "Read failed at offset " + std::to_string(offset + (bytes - remaining)) + "."; + return false; + } + if (!out.write(buffer.data(), static_cast(step))) { + error = "Write failed after " + std::to_string(bytes - remaining) + " bytes."; + return false; + } + remaining -= step; + } + return true; + } + + } // namespace Framing +} // namespace VTX diff --git a/sdk/include/vtx/reader/core/vtx_reader.h b/sdk/include/vtx/reader/core/vtx_reader.h index fa3c825..071b508 100644 --- a/sdk/include/vtx/reader/core/vtx_reader.h +++ b/sdk/include/vtx/reader/core/vtx_reader.h @@ -19,6 +19,7 @@ #include "vtx_deserializer_service.h" #include "vtx/common/vtx_diagnostics.h" +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/common/vtx_types_helpers.h" #include "vtx/common/vtx_concepts.h" @@ -760,9 +761,9 @@ namespace VTX { return {}; } - if (raw_buffer.size() <= 4) + if (raw_buffer.size() <= Framing::kSizePrefixSize) return {}; - compressed_blob = raw_buffer.substr(4); + compressed_blob = raw_buffer.substr(Framing::kSizePrefixSize); } if (stop_token.stop_requested()) diff --git a/sdk/include/vtx/transform/vtx_replay_cut.h b/sdk/include/vtx/transform/vtx_replay_cut.h new file mode 100644 index 0000000..36763c7 --- /dev/null +++ b/sdk/include/vtx/transform/vtx_replay_cut.h @@ -0,0 +1,88 @@ +/** + * @file vtx_replay_cut.h + * @brief Write a copy of a .vtx replay holding only a sub-range of its frames. + * + * @details CutReplayFile() slices a finished replay in time: the header block is + * copied byte for byte, chunks fully inside the range are copied verbatim, the two + * edge chunks are re-serialized with only the kept frames when the range starts or + * ends inside them, and the footer is rebuilt -- frame numbering rebased to 0, the + * per-frame time table sliced (tick values stay absolute), gaps and segments + * filtered and rebased, duration recomputed from the kept stamps. Timeline events + * are not carried over. + * + * const VTX::ReplayCutPlan plan = VTX::PlanCutFrames(footer, 1200, 1799); + * if (plan.valid) { + * const auto r = VTX::CutReplayFile("match.vtx", footer, plan, "match_round2.vtx"); + * if (!r.ok()) { ... r.error ... } + * } + * + * The plan is computed from the source footer so a UI can preview the resolved + * chunks/frames before anything is written; CutReplayFile() executes that plan + * against the same footer. Both FlatBuffers and Protobuf files are supported and + * the output keeps the source format. Its sibling FilterReplayFile() slices *what* + * is in every frame instead. + * + * @author Zenos Interactive + */ +#pragma once + +#include +#include + +#include "vtx/common/vtx_types.h" + +namespace VTX { + + /** + * @brief Resolved cut request. + * @details Interior chunks are copied verbatim; when the range starts or ends + * inside a chunk (trims_head / trims_tail) that edge chunk is + * re-serialized with only the kept frames and gets a fresh seek-table + * entry (size, checksum, frame range). + */ + struct ReplayCutPlan { + bool valid = false; + std::string error; ///< Set when !valid. + + int32_t first_chunk = 0; ///< Index into footer.chunk_index. + int32_t last_chunk = 0; + int32_t first_frame = 0; ///< Exact kept range, source frame numbering. + int32_t last_frame = 0; + bool trims_head = false; ///< first_chunk starts before first_frame. + bool trims_tail = false; ///< last_chunk ends after last_frame. + uint64_t chunk_bytes = 0; ///< On-disk bytes of the involved source chunks. + + int32_t FrameCount() const { return valid ? last_frame - first_frame + 1 : 0; } + }; + + /// Exact cut: keeps [start_frame, end_frame] inclusive, trimming inside the edge + /// chunks when the bounds do not fall on chunk boundaries. Bounds are clamped to + /// the replay; a start that falls in a gap between chunks snaps to the next frame. + ReplayCutPlan PlanCutFrames(const FileFooter& footer, int32_t start_frame, int32_t end_frame); + + /// Whole-chunk cut: keeps chunks [first_chunk, last_chunk] untouched. + ReplayCutPlan PlanCutChunks(const FileFooter& footer, int32_t first_chunk, int32_t last_chunk); + + struct ReplayCutResult { + std::string error; ///< Non-empty on failure (the destination is removed). + + int32_t total_frames = 0; ///< Frames in the output. + int32_t chunks_written = 0; ///< Chunks in the output. + int32_t chunks_rewritten = 0; ///< Edge chunks re-serialized (0, 1 or 2). + uint64_t output_bytes = 0; + + bool ok() const { return error.empty(); } + }; + + /** + * @brief Writes @p dest_path as the cut of @p source_path described by @p plan. + * @param footer The source footer the plan was computed from. + * @details The format is taken from the source's magic bytes. When the plan trims + * an edge chunk the source is reopened with the standard reader to + * re-serialize it; whole-chunk cuts never decode a frame. dest_path must + * differ from source_path. + */ + ReplayCutResult CutReplayFile(const std::string& source_path, const FileFooter& footer, const ReplayCutPlan& plan, + const std::string& dest_path); + +} // namespace VTX diff --git a/sdk/include/vtx/writer/core/vtx_replay_filter.h b/sdk/include/vtx/transform/vtx_replay_filter.h similarity index 97% rename from sdk/include/vtx/writer/core/vtx_replay_filter.h rename to sdk/include/vtx/transform/vtx_replay_filter.h index 7ff8935..bd7983b 100644 --- a/sdk/include/vtx/writer/core/vtx_replay_filter.h +++ b/sdk/include/vtx/transform/vtx_replay_filter.h @@ -24,8 +24,10 @@ * emptied in every frame rather than removed: readers still see it, with no * entities. * - * Requires the reader module (VTX_BUILD_READER): the source replay is read through - * the standard reader, so the function is only compiled when vtx_reader is built. + * Part of vtx_transform (VTX_BUILD_TRANSFORM): the source is read through the + * standard reader and re-serialized through the writer's formatter policies, so the + * module needs both vtx_reader and vtx_writer. Its sibling CutReplayFile() slices + * *when* (a frame range) instead of *what*. * * @author Zenos Interactive */ diff --git a/sdk/include/vtx/writer/core/vtx_replay_recovery.h b/sdk/include/vtx/transform/vtx_replay_recovery.h similarity index 88% rename from sdk/include/vtx/writer/core/vtx_replay_recovery.h rename to sdk/include/vtx/transform/vtx_replay_recovery.h index ff8bda4..972fbf8 100644 --- a/sdk/include/vtx/writer/core/vtx_replay_recovery.h +++ b/sdk/include/vtx/transform/vtx_replay_recovery.h @@ -19,6 +19,12 @@ * } * auto ctx = VTX::OpenReplayFile(path); * + * Part of vtx_transform (VTX_BUILD_TRANSFORM), next to CutReplayFile() and + * FilterReplayFile(): every tool that modifies an existing .vtx lives there, while the + * writer only produces new recordings. Repair needs no reader -- it works from the + * journal and the raw bytes through the writer's RecoveryJournal and the shared + * framing. + * * @author Zenos Interactive */ #pragma once diff --git a/sdk/include/vtx/writer/policies/sinks/file_sink.h b/sdk/include/vtx/writer/policies/sinks/file_sink.h index 7d2cb7e..48fa3f1 100644 --- a/sdk/include/vtx/writer/policies/sinks/file_sink.h +++ b/sdk/include/vtx/writer/policies/sinks/file_sink.h @@ -6,8 +6,7 @@ #include #include #include -#include -#include +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/common/vtx_concepts.h" #include "vtx/writer/policies/sinks/durable_file.h" @@ -93,9 +92,10 @@ namespace VTX { std::string header_payload = SerializerPolicy::SerializeHeader(config_.header_config, schema); NotifySerialize(serialize_start); header_payload = CompressIfBeneficial(std::move(header_payload)); - uint32_t final_size = static_cast(header_payload.size()); - TimedWrite(&final_size, sizeof(final_size)); - TimedWrite(header_payload.data(), final_size); + const Framing::SizePrefix header_prefix = + Framing::EncodeSizePrefix(static_cast(header_payload.size())); + TimedWrite(header_prefix.data(), header_prefix.size()); + TimedWrite(header_payload.data(), header_payload.size()); TimedSyncOrFlush(); // Start the crash-recovery journal only once the header is durable. If it @@ -140,10 +140,10 @@ namespace VTX { payload = CompressIfBeneficial(std::move(payload)); uint64_t current_offset = file_.Tell(); - uint32_t final_size = static_cast(payload.size()); + const Framing::SizePrefix prefix = Framing::EncodeSizePrefix(static_cast(payload.size())); - TimedWrite(&final_size, sizeof(final_size)); - TimedWrite(payload.data(), final_size); + TimedWrite(prefix.data(), prefix.size()); + TimedWrite(payload.data(), payload.size()); TimedSyncOrFlush(); ChunkIndexData indexEntry; @@ -151,8 +151,8 @@ namespace VTX { indexEntry.file_offset = current_offset; indexEntry.start_frame = start_frame; indexEntry.end_frame = total_frames - 1; - indexEntry.chunk_size_bytes = final_size + sizeof(uint32_t); - indexEntry.checksum = XXH3_64bits(payload.data(), payload.size()); + indexEntry.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + indexEntry.checksum = Framing::PayloadChecksum(payload); seek_table_.push_back(indexEntry); // Commit the chunk to the journal AFTER its bytes are durable on disk @@ -186,9 +186,8 @@ namespace VTX { footer_payload = CompressIfBeneficial(std::move(footer_payload)); TimedWrite(footer_payload.data(), footer_payload.size()); - uint32_t final_size = static_cast(footer_payload.size()); - TimedWrite(&final_size, sizeof(final_size)); - WriteBlob(SerializerPolicy::GetMagicBytes()); + WriteBlob(Framing::FooterTrailer(static_cast(footer_payload.size()), + SerializerPolicy::GetMagicBytes())); TimedSyncOrFlush(); // Clean shutdown: the footer is durable, so the recovery journal is no @@ -234,29 +233,16 @@ namespace VTX { } void WriteBlob(const std::string& data) { TimedWrite(data.data(), data.size()); } + // The shared zstd-if-beneficial rule (vtx_replay_framing.h); only the actual + // zstd call is timed for the perf observer. std::string CompressIfBeneficial(std::string payload) { - if (!config_.b_use_compression || payload.size() < 512) { + if (!Framing::ShouldCompress(payload.size(), config_.b_use_compression)) { return payload; } - const auto compression_start = std::chrono::steady_clock::now(); - size_t const max_size = ZSTD_compressBound(payload.size()); - std::string compressed_blob(max_size, '\0'); - - size_t const compressed_size = ZSTD_compress(compressed_blob.data(), max_size, payload.data(), - payload.size(), config_.compression_level); + payload = Framing::Compress(std::move(payload), config_.compression_level); NotifyCompress(compression_start); - - if (ZSTD_isError(compressed_size)) { - return payload; - } - - if (compressed_size >= payload.size()) { - return payload; - } - - compressed_blob.resize(compressed_size); - return compressed_blob; + return payload; } Config config_; diff --git a/sdk/include/vtx/writer/policies/sinks/network_sink.h b/sdk/include/vtx/writer/policies/sinks/network_sink.h index c0afd67..9c19f8e 100644 --- a/sdk/include/vtx/writer/policies/sinks/network_sink.h +++ b/sdk/include/vtx/writer/policies/sinks/network_sink.h @@ -40,9 +40,9 @@ using vtx_socklen_t = socklen_t; #include #include #include -#include #include "vtx/common/vtx_concepts.h" +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" namespace VTX { @@ -103,8 +103,9 @@ namespace VTX { std::string header_payload = SerializerPolicy::SerializeHeader(config_.header_config, schema); header_payload = CompressIfBeneficial(std::move(header_payload)); - uint32_t final_size = static_cast(header_payload.size()); - SendAll(reinterpret_cast(&final_size), sizeof(final_size)); + const Framing::SizePrefix header_prefix = + Framing::EncodeSizePrefix(static_cast(header_payload.size())); + SendAll(header_prefix.data(), header_prefix.size()); SendAll(header_payload); } @@ -117,8 +118,8 @@ namespace VTX { payload = CompressIfBeneficial(std::move(payload)); uint64_t current_offset = bytes_sent_; - uint32_t final_size = static_cast(payload.size()); - SendAll(reinterpret_cast(&final_size), sizeof(final_size)); + const Framing::SizePrefix prefix = Framing::EncodeSizePrefix(static_cast(payload.size())); + SendAll(prefix.data(), prefix.size()); SendAll(payload); ChunkIndexData entry; @@ -126,7 +127,8 @@ namespace VTX { entry.file_offset = current_offset; entry.start_frame = start_frame; entry.end_frame = total_frames - 1; - entry.chunk_size_bytes = final_size + static_cast(sizeof(uint32_t)); + entry.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + entry.checksum = Framing::PayloadChecksum(payload); // same seek-table contract as the file sink seek_table_.push_back(entry); } @@ -143,9 +145,8 @@ namespace VTX { std::string footer_payload = SerializerPolicy::SerializeFooter(seek_table_, footerData); footer_payload = CompressIfBeneficial(std::move(footer_payload)); SendAll(footer_payload); - uint32_t final_size = static_cast(footer_payload.size()); - SendAll(reinterpret_cast(&final_size), sizeof(final_size)); - SendAll(SerializerPolicy::GetMagicBytes()); + SendAll(Framing::FooterTrailer(static_cast(footer_payload.size()), + SerializerPolicy::GetMagicBytes())); vtx_close_socket(socket_); socket_ = kVtxInvalidSocket; @@ -166,19 +167,8 @@ namespace VTX { void SendAll(const std::string& data) { SendAll(data.data(), data.size()); } std::string CompressIfBeneficial(std::string payload) { - if (!config_.b_use_compression || payload.size() < 512) - return payload; - - size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), config_.compression_level); - - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) - return payload; - - compressed.resize(compressed_size); - return compressed; + return Framing::CompressIfBeneficial(std::move(payload), config_.b_use_compression, + config_.compression_level); } #ifdef _WIN32 diff --git a/sdk/src/CMakeLists.txt b/sdk/src/CMakeLists.txt index c274dc2..0b020ad 100644 --- a/sdk/src/CMakeLists.txt +++ b/sdk/src/CMakeLists.txt @@ -15,3 +15,11 @@ endif() if(VTX_BUILD_DIFFER) add_subdirectory(vtx_differ) endif() + +# Tools that modify an existing replay (repair, cut, entity filter): they read +# through vtx_reader and re-serialize through vtx_writer's policies, so it needs both. +if(VTX_BUILD_TRANSFORM AND VTX_BUILD_READER AND VTX_BUILD_WRITER) + add_subdirectory(vtx_transform) +elseif(VTX_BUILD_TRANSFORM) + message(STATUS "VTX: vtx_transform skipped (requires VTX_BUILD_READER and VTX_BUILD_WRITER).") +endif() diff --git a/sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp b/sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp new file mode 100644 index 0000000..0540bf3 --- /dev/null +++ b/sdk/src/vtx_common/src/vtx/common/vtx_replay_framing.cpp @@ -0,0 +1,25 @@ +#include "vtx/common/vtx_replay_framing.h" + +#include + +#include + +namespace VTX { + namespace Framing { + + // The one zstd call in the framing; kept out of the header so consumers that + // only include reader/common headers need no zstd headers on their path. + std::string Compress(std::string payload, int8_t level) { + const size_t max_size = ZSTD_compressBound(payload.size()); + std::string compressed(max_size, '\0'); + const size_t compressed_size = + ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), level); + if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) { + return payload; + } + compressed.resize(compressed_size); + return compressed; + } + + } // namespace Framing +} // namespace VTX diff --git a/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp b/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp index fe952e1..b99b1c8 100644 --- a/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp +++ b/sdk/src/vtx_reader/src/vtx/reader/core/vtx_deserializer_service.cpp @@ -3,20 +3,14 @@ #include #include +#include "vtx/common/vtx_replay_framing.h" #include "vtx_schema.pb.h" using namespace VTX; std::string VTX::ReplayUnpacker::Decompress(const std::string& compressed_data) { - //zstd includes a magic number of 4 bytes, if is smaller than that is not a valid zstd buffer - if (compressed_data.size() < 4) { - return compressed_data; - } - - //get magic number - const uint8_t* bytes = reinterpret_cast(compressed_data.data()); - bool is_zstd = (bytes[0] == 0x28 && bytes[1] == 0xB5 && bytes[2] == 0x2F && bytes[3] == 0xFD); - - if (!is_zstd) { + // Blobs are stored raw when zstd would not help; the zstd magic decides (one + // definition for every producer and consumer in vtx_replay_framing.h). + if (!Framing::LooksLikeZstd(compressed_data)) { return compressed_data; } diff --git a/sdk/src/vtx_transform/CMakeLists.txt b/sdk/src/vtx_transform/CMakeLists.txt new file mode 100644 index 0000000..91de2dc --- /dev/null +++ b/sdk/src/vtx_transform/CMakeLists.txt @@ -0,0 +1,67 @@ +project(vtx_transform) + +# ============================================================================== +# vtx_transform -- tools that modify an existing .vtx (repair, cut, entity filter). +# ============================================================================== +# Sits ABOVE vtx_reader and vtx_writer: cut and filter read a finished .vtx +# through the reader and re-serialize through the writer's formatter policies; +# repair reconstructs a crashed one from its recovery journal (writer sink +# headers, no reader). All of them frame bytes with the shared +# vtx/common/vtx_replay_framing.h. Keeping them in their own module leaves reader +# and writer independent siblings on vtx_common -- the writer only produces new +# recordings. Requires VTX_BUILD_READER and VTX_BUILD_WRITER (enforced in sdk/src). +# ============================================================================== + +# ------------------------------------------------------------------------------ +# 1. Source files and headers +# ------------------------------------------------------------------------------ +file(GLOB_RECURSE HEADERS CONFIGURE_DEPENDS + "${VTX_SDK_SOURCE_DIR}/sdk/include/vtx/transform/*.h" + "${VTX_SDK_SOURCE_DIR}/sdk/include/vtx/transform/*.hpp" +) + +file(GLOB_RECURSE SOURCES CONFIGURE_DEPENDS + "${CMAKE_CURRENT_SOURCE_DIR}/*.cpp" +) + +# ------------------------------------------------------------------------------ +# 2. Target and deps +# ------------------------------------------------------------------------------ +# Library type (STATIC/SHARED) follows BUILD_SHARED_LIBS (VTX_BUILD_SHARED). +add_library(vtx_transform ${SOURCES} ${HEADERS}) + +target_include_directories(vtx_transform + PRIVATE + "${VTX_COMMON_GENERATED_DIR}" +) + +# The public headers only need vtx_common types; reader and writer are +# implementation details (they still propagate as link dependencies). +target_link_libraries(vtx_transform + PUBLIC + vtx_common + PRIVATE + vtx_reader + vtx_writer + $ + $ + $ +) + +# See note in vtx_reader/CMakeLists.txt -- Windows shared DLLs duplicate the +# generated protobuf/flatbuffers objects into every module that references them. +if(BUILD_SHARED_LIBS AND WIN32) + target_link_libraries(vtx_transform PRIVATE $) +endif() + +# ------------------------------------------------------------------------------ +# 3. VS filters +# ------------------------------------------------------------------------------ +source_group(TREE "${VTX_SDK_SOURCE_DIR}/sdk/include" PREFIX "Header Files" FILES ${HEADERS}) +source_group(TREE "${CMAKE_CURRENT_SOURCE_DIR}" PREFIX "Source Files" FILES ${SOURCES}) + +install(TARGETS vtx_transform EXPORT VTXTargets + ARCHIVE DESTINATION lib + LIBRARY DESTINATION lib + RUNTIME DESTINATION bin +) diff --git a/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp new file mode 100644 index 0000000..93f56f3 --- /dev/null +++ b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_cut.cpp @@ -0,0 +1,364 @@ +#include "vtx/transform/vtx_replay_cut.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers +#include "vtx_schema.pb.h" + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" +#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" +#include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" + +namespace VTX { + + namespace { + + // Duration of a sliced tick table (first..last nonzero stamp), or negative when + // the table cannot answer. + double DurationFromTicks(const std::vector& ticks) { + int64_t first = 0; + int64_t last = 0; + for (const int64_t t : ticks) { + if (t != 0) { + if (first == 0) { + first = t; + } + last = t; + } + } + if (first == 0 || last < first) { + return -1.0; + } + return static_cast(last - first) / static_cast(GameTime::TICKS_PER_SECOND); + } + + ReplayCutPlan PlanForChunkSpan(const FileFooter& footer, int32_t first_chunk, int32_t last_chunk, + int32_t first_frame, int32_t last_frame) { + ReplayCutPlan plan; + plan.first_chunk = first_chunk; + plan.last_chunk = last_chunk; + plan.first_frame = first_frame; + plan.last_frame = last_frame; + plan.trims_head = footer.chunk_index[static_cast(first_chunk)].start_frame < first_frame; + plan.trims_tail = footer.chunk_index[static_cast(last_chunk)].end_frame > last_frame; + for (int32_t i = first_chunk; i <= last_chunk; ++i) { + plan.chunk_bytes += footer.chunk_index[static_cast(i)].chunk_size_bytes; + } + plan.valid = true; + return plan; + } + + // Frames [kept_lo, kept_hi] of the source re-serialized as one chunk through the + // shared framing: per-chunk zstd decision mirrored from the source chunk, fresh + // seek-table entry. + template + bool WriteRewrittenChunk(IVtxReaderFacade& reader, std::ofstream& out, int32_t kept_lo, int32_t kept_hi, + int32_t new_chunk_index, int32_t rebase_frame, bool compress, + ChunkIndexData& entry_out, std::string& error) { + std::vector> frames; + frames.reserve(static_cast(kept_hi - kept_lo + 1)); + for (int32_t frame_index = kept_lo; frame_index <= kept_hi; ++frame_index) { + const Frame* source_frame = reader.GetFrameSync(frame_index); + if (!source_frame) { + error = "Could not read frame " + std::to_string(frame_index) + " from the source replay."; + return false; + } + frames.push_back(Policy::FromNative(Frame(*source_frame))); + } + + std::string payload = Policy::SerializeChunk(frames, new_chunk_index, compress); + payload = Framing::CompressIfBeneficial(std::move(payload), compress, Framing::kDefaultCompressionLevel); + + entry_out.chunk_index = new_chunk_index; + entry_out.file_offset = static_cast(out.tellp()); + entry_out.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); + entry_out.start_frame = kept_lo - rebase_frame; + entry_out.end_frame = kept_hi - rebase_frame; + entry_out.checksum = Framing::PayloadChecksum(payload); + + const std::string block = Framing::ChunkBlock(payload); + if (!out.write(block.data(), static_cast(block.size()))) { + error = "Write failed for rewritten chunk " + std::to_string(new_chunk_index) + "."; + return false; + } + return true; + } + + template + void RunCut(const std::string& source_path, const FileFooter& footer, const ReplayCutPlan& plan, + const std::string& dest_path, ReplayCutResult& result) { + namespace fs = std::filesystem; + std::string error; + + // Edge chunks that start/end inside the kept range are re-serialized frame + // by frame; open a private reader for that (a caller's reader may be busy on + // its own thread). Whole-chunk cuts never decode a frame. + ReaderContext reader_context; + if (plan.trims_head || plan.trims_tail) { + reader_context = OpenReplayFile(source_path); + if (!reader_context.Loaded()) { + result.error = "Could not reopen the source replay for edge-chunk rewriting: " + + reader_context.GetError().message; + return; + } + if (!reader_context.WaitUntilReady()) { + result.error = "The source replay failed to load: " + reader_context.GetReadyError().message; + return; + } + } + + std::ifstream in(source_path, std::ios::binary); + if (!in) { + result.error = "Could not open the source replay: " + source_path; + return; + } + Framing::FileLayout layout; + if (!Framing::ProbeLayout(in, Policy::GetMagicBytes(), layout, error)) { + result.error = error; + return; + } + for (int32_t i = plan.first_chunk; i <= plan.last_chunk; ++i) { + const ChunkIndexEntry& chunk = footer.chunk_index[static_cast(i)]; + if (chunk.chunk_size_bytes <= Framing::kSizePrefixSize || chunk.file_offset < layout.header_end || + chunk.file_offset + chunk.chunk_size_bytes > layout.footer_offset) { + result.error = "The source replay seek table is inconsistent with the file layout (chunk " + + std::to_string(chunk.chunk_index) + ")."; + return; + } + } + + std::ofstream out(dest_path, std::ios::binary | std::ios::trunc); + if (!out) { + result.error = "Could not create the output file: " + dest_path; + return; + } + const auto abort_output = [&](std::string message) { + out.close(); + std::error_code ec; + fs::remove(dest_path, ec); + result.error = std::move(message); + }; + + // 1) Header block, verbatim: it carries no frame-dependent data, so copying + // it keeps the schema and recording metadata bit-identical. + if (!Framing::CopyRange(in, out, 0, layout.header_end, error)) { + abort_output(error); + return; + } + + // 2) Chunks. Whole chunks are copied verbatim (their seek entry is rebased); + // partial edge chunks are rebuilt with only the kept frames. + std::vector seek_table; + seek_table.reserve(static_cast(plan.last_chunk - plan.first_chunk + 1)); + for (int32_t i = plan.first_chunk; i <= plan.last_chunk; ++i) { + const ChunkIndexEntry& entry = footer.chunk_index[static_cast(i)]; + const int32_t kept_lo = std::max(entry.start_frame, plan.first_frame); + const int32_t kept_hi = std::min(entry.end_frame, plan.last_frame); + const int32_t new_index = static_cast(seek_table.size()); + + if (kept_lo == entry.start_frame && kept_hi == entry.end_frame) { + ChunkIndexData rebased; + rebased.chunk_index = new_index; + rebased.file_offset = static_cast(out.tellp()); + rebased.chunk_size_bytes = entry.chunk_size_bytes; + rebased.start_frame = entry.start_frame - plan.first_frame; + rebased.end_frame = entry.end_frame - plan.first_frame; + rebased.checksum = entry.checksum; + if (!Framing::CopyRange(in, out, entry.file_offset, entry.chunk_size_bytes, error)) { + abort_output(error); + return; + } + seek_table.push_back(rebased); + } else { + const bool compress = Framing::IsZstdAt(in, entry.file_offset + Framing::kSizePrefixSize, + entry.chunk_size_bytes - Framing::kSizePrefixSize); + ChunkIndexData rewritten; + if (!WriteRewrittenChunk(*reader_context.reader, out, kept_lo, kept_hi, new_index, + plan.first_frame, compress, rewritten, error)) { + abort_output(error); + return; + } + seek_table.push_back(rewritten); + ++result.chunks_rewritten; + } + } + + // 3) Footer: time table sliced to the kept frames (tick values stay absolute, + // only frame indices are rebased), duration recomputed, fresh seek table. + const auto slice_ticks = [&](const std::vector& ticks) { + std::vector sliced; + const size_t begin = std::min(static_cast(plan.first_frame), ticks.size()); + const size_t end = std::min(static_cast(plan.last_frame) + 1, ticks.size()); + sliced.reserve(end - begin); + for (size_t i = begin; i < end; ++i) { + sliced.push_back(static_cast(ticks[i])); + } + return sliced; + }; + const auto slice_frame_indices = [&](const std::vector& values) { + std::vector sliced; + for (const uint32_t value : values) { + const int32_t frame = static_cast(value); + if (frame >= plan.first_frame && frame <= plan.last_frame) { + sliced.push_back(frame - plan.first_frame); + } + } + return sliced; + }; + + const std::vector game_times = slice_ticks(footer.times.game_time); + const std::vector created_utc = slice_ticks(footer.times.created_utc); + const std::vector gaps = slice_frame_indices(footer.times.gaps); + const std::vector segments = slice_frame_indices(footer.times.segments); + + SessionFooter session_footer; + session_footer.total_frames = plan.FrameCount(); + double duration = DurationFromTicks(created_utc); + if (duration < 0.0) { + duration = DurationFromTicks(game_times); + } + if (duration < 0.0 && footer.total_frames > 0) { + duration = + static_cast(footer.duration_seconds) * session_footer.total_frames / footer.total_frames; + } + session_footer.duration_seconds = std::max(duration, 0.0); + session_footer.game_times = &game_times; + session_footer.created_utc = &created_utc; + session_footer.gaps = &gaps; + session_footer.segments = &segments; + + std::string footer_payload = Policy::SerializeFooter(seek_table, session_footer); + footer_payload = Framing::CompressIfBeneficial(std::move(footer_payload), layout.footer_compressed, + Framing::kDefaultCompressionLevel); + const std::string footer_block = Framing::FooterBlock(footer_payload, Policy::GetMagicBytes()); + if (!out.write(footer_block.data(), static_cast(footer_block.size()))) { + abort_output("Write failed while finishing the footer."); + return; + } + out.flush(); + if (!out) { + abort_output("Flush failed for the output file."); + return; + } + out.close(); + + result.total_frames = session_footer.total_frames; + result.chunks_written = static_cast(seek_table.size()); + std::error_code ec; + result.output_bytes = fs::file_size(dest_path, ec); + } + + } // namespace + + ReplayCutPlan PlanCutFrames(const FileFooter& footer, int32_t start_frame, int32_t end_frame) { + ReplayCutPlan plan; + if (footer.chunk_index.empty() || footer.total_frames <= 0) { + plan.error = "The replay has no chunk index."; + return plan; + } + start_frame = std::clamp(start_frame, 0, footer.total_frames - 1); + end_frame = std::clamp(end_frame, 0, footer.total_frames - 1); + if (end_frame < start_frame) { + plan.error = "End of range is before its start."; + return plan; + } + + int32_t first_chunk = -1; + int32_t last_chunk = -1; + for (int32_t i = 0; i < static_cast(footer.chunk_index.size()); ++i) { + const auto& entry = footer.chunk_index[static_cast(i)]; + if (first_chunk < 0 && start_frame <= entry.end_frame) { + first_chunk = i; + } + if (end_frame >= entry.start_frame) { + last_chunk = i; + } + } + if (first_chunk < 0 || last_chunk < first_chunk) { + plan.error = "The requested range does not overlap any chunk."; + return plan; + } + // The exact bounds are clamped into the covered span (a start that falls in a + // gap between chunks snaps to the first covered frame). + const int32_t lo = std::max(start_frame, footer.chunk_index[static_cast(first_chunk)].start_frame); + const int32_t hi = std::min(end_frame, footer.chunk_index[static_cast(last_chunk)].end_frame); + return PlanForChunkSpan(footer, first_chunk, last_chunk, lo, hi); + } + + ReplayCutPlan PlanCutChunks(const FileFooter& footer, int32_t first_chunk, int32_t last_chunk) { + ReplayCutPlan plan; + if (footer.chunk_index.empty()) { + plan.error = "The replay has no chunk index."; + return plan; + } + const int32_t max_chunk = static_cast(footer.chunk_index.size()) - 1; + first_chunk = std::clamp(first_chunk, 0, max_chunk); + last_chunk = std::clamp(last_chunk, 0, max_chunk); + if (last_chunk < first_chunk) { + plan.error = "End chunk is before start chunk."; + return plan; + } + return PlanForChunkSpan(footer, first_chunk, last_chunk, + footer.chunk_index[static_cast(first_chunk)].start_frame, + footer.chunk_index[static_cast(last_chunk)].end_frame); + } + + ReplayCutResult CutReplayFile(const std::string& source_path, const FileFooter& footer, const ReplayCutPlan& plan, + const std::string& dest_path) { + namespace fs = std::filesystem; + ReplayCutResult result; + + if (!plan.valid) { + result.error = plan.error.empty() ? "Invalid cut plan." : plan.error; + return result; + } + if (source_path.empty() || dest_path.empty()) { + result.error = "Source and destination paths are required."; + return result; + } + if (plan.first_chunk < 0 || plan.last_chunk < plan.first_chunk || + static_cast(plan.last_chunk) >= footer.chunk_index.size()) { + result.error = "The cut plan does not match the footer's chunk index."; + return result; + } + std::error_code ec; + if (!fs::is_regular_file(source_path, ec)) { + result.error = "Source replay does not exist: " + source_path; + return result; + } + const fs::path source_abs = fs::weakly_canonical(fs::path(source_path), ec); + const fs::path dest_abs = fs::weakly_canonical(fs::path(dest_path), ec); + if (source_abs == dest_abs || (fs::exists(dest_path, ec) && fs::equivalent(source_path, dest_path, ec))) { + result.error = "Destination must be a different file than the source."; + return result; + } + + // The format is decided by the source's leading magic. + std::string magic(Framing::kMagicSize, '\0'); + { + std::ifstream in(source_path, std::ios::binary); + if (!in || !in.read(magic.data(), static_cast(magic.size()))) { + result.error = "Could not read the source replay: " + source_path; + return result; + } + } + if (magic == FlatBuffersVtxPolicy::GetMagicBytes()) { + RunCut(source_path, footer, plan, dest_path, result); + } else if (magic == ProtobufVtxPolicy::GetMagicBytes()) { + RunCut(source_path, footer, plan, dest_path, result); + } else { + result.error = "The source replay does not start with a known VTX magic."; + } + return result; + } + +} // namespace VTX diff --git a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_filter.cpp similarity index 78% rename from sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp rename to sdk/src/vtx_transform/src/vtx/transform/vtx_replay_filter.cpp index 4531dcc..8941333 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_filter.cpp +++ b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_filter.cpp @@ -1,12 +1,10 @@ -#include "vtx/writer/core/vtx_replay_filter.h" +#include "vtx/transform/vtx_replay_filter.h" #include #include #include #include #include -#include -#include #include #include #include @@ -15,12 +13,10 @@ #include #include -#include -#include - #include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers #include "vtx_schema.pb.h" +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/reader/core/vtx_reader_facade.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" @@ -30,13 +26,6 @@ namespace VTX { namespace { - // Mirrors ChunkedFileSink::CompressIfBeneficial with the writer defaults: - // level 10, tiny payloads left raw, raw kept when zstd does not shrink it. - constexpr int kCompressionLevel = 10; - constexpr size_t kCompressMinBytes = 512; - // zstd frame magic, little-endian on disk: 28 B5 2F FD. - constexpr unsigned char kZstdMagic[4] = {0x28, 0xB5, 0x2F, 0xFD}; - char ToLowerAscii(char c) { return static_cast(std::tolower(static_cast(c))); } @@ -126,118 +115,10 @@ namespace VTX { } // ------------------------------------------------------------------ - // File plumbing (mirrors ChunkedFileSink framing: [magic][u32 header] - // [u32 chunk]* [footer][u32][magic]; xxHash64 over the on-disk payload). + // Rewrite. File framing (size prefixes, footer trailer, zstd rule, chunk + // checksums, layout probe) is the shared vtx_replay_framing.h. // ------------------------------------------------------------------ - struct SourceLayout { - uint64_t file_size = 0; - uint64_t header_end = 0; ///< First byte after the header block. - uint64_t footer_offset = 0; - bool footer_compressed = false; - }; - - bool ReadBytesAt(std::ifstream& in, uint64_t offset, void* dst, size_t size) { - in.clear(); - in.seekg(static_cast(offset)); - return static_cast(in.read(static_cast(dst), static_cast(size))); - } - - bool IsZstdAt(std::ifstream& in, uint64_t offset, uint64_t available) { - unsigned char magic[4]; - if (available < sizeof(magic) || !ReadBytesAt(in, offset, magic, sizeof(magic))) { - return false; - } - return std::memcmp(magic, kZstdMagic, sizeof(magic)) == 0; - } - - bool ProbeSource(std::ifstream& in, const std::string& expected_magic, SourceLayout& layout, - std::string& error) { - in.clear(); - in.seekg(0, std::ios::end); - const std::streamoff end = in.tellg(); - if (end < 0) { - error = "Could not determine the source replay size."; - return false; - } - layout.file_size = static_cast(end); - const size_t magic_size = expected_magic.size(); - // magic + header size + footer size + trailing magic. - if (layout.file_size < magic_size * 2 + sizeof(uint32_t) * 2) { - error = "The source replay is too small to be a valid .vtx file."; - return false; - } - - std::string magic(magic_size, '\0'); - if (!ReadBytesAt(in, 0, magic.data(), magic_size) || magic != expected_magic) { - error = "The source replay does not start with the expected magic bytes."; - return false; - } - uint32_t header_size = 0; - if (!ReadBytesAt(in, magic_size, &header_size, sizeof(header_size))) { - error = "Could not read the header size of the source replay."; - return false; - } - layout.header_end = magic_size + sizeof(uint32_t) + header_size; - - std::string trailing(magic_size, '\0'); - if (!ReadBytesAt(in, layout.file_size - magic_size, trailing.data(), magic_size) || - trailing != expected_magic) { - error = "The source replay has no footer (incomplete recording?). Repair it first."; - return false; - } - uint32_t footer_size = 0; - if (!ReadBytesAt(in, layout.file_size - magic_size - sizeof(uint32_t), &footer_size, sizeof(footer_size))) { - error = "Could not read the footer size of the source replay."; - return false; - } - const uint64_t footer_block = static_cast(footer_size) + sizeof(uint32_t) + magic_size; - if (footer_block > layout.file_size || layout.file_size - footer_block < layout.header_end) { - error = "The source replay header and footer overlap; the file is corrupt."; - return false; - } - layout.footer_offset = layout.file_size - footer_block; - layout.footer_compressed = IsZstdAt(in, layout.footer_offset, footer_size); - return true; - } - - bool CopyRange(std::ifstream& in, std::ofstream& out, uint64_t offset, uint64_t bytes, std::string& error) { - constexpr size_t kBufferSize = 4 * 1024 * 1024; - std::vector buffer( - static_cast(std::min(std::max(bytes, 1), kBufferSize))); - in.clear(); - in.seekg(static_cast(offset)); - uint64_t remaining = bytes; - while (remaining > 0) { - const size_t step = static_cast(std::min(remaining, buffer.size())); - if (!in.read(buffer.data(), static_cast(step))) { - error = "Read failed at offset " + std::to_string(offset + (bytes - remaining)) + "."; - return false; - } - if (!out.write(buffer.data(), static_cast(step))) { - error = "Write failed while copying the header block."; - return false; - } - remaining -= step; - } - return true; - } - - std::string CompressIfBeneficial(std::string payload) { - if (payload.size() < kCompressMinBytes) { - return payload; - } - const size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - const size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), kCompressionLevel); - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) { - return payload; - } - compressed.resize(compressed_size); - return compressed; - } - template void RunFilter(const std::string& source_path, const std::string& dest_path, IVtxReaderFacade& reader, const FileFooter& footer, const ReplayFilterMatcher& matcher, bool prune_events, @@ -253,8 +134,8 @@ namespace VTX { fail("Could not open the source replay: " + source_path); return; } - SourceLayout layout; - if (!ProbeSource(in, Policy::GetMagicBytes(), layout, error)) { + Framing::FileLayout layout; + if (!Framing::ProbeLayout(in, Policy::GetMagicBytes(), layout, error)) { fail(error); return; } @@ -265,7 +146,7 @@ namespace VTX { std::sort(chunks.begin(), chunks.end(), [](const ChunkIndexEntry& a, const ChunkIndexEntry& b) { return a.start_frame < b.start_frame; }); for (const ChunkIndexEntry& chunk : chunks) { - if (chunk.end_frame < chunk.start_frame || chunk.chunk_size_bytes <= sizeof(uint32_t) || + if (chunk.end_frame < chunk.start_frame || chunk.chunk_size_bytes <= Framing::kSizePrefixSize || chunk.file_offset < layout.header_end || chunk.file_offset + chunk.chunk_size_bytes > layout.footer_offset) { fail("The source replay seek table is inconsistent with the file layout (chunk " + @@ -292,7 +173,7 @@ namespace VTX { // 1) Header block, verbatim (schema, uuid, name, recording timestamp, // metadata). Nothing in it depends on the entity payload. - if (!CopyRange(in, out, 0, layout.header_end, error)) { + if (!Framing::CopyRange(in, out, 0, layout.header_end, error)) { abort_output(error); return; } @@ -347,24 +228,23 @@ namespace VTX { frames.push_back(Policy::FromNative(std::move(frame))); } - const bool compress = - IsZstdAt(in, chunk.file_offset + sizeof(uint32_t), chunk.chunk_size_bytes - sizeof(uint32_t)); + // Per-chunk zstd decision mirrored from the source chunk's bytes. + const bool compress = Framing::IsZstdAt(in, chunk.file_offset + Framing::kSizePrefixSize, + chunk.chunk_size_bytes - Framing::kSizePrefixSize); std::string payload = Policy::SerializeChunk(frames, i, compress); - if (compress) { - payload = CompressIfBeneficial(std::move(payload)); - } + payload = + Framing::CompressIfBeneficial(std::move(payload), compress, Framing::kDefaultCompressionLevel); ChunkIndexData entry; entry.chunk_index = i; entry.file_offset = static_cast(out.tellp()); - entry.chunk_size_bytes = static_cast(payload.size() + sizeof(uint32_t)); + entry.chunk_size_bytes = Framing::ChunkSizeOnDisk(payload.size()); entry.start_frame = chunk.start_frame; entry.end_frame = chunk.end_frame; - entry.checksum = XXH3_64bits(payload.data(), payload.size()); + entry.checksum = Framing::PayloadChecksum(payload); - const uint32_t payload_size = static_cast(payload.size()); - if (!out.write(reinterpret_cast(&payload_size), sizeof(payload_size)) || - !out.write(payload.data(), static_cast(payload.size()))) { + const std::string block = Framing::ChunkBlock(payload); + if (!out.write(block.data(), static_cast(block.size()))) { abort_output("Write failed for chunk " + std::to_string(i) + "."); return; } @@ -408,14 +288,10 @@ namespace VTX { session_footer.events = &events; std::string footer_payload = Policy::SerializeFooter(seek_table, session_footer); - if (layout.footer_compressed) { - footer_payload = CompressIfBeneficial(std::move(footer_payload)); - } - const uint32_t footer_size = static_cast(footer_payload.size()); - const std::string magic = Policy::GetMagicBytes(); - if (!out.write(footer_payload.data(), static_cast(footer_payload.size())) || - !out.write(reinterpret_cast(&footer_size), sizeof(footer_size)) || - !out.write(magic.data(), static_cast(magic.size()))) { + footer_payload = Framing::CompressIfBeneficial(std::move(footer_payload), layout.footer_compressed, + Framing::kDefaultCompressionLevel); + const std::string footer_block = Framing::FooterBlock(footer_payload, Policy::GetMagicBytes()); + if (!out.write(footer_block.data(), static_cast(footer_block.size()))) { abort_output("Write failed while finishing the footer."); return; } diff --git a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_recovery.cpp b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_recovery.cpp similarity index 79% rename from sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_recovery.cpp rename to sdk/src/vtx_transform/src/vtx/transform/vtx_replay_recovery.cpp index 9f4e98f..78caf1b 100644 --- a/sdk/src/vtx_writer/src/vtx/writer/core/vtx_replay_recovery.cpp +++ b/sdk/src/vtx_transform/src/vtx/transform/vtx_replay_recovery.cpp @@ -1,16 +1,14 @@ -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" #include #include #include -#include #include #include #include #include -#include -#include +#include "vtx/common/vtx_replay_framing.h" #include "vtx/common/vtx_types.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" #include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" @@ -31,63 +29,33 @@ namespace VTX { // crash-truncated (footerless) file almost never does. Used to detect a // complete file with only a leftover journal, so we DON'T rewrite its footer. bool HasValidTrailingFooter(const std::string& path, const std::string& magic, int64_t file_size) { - if (file_size < 8) + if (file_size < static_cast(Framing::kFooterTrailerSize)) return false; std::ifstream in(path, std::ios::binary); if (!in.is_open()) return false; - in.seekg(file_size - 8); - char buf[8]; - in.read(buf, 8); - if (in.gcount() != 8) - return false; + uint64_t footer_offset = 0; uint32_t footer_size = 0; - std::memcpy(&footer_size, buf, sizeof(footer_size)); // host-native, matches the sink - if (std::string(buf + 4, 4) != magic) - return false; - return footer_size > 0 && static_cast(footer_size) + 8 <= file_size; - } - - // Mirror of ChunkedFileSink::CompressIfBeneficial, so a footer synthesized by - // repair is byte-identical to one written by a clean Close() under the same - // (journaled) compression settings. - std::string CompressIfBeneficial(std::string payload, bool use_compression, int8_t level) { - if (!use_compression || payload.size() < 512) - return payload; - const size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - const size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), level); - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) - return payload; - compressed.resize(compressed_size); - return compressed; + std::string ignored; + return Framing::ProbeFooterTrailer(in, magic, static_cast(file_size), footer_offset, footer_size, + ignored); } - // Byte offset where chunks begin (magic(4) + u32 header_size + header), or -1 if - // the header framing is not intact. + // Byte offset where chunks begin (magic + u32 header_size + header), or -1 if the + // header framing is not intact. int64_t ReadHeaderEnd(std::ifstream& in, const std::string& expect_magic, int64_t file_size) { - in.clear(); - in.seekg(0); - char magic[4]; - in.read(magic, 4); - if (in.gcount() != 4 || std::string(magic, 4) != expect_magic) - return -1; - uint32_t header_size = 0; - in.read(reinterpret_cast(&header_size), sizeof(header_size)); - if (in.gcount() != static_cast(sizeof(header_size))) - return -1; - const int64_t header_end = 8 + static_cast(header_size); - if (header_size == 0 || header_end > file_size) + uint64_t header_end = 0; + std::string ignored; + if (!Framing::ProbeHeader(in, expect_magic, static_cast(file_size), header_end, ignored)) return -1; - return header_end; + return static_cast(header_end); } template RepairResult RepairImpl(const std::string& path, const RecoveryJournal::Parsed& journal) { RepairResult r; const int64_t file_size = FileSizeOf(path); - if (file_size < 8) { + if (file_size < static_cast(Framing::kMagicSize + Framing::kSizePrefixSize)) { r.error = "main file too small or missing"; return r; } @@ -110,7 +78,7 @@ namespace VTX { for (const auto& c : journal.chunks) { const int64_t off = c.file_offset; const int64_t end = off + static_cast(c.chunk_size_bytes); - if (off < header_end || c.chunk_size_bytes <= sizeof(uint32_t) || end > file_size) + if (off < header_end || c.chunk_size_bytes <= Framing::kSizePrefixSize || end > file_size) break; // torn tail / beyond EOF std::string bytes; @@ -121,9 +89,9 @@ namespace VTX { if (in.gcount() != static_cast(c.chunk_size_bytes)) break; - // The stored checksum covers the payload after the 4-byte length prefix. - const uint64_t hash = - XXH3_64bits(bytes.data() + sizeof(uint32_t), c.chunk_size_bytes - sizeof(uint32_t)); + // The stored checksum covers the payload after the size prefix. + const uint64_t hash = Framing::PayloadChecksum(bytes.data() + Framing::kSizePrefixSize, + c.chunk_size_bytes - Framing::kSizePrefixSize); if (c.checksum != 0 && hash != c.checksum) break; // corrupt chunk -> stop, drop it and everything after @@ -166,8 +134,8 @@ namespace VTX { if (f->index != expected || f->payload.empty()) break; // gap / duplicate / empty -> stop at the first hole const uint64_t offset = out.Tell(); - const uint32_t size = static_cast(f->payload.size()); - out.Write(&size, sizeof(size)); + const Framing::SizePrefix prefix = Framing::EncodeSizePrefix(static_cast(f->payload.size())); + out.Write(prefix.data(), prefix.size()); out.Write(f->payload.data(), f->payload.size()); ChunkIndexData e; @@ -175,8 +143,8 @@ namespace VTX { e.file_offset = static_cast(offset); e.start_frame = f->index; e.end_frame = f->index; - e.chunk_size_bytes = size + static_cast(sizeof(uint32_t)); - e.checksum = XXH3_64bits(f->payload.data(), f->payload.size()); + e.chunk_size_bytes = Framing::ChunkSizeOnDisk(f->payload.size()); + e.checksum = Framing::PayloadChecksum(f->payload); good.push_back(e); last_frame = f->index; @@ -239,14 +207,13 @@ namespace VTX { footer_data.segments = &segments; // Compress exactly as the sink's Close() would have (settings journaled in // the 'S' record), so large recovered footers also match byte-for-byte. - const std::string footer_payload = CompressIfBeneficial(Policy::SerializeFooter(good, footer_data), - journal.use_compression, journal.compression_level); + const std::string footer_payload = Framing::CompressIfBeneficial( + Policy::SerializeFooter(good, footer_data), journal.use_compression, journal.compression_level); bool footer_ok = out.Write(footer_payload.data(), footer_payload.size()); - const uint32_t footer_size = static_cast(footer_payload.size()); - footer_ok = out.Write(&footer_size, sizeof(footer_size)) && footer_ok; - const std::string magic = Policy::GetMagicBytes(); - footer_ok = out.Write(magic.data(), magic.size()) && footer_ok; + const std::string trailer = + Framing::FooterTrailer(static_cast(footer_payload.size()), Policy::GetMagicBytes()); + footer_ok = out.Write(trailer.data(), trailer.size()) && footer_ok; footer_ok = out.Sync() && footer_ok; // Good() also catches any failure in the pending-frame append writes above. const bool all_ok = footer_ok && out.Good(); @@ -296,10 +263,10 @@ namespace VTX { r.error = "cannot open main file: " + path; return r; } - char magic[4] = {0, 0, 0, 0}; - in.read(magic, 4); + char magic[Framing::kMagicSize] = {}; + in.read(magic, Framing::kMagicSize); in.close(); - const std::string m(magic, 4); + const std::string m(magic, Framing::kMagicSize); const int64_t file_size = FileSizeOf(path); diff --git a/sdk/src/vtx_writer/CMakeLists.txt b/sdk/src/vtx_writer/CMakeLists.txt index a994b7c..41b535e 100644 --- a/sdk/src/vtx_writer/CMakeLists.txt +++ b/sdk/src/vtx_writer/CMakeLists.txt @@ -16,12 +16,6 @@ file(GLOB_RECURSE SOURCES CONFIGURE_DEPENDS # 2. Target and deps # ============================================================================== # Library type (STATIC/SHARED) follows BUILD_SHARED_LIBS (VTX_BUILD_SHARED). -# vtx_replay_filter reads the source replay through the reader module, so it is -# only part of the writer when vtx_reader is built (VTX_BUILD_READER). -if(NOT VTX_BUILD_READER) - list(FILTER SOURCES EXCLUDE REGEX "vtx_replay_filter\\.cpp$") -endif() - add_library(vtx_writer ${SOURCES} ${HEADERS}) target_include_directories(vtx_writer @@ -48,10 +42,6 @@ target_link_libraries(vtx_writer # See note in vtx_reader/CMakeLists.txt -- Windows shared DLLs duplicate the # generated protobuf/flatbuffers objects into every module that references them. -if(VTX_BUILD_READER) - target_link_libraries(vtx_writer PRIVATE vtx_reader) # VTX::FilterReplayFile opens the source replay -endif() - if(BUILD_SHARED_LIBS AND WIN32) target_link_libraries(vtx_writer PRIVATE $) endif() diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 302324c..be39fd0 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -49,6 +49,7 @@ include(GoogleTest) add_executable(vtx_tests util/test_main.cpp util/test_fixtures.h + util/replay_snapshot.h common/test_flat_array.cpp common/test_flat_array_edges.cpp @@ -67,6 +68,7 @@ add_executable(vtx_tests common/test_native_loader.cpp common/test_binary_loader.cpp common/test_diagnostics.cpp + common/test_replay_framing.cpp writer/test_writer_basic.cpp writer/test_writer_edges.cpp @@ -76,8 +78,6 @@ add_executable(vtx_tests writer/test_schema_in_memory.cpp writer/test_frame_post_processor.cpp writer/test_frame_finalization.cpp - writer/test_crash_recovery.cpp - writer/test_replay_filter.cpp writer/test_network_sink.cpp writer/test_pipe_source.cpp writer/test_websocket_source.cpp @@ -106,6 +106,18 @@ target_link_libraries(vtx_tests PRIVATE $ ) +# The tools that modify an existing replay (repair / cut / filter) live in +# vtx_transform, which only exists when reader + writer are built; their tests join +# the executable only then. +if(TARGET vtx_transform) + target_sources(vtx_tests PRIVATE + transform/test_crash_recovery.cpp + transform/test_replay_cut.cpp + transform/test_replay_filter.cpp + ) + target_link_libraries(vtx_tests PRIVATE vtx_transform) +endif() + target_include_directories(vtx_tests PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}" "${VTX_COMMON_GENERATED_DIR}" @@ -135,6 +147,7 @@ source_group("common" REGULAR_EXPRESSION "common/.*") source_group("writer" REGULAR_EXPRESSION "writer/.*") source_group("reader" REGULAR_EXPRESSION "reader/.*") source_group("differ" REGULAR_EXPRESSION "differ/.*") +source_group("transform" REGULAR_EXPRESSION "transform/.*") # Register every TEST(suite, name) with ctest gtest_discover_tests(vtx_tests diff --git a/tests/common/test_replay_framing.cpp b/tests/common/test_replay_framing.cpp new file mode 100644 index 0000000..dc32295 --- /dev/null +++ b/tests/common/test_replay_framing.cpp @@ -0,0 +1,265 @@ +// vtx_replay_framing.h is the one definition of the .vtx on-disk framing. These +// tests pin the byte layout of every helper and prove that a file written by the +// real sink is described exactly by ProbeLayout / PayloadChecksum -- which is what +// lets the sinks, repair, cut and filter share it without drifting apart. + +#include + +#include +#include +#include +#include +#include +#include + +#include +#include // the round-trip check decompresses directly + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/writer/core/vtx_writer_facade.h" + +#include "util/replay_snapshot.h" +#include "util/test_fixtures.h" + +using namespace VTX::Framing; + +namespace { + + std::string Repeated(size_t n, char c = 'a') { + return std::string(n, c); + } + + std::string RandomBytes(size_t n, uint32_t seed = 7) { + std::mt19937 rng(seed); + std::string out(n, '\0'); + for (char& c : out) { + c = static_cast(rng() & 0xFF); + } + return out; + } + + // Host-order bytes of a u32, the way the sink has always written them. + std::string HostBytes(uint32_t v) { + std::string s(sizeof(v), '\0'); + std::memcpy(s.data(), &v, sizeof(v)); + return s; + } + +} // namespace + +TEST(ReplayFraming, SizePrefixRoundTripsAndMatchesHostOrder) { + for (uint32_t v : {0u, 1u, 512u, 0xDEADBEEFu, 0xFFFFFFFFu}) { + const SizePrefix p = EncodeSizePrefix(v); + EXPECT_EQ(std::string(p.data(), p.size()), HostBytes(v)) << v; + EXPECT_EQ(DecodeSizePrefix(p.data()), v); + } + EXPECT_EQ(ChunkSizeOnDisk(0), kSizePrefixSize); + EXPECT_EQ(ChunkSizeOnDisk(100), 104u); +} + +TEST(ReplayFraming, BlocksHaveTheDocumentedLayout) { + const std::string payload = "PAYLOAD"; + const std::string magic = "VTXF"; + + const std::string chunk = ChunkBlock(payload); + ASSERT_EQ(chunk.size(), kSizePrefixSize + payload.size()); + EXPECT_EQ(chunk.substr(0, 4), HostBytes(7)); + EXPECT_EQ(chunk.substr(4), payload); + + const std::string header = HeaderBlock(magic, payload); + ASSERT_EQ(header.size(), 4 + 4 + payload.size()); + EXPECT_EQ(header.substr(0, 4), magic); + EXPECT_EQ(header.substr(4, 4), HostBytes(7)); + EXPECT_EQ(header.substr(8), payload); + + const std::string trailer = FooterTrailer(7, magic); + ASSERT_EQ(trailer.size(), kFooterTrailerSize); + EXPECT_EQ(trailer.substr(0, 4), HostBytes(7)); + EXPECT_EQ(trailer.substr(4), magic); + + const std::string footer = FooterBlock(payload, magic); + EXPECT_EQ(footer, payload + trailer); +} + +TEST(ReplayFraming, CompressionRuleMatchesTheWriter) { + EXPECT_FALSE(ShouldCompress(kCompressMinBytes - 1, true)); + EXPECT_TRUE(ShouldCompress(kCompressMinBytes, true)); + EXPECT_FALSE(ShouldCompress(1 << 20, false)); + + // Compressible data shrinks and carries the zstd magic; the sniff agrees. + const std::string big = Repeated(64 * 1024); + const std::string compressed = Compress(big, kDefaultCompressionLevel); + EXPECT_LT(compressed.size(), big.size()); + EXPECT_TRUE(LooksLikeZstd(compressed)); + EXPECT_FALSE(LooksLikeZstd(big)); + + // Incompressible data is returned as-is (never grown). + const std::string noise = RandomBytes(4096); + EXPECT_EQ(Compress(noise, kDefaultCompressionLevel), noise); + + // The combined rule: threshold, enabled flag, then Compress. + EXPECT_EQ(CompressIfBeneficial(Repeated(100)), Repeated(100)); // below threshold + EXPECT_EQ(CompressIfBeneficial(big, /*enabled=*/false), big); + EXPECT_EQ(CompressIfBeneficial(big), compressed); + + // Decompresses back to the input. + std::string round(big.size(), '\0'); + const size_t n = ZSTD_decompress(round.data(), round.size(), compressed.data(), compressed.size()); + ASSERT_FALSE(ZSTD_isError(n)); + EXPECT_EQ(round, big); +} + +TEST(ReplayFraming, ChecksumIsXxHash64OfThePayload) { + const std::string a = RandomBytes(1000, 1); + const std::string b = RandomBytes(1000, 2); + EXPECT_EQ(PayloadChecksum(a), XXH3_64bits(a.data(), a.size())); + EXPECT_EQ(PayloadChecksum(a.data(), a.size()), PayloadChecksum(a)); + EXPECT_NE(PayloadChecksum(a), PayloadChecksum(b)); +} + +TEST(ReplayFraming, ProbeLayoutLocatesHeaderAndFooterOfASyntheticFile) { + const std::string magic = "VTXF"; + const std::string header = Repeated(40, 'h'); + const std::string chunk0 = Repeated(300, 'c'); + const std::string chunk1 = Repeated(20, 'd'); + const std::string footer = Repeated(64, 'f'); + + std::stringstream file; + file << HeaderBlock(magic, header) << ChunkBlock(chunk0) << ChunkBlock(chunk1) << FooterBlock(footer, magic); + + FileLayout layout; + std::string error; + ASSERT_TRUE(ProbeLayout(file, magic, layout, error)) << error; + EXPECT_EQ(layout.file_size, file.str().size()); + EXPECT_EQ(layout.header_end, 4 + 4 + header.size()); + EXPECT_EQ(layout.footer_size, footer.size()); + EXPECT_EQ(layout.footer_offset, layout.file_size - kFooterTrailerSize - footer.size()); + EXPECT_FALSE(layout.footer_compressed); + // The chunk region is exactly what sits between header and footer. + EXPECT_EQ(layout.footer_offset - layout.header_end, ChunkBlock(chunk0).size() + ChunkBlock(chunk1).size()); + EXPECT_TRUE(IsZstdAt(file, layout.header_end + kSizePrefixSize, chunk0.size()) == false); + + // A compressed footer is detected through the sniff. + const std::string zfooter = Compress(Repeated(4096, 'f'), kDefaultCompressionLevel); + std::stringstream zfile; + zfile << HeaderBlock(magic, header) << ChunkBlock(chunk0) << FooterBlock(zfooter, magic); + ASSERT_TRUE(ProbeLayout(zfile, magic, layout, error)) << error; + EXPECT_TRUE(layout.footer_compressed); + EXPECT_EQ(layout.footer_size, zfooter.size()); +} + +TEST(ReplayFraming, ProbeRejectsBrokenLayouts) { + const std::string magic = "VTXF"; + const std::string header = Repeated(40, 'h'); + FileLayout layout; + std::string error; + + // Wrong magic. + std::stringstream wrong; + wrong << HeaderBlock("VTXP", header) << FooterBlock("f", "VTXP"); + EXPECT_FALSE(ProbeLayout(wrong, magic, layout, error)); + EXPECT_NE(error.find("magic"), std::string::npos) << error; + + // Footerless (crash-truncated): the header probes fine, the trailer does not. + std::stringstream truncated; + truncated << HeaderBlock(magic, header) << ChunkBlock(Repeated(300)); + uint64_t header_end = 0; + ASSERT_TRUE(ProbeHeader(truncated, magic, StreamSize(truncated), header_end, error)) << error; + EXPECT_EQ(header_end, 4 + 4 + header.size()); + EXPECT_FALSE(ProbeLayout(truncated, magic, layout, error)); + EXPECT_NE(error.find("no footer"), std::string::npos) << error; + + // A footer size that reaches back into the header. + std::stringstream overlap; + overlap << HeaderBlock(magic, header) << Repeated(10, 'x') << FooterTrailer(1000, magic); + EXPECT_FALSE(ProbeLayout(overlap, magic, layout, error)); + EXPECT_FALSE(error.empty()); + + // Empty / too small. + std::stringstream empty; + EXPECT_FALSE(ProbeLayout(empty, magic, layout, error)); +} + +TEST(ReplayFraming, CopyRangeStreamsExactBytes) { + const std::string data = RandomBytes(10 * 1024 + 17); + std::stringstream in(data); + std::stringstream out; + std::string error; + ASSERT_TRUE(CopyRange(in, out, 100, 5000, error)) << error; + EXPECT_EQ(out.str(), data.substr(100, 5000)); + + std::stringstream none; + ASSERT_TRUE(CopyRange(in, none, 0, 0, error)) << error; + EXPECT_TRUE(none.str().empty()); + + std::stringstream past; + EXPECT_FALSE(CopyRange(in, past, data.size() - 10, 100, error)); + EXPECT_NE(error.find("Read failed"), std::string::npos) << error; +} + +// --------------------------------------------------------------------------- +// The real writer and the framing helpers must agree byte for byte: probe a +// freshly recorded file and re-derive every seek-table field from the bytes. +// --------------------------------------------------------------------------- +class ReplayFramingOnDisk : public ::testing::TestWithParam {}; + +TEST_P(ReplayFramingOnDisk, SinkOutputIsDescribedByTheSharedFraming) { + const bool flat = GetParam() == VTX::VtxFormat::FlatBuffers; + const std::string magic = flat ? "VTXF" : "VTXP"; + + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = VtxTest::OutputPath(std::string("framing_") + (flat ? "fb" : "pb") + ".vtx"); + cfg.schema_json_path = VtxTest::FixturePath("test_schema.json"); + cfg.replay_name = "FramingTest"; + cfg.replay_uuid = "uuid-framing"; + cfg.chunk_max_frames = 25; + cfg.use_compression = true; + auto writer = flat ? VTX::CreateFlatBuffersWriterFacade(cfg) : VTX::CreateProtobufWriterFacade(cfg); + ASSERT_TRUE(writer); + for (int i = 0; i < 60; ++i) { + VTX::Frame frame; + auto& bucket = frame.CreateBucket("entity"); + VTX::PropertyContainer pc; + pc.entity_type_id = 0; + pc.string_properties = {"p", "n"}; + pc.int32_properties = {1, i, 0}; + pc.float_properties = {float(i), 1.0f}; + bucket.unique_ids.push_back("p"); + bucket.entities.push_back(std::move(pc)); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / 60.0f; + writer->RecordFrame(frame, t); + } + writer->Stop(); + + const VtxTest::ReplaySnapshot s = VtxTest::ReadReplay(cfg.output_filepath); + ASSERT_TRUE(s.ok); + + std::ifstream in(cfg.output_filepath, std::ios::binary); + ASSERT_TRUE(in); + FileLayout layout; + std::string error; + ASSERT_TRUE(ProbeLayout(in, magic, layout, error)) << error; + + // Chunks begin right after the header block and end right before the footer. + ASSERT_FALSE(s.footer.chunk_index.empty()); + EXPECT_EQ(s.footer.chunk_index.front().file_offset, layout.header_end); + const auto& last = s.footer.chunk_index.back(); + EXPECT_EQ(last.file_offset + last.chunk_size_bytes, layout.footer_offset); + + // Every entry's size prefix and checksum re-derive from the bytes on disk. + VtxTest::ExpectSeekTableMatchesFile(cfg.output_filepath, s.footer); + + // Each chunk is either a zstd frame or raw, exactly as the sniff says. + for (const auto& e : s.footer.chunk_index) { + const bool z = IsZstdAt(in, e.file_offset + kSizePrefixSize, e.chunk_size_bytes - kSizePrefixSize); + EXPECT_TRUE(z || e.chunk_size_bytes - kSizePrefixSize < kCompressMinBytes) << "chunk " << e.chunk_index; + } +} + +INSTANTIATE_TEST_SUITE_P(Formats, ReplayFramingOnDisk, + ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), + [](const ::testing::TestParamInfo& info) { + return info.param == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; + }); diff --git a/tests/writer/test_crash_recovery.cpp b/tests/transform/test_crash_recovery.cpp similarity index 99% rename from tests/writer/test_crash_recovery.cpp rename to tests/transform/test_crash_recovery.cpp index 1b57934..af873e5 100644 --- a/tests/writer/test_crash_recovery.cpp +++ b/tests/transform/test_crash_recovery.cpp @@ -39,8 +39,8 @@ #include "vtx/common/vtx_types.h" #include "vtx/reader/core/vtx_reader_facade.h" #include "vtx/reader/core/vtx_replay_validation.h" +#include "vtx/transform/vtx_replay_recovery.h" #include "vtx/writer/core/vtx_frame_post_processor.h" -#include "vtx/writer/core/vtx_replay_recovery.h" #include "vtx/writer/core/vtx_writer_facade.h" #include "vtx/writer/core/writer.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" diff --git a/tests/transform/test_replay_cut.cpp b/tests/transform/test_replay_cut.cpp new file mode 100644 index 0000000..fb5399e --- /dev/null +++ b/tests/transform/test_replay_cut.cpp @@ -0,0 +1,335 @@ +// Replay cut tests: CutReplayFile() writes a copy of a replay holding a sub-range +// of its frames. Whole chunks are copied verbatim (same checksum), edge chunks are +// re-serialized with only the kept frames, frames are renumbered from 0, the time +// table is sliced with absolute tick values, and the seek table describes the bytes +// on disk. Runs as a TEST_P over both backends. + +#include + +#include +#include +#include +#include +#include + +#include "vtx/common/vtx_types.h" +#include "vtx/transform/vtx_replay_cut.h" +#include "vtx/writer/core/vtx_writer_facade.h" + +#include "util/replay_snapshot.h" +#include "util/test_fixtures.h" + +namespace { + + constexpr int32_t kFrames = 100; // 3 chunks: [0,39] [40,79] [80,99] + constexpr int32_t kChunkFrames = 40; + constexpr float kFps = 60.0f; + constexpr int64_t kBaseUtc = 17'000'000'000'000'000LL; // unix-relative 100 ns ticks + constexpr int64_t kFrameTicks = 166'666; + + const char* FormatName(VTX::VtxFormat f) { + return f == VTX::VtxFormat::FlatBuffers ? "FlatBuffers" : "Protobuf"; + } + + // Fixture schema (tests/fixtures/test_schema.json), Player (type 0): strings + // [UniqueID, Name], int32 [Team, Score, Deaths], float [Health, Armor], ... + VTX::Frame BuildFrame(int32_t frame) { + VTX::Frame f; + auto& bucket = f.CreateBucket("entity"); + for (int32_t p = 0; p < 2; ++p) { + VTX::PropertyContainer pc; + pc.entity_type_id = 0; + const std::string id = "player_" + std::to_string(p); + pc.string_properties = {id, p == 0 ? "Alpha" : "Bravo"}; + pc.int32_properties = {p + 1, frame, frame / 10}; // Score carries the frame index + pc.float_properties = {100.0f - float(frame % 50), 25.0f + float(p)}; + pc.vector_properties = {VTX::Vector {double(frame), double(p), 0.0}, VTX::Vector {1.0, 0.0, 0.0}}; + pc.quat_properties = {VTX::Quat {0.0f, 0.0f, 0.0f, 1.0f}}; + pc.bool_properties = {frame % 7 != 0}; + bucket.unique_ids.push_back(id); + bucket.entities.push_back(std::move(pc)); + } + return f; + } + + // Output frame k must equal source frame (first + k), entity by entity. + void ExpectFramesAreSourceRange(const VtxTest::ReplaySnapshot& source, const VtxTest::ReplaySnapshot& output, + int32_t first, int32_t last) { + ASSERT_EQ(output.frames.size(), static_cast(last - first + 1)); + for (size_t k = 0; k < output.frames.size(); ++k) { + const VTX::Frame& s = source.frames[static_cast(first) + k]; + const VTX::Frame& o = output.frames[k]; + ASSERT_EQ(o.GetBuckets().size(), s.GetBuckets().size()) << "frame " << k; + for (size_t b = 0; b < s.GetBuckets().size(); ++b) { + const VTX::Bucket& sb = s.GetBuckets()[b]; + const VTX::Bucket& ob = o.GetBuckets()[b]; + ASSERT_EQ(ob.unique_ids, sb.unique_ids) << "frame " << k; + ASSERT_EQ(ob.entities.size(), sb.entities.size()) << "frame " << k; + for (size_t i = 0; i < sb.entities.size(); ++i) { + VtxTest::ExpectEntityEqual(sb.entities[i], ob.entities[i], + "frame " + std::to_string(k) + " entity " + sb.unique_ids[i]); + } + } + } + } + + // The footer's time table is the source table sliced to [first, last], values untouched. + void ExpectTimeTableSliced(const VtxTest::ReplaySnapshot& source, const VtxTest::ReplaySnapshot& output, + int32_t first, int32_t last) { + const size_t n = static_cast(last - first + 1); + ASSERT_EQ(output.footer.times.created_utc.size(), n); + ASSERT_EQ(output.footer.times.game_time.size(), n); + for (size_t k = 0; k < n; ++k) { + EXPECT_EQ(output.footer.times.created_utc[k], + source.footer.times.created_utc[static_cast(first) + k]) + << k; + EXPECT_EQ(output.footer.times.game_time[k], source.footer.times.game_time[static_cast(first) + k]) + << k; + } + const double expected_duration = double(source.footer.times.created_utc[static_cast(last)] - + source.footer.times.created_utc[static_cast(first)]) / + double(VTX::GameTime::TICKS_PER_SECOND); + EXPECT_NEAR(output.footer.duration_seconds, expected_duration, 1e-3); + EXPECT_EQ(output.footer.total_frames, last - first + 1); + } + +} // namespace + +class ReplayCutTest : public ::testing::TestWithParam { +protected: + std::string Path(const std::string& suffix) const { + return VtxTest::OutputPath(std::string("replay_cut_") + FormatName(GetParam()) + "_" + suffix + ".vtx"); + } + + std::string WriteSource(const std::string& suffix) const { + VTX::WriterFacadeConfig cfg; + cfg.output_filepath = Path(suffix + "_src"); + cfg.schema_json_path = VtxTest::FixturePath("test_schema.json"); + cfg.replay_name = "ReplayCutTest"; + cfg.replay_uuid = "uuid-cut-" + suffix; + cfg.default_fps = kFps; + cfg.chunk_max_frames = kChunkFrames; + cfg.use_compression = true; + + auto writer = GetParam() == VTX::VtxFormat::FlatBuffers ? VTX::CreateFlatBuffersWriterFacade(cfg) + : VTX::CreateProtobufWriterFacade(cfg); + EXPECT_TRUE(writer); + if (!writer) { + return {}; + } + for (int32_t i = 0; i < kFrames; ++i) { + auto frame = BuildFrame(i); + VTX::GameTime::GameTimeRegister t; + t.game_time = float(i) / kFps; + t.created_utc_time = kBaseUtc + int64_t(i) * kFrameTicks; + writer->RecordFrame(frame, t); + } + writer->Stop(); + return cfg.output_filepath; + } +}; + +// --------------------------------------------------------------------------- +// A whole-chunk cut copies the chunk verbatim: same bytes (same checksum), frames +// renumbered from 0, time table sliced, header identical. +// --------------------------------------------------------------------------- +TEST_P(ReplayCutTest, WholeChunkCutCopiesVerbatim) { + const std::string src = WriteSource("wholechunk"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + ASSERT_EQ(source.footer.chunk_index.size(), 3u); + + const VTX::ReplayCutPlan plan = VTX::PlanCutChunks(source.footer, 1, 1); + ASSERT_TRUE(plan.valid) << plan.error; + EXPECT_EQ(plan.first_frame, 40); + EXPECT_EQ(plan.last_frame, 79); + EXPECT_FALSE(plan.trims_head); + EXPECT_FALSE(plan.trims_tail); + EXPECT_EQ(plan.FrameCount(), 40); + EXPECT_EQ(plan.chunk_bytes, source.footer.chunk_index[1].chunk_size_bytes); + + const std::string dst = Path("wholechunk_out"); + const VTX::ReplayCutResult r = VTX::CutReplayFile(src, source.footer, plan, dst); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, 40); + EXPECT_EQ(r.chunks_written, 1); + EXPECT_EQ(r.chunks_rewritten, 0); + EXPECT_GT(r.output_bytes, 0u); + + const VtxTest::ReplaySnapshot output = VtxTest::ReadReplay(dst); + ASSERT_TRUE(output.ok); + EXPECT_EQ(output.format, source.format); + EXPECT_EQ(output.header.replay_uuid, source.header.replay_uuid); + EXPECT_EQ(output.header.recorded_utc_timestamp, source.header.recorded_utc_timestamp); + EXPECT_EQ(output.schema.property_mapping, source.schema.property_mapping); + + ASSERT_EQ(output.footer.chunk_index.size(), 1u); + EXPECT_EQ(output.footer.chunk_index[0].chunk_index, 0); + EXPECT_EQ(output.footer.chunk_index[0].start_frame, 0); + EXPECT_EQ(output.footer.chunk_index[0].end_frame, 39); + EXPECT_EQ(output.footer.chunk_index[0].checksum, source.footer.chunk_index[1].checksum); // verbatim copy + EXPECT_EQ(output.footer.chunk_index[0].chunk_size_bytes, source.footer.chunk_index[1].chunk_size_bytes); + + ExpectFramesAreSourceRange(source, output, 40, 79); + ExpectTimeTableSliced(source, output, 40, 79); + VtxTest::ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// An exact-frame cut rewrites both edge chunks and keeps the middle one verbatim. +// --------------------------------------------------------------------------- +TEST_P(ReplayCutTest, ExactFrameCutRewritesEdges) { + const std::string src = WriteSource("exact"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + + const VTX::ReplayCutPlan plan = VTX::PlanCutFrames(source.footer, 30, 85); + ASSERT_TRUE(plan.valid) << plan.error; + EXPECT_EQ(plan.first_chunk, 0); + EXPECT_EQ(plan.last_chunk, 2); + EXPECT_EQ(plan.first_frame, 30); + EXPECT_EQ(plan.last_frame, 85); + EXPECT_TRUE(plan.trims_head); + EXPECT_TRUE(plan.trims_tail); + EXPECT_EQ(plan.FrameCount(), 56); + + const std::string dst = Path("exact_out"); + const VTX::ReplayCutResult r = VTX::CutReplayFile(src, source.footer, plan, dst); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, 56); + EXPECT_EQ(r.chunks_written, 3); + EXPECT_EQ(r.chunks_rewritten, 2); + + const VtxTest::ReplaySnapshot output = VtxTest::ReadReplay(dst); + ASSERT_TRUE(output.ok); + ASSERT_EQ(output.footer.chunk_index.size(), 3u); + EXPECT_EQ(output.footer.chunk_index[0].start_frame, 0); + EXPECT_EQ(output.footer.chunk_index[0].end_frame, 9); // source 30..39 + EXPECT_EQ(output.footer.chunk_index[1].start_frame, 10); + EXPECT_EQ(output.footer.chunk_index[1].end_frame, 49); // source 40..79, verbatim + EXPECT_EQ(output.footer.chunk_index[1].checksum, source.footer.chunk_index[1].checksum); + EXPECT_EQ(output.footer.chunk_index[2].start_frame, 50); + EXPECT_EQ(output.footer.chunk_index[2].end_frame, 55); // source 80..85 + EXPECT_NE(output.footer.chunk_index[0].checksum, source.footer.chunk_index[0].checksum); // rewritten + EXPECT_NE(output.footer.chunk_index[2].checksum, source.footer.chunk_index[2].checksum); + + ExpectFramesAreSourceRange(source, output, 30, 85); + ExpectTimeTableSliced(source, output, 30, 85); + VtxTest::ExpectSeekTableMatchesFile(dst, output.footer); +} + +// Bounds are clamped to the replay and a head cut alone rewrites one chunk. +TEST_P(ReplayCutTest, ClampsBoundsAndTrimsOnlyOneEdge) { + const std::string src = WriteSource("clamp"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + + const VTX::ReplayCutPlan plan = VTX::PlanCutFrames(source.footer, 95, 100'000); + ASSERT_TRUE(plan.valid) << plan.error; + EXPECT_EQ(plan.first_frame, 95); + EXPECT_EQ(plan.last_frame, kFrames - 1); + EXPECT_TRUE(plan.trims_head); + EXPECT_FALSE(plan.trims_tail); + + const std::string dst = Path("clamp_out"); + const VTX::ReplayCutResult r = VTX::CutReplayFile(src, source.footer, plan, dst); + ASSERT_TRUE(r.ok()) << r.error; + EXPECT_EQ(r.total_frames, 5); + EXPECT_EQ(r.chunks_written, 1); + EXPECT_EQ(r.chunks_rewritten, 1); + + const VtxTest::ReplaySnapshot output = VtxTest::ReadReplay(dst); + ASSERT_TRUE(output.ok); + ExpectFramesAreSourceRange(source, output, 95, kFrames - 1); + VtxTest::ExpectSeekTableMatchesFile(dst, output.footer); +} + +// --------------------------------------------------------------------------- +// Failure paths report an error and never leave a destination behind. +// --------------------------------------------------------------------------- +TEST_P(ReplayCutTest, ErrorsAreReported) { + namespace fs = std::filesystem; + const std::string src = WriteSource("errors"); + ASSERT_FALSE(src.empty()); + const VtxTest::ReplaySnapshot source = VtxTest::ReadReplay(src); + ASSERT_TRUE(source.ok); + const std::string dst = Path("errors_out"); + + { + VTX::ReplayCutPlan invalid; // valid == false + const auto r = VTX::CutReplayFile(src, source.footer, invalid, dst); + EXPECT_FALSE(r.ok()); + EXPECT_FALSE(fs::exists(dst)); + } + { + const VTX::ReplayCutPlan plan = VTX::PlanCutChunks(source.footer, 0, 0); + ASSERT_TRUE(plan.valid); + const auto r = VTX::CutReplayFile(src, source.footer, plan, src); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("different file"), std::string::npos) << r.error; + EXPECT_TRUE(fs::exists(src)); + } + { + const VTX::ReplayCutPlan plan = VTX::PlanCutChunks(source.footer, 0, 0); + const auto r = VTX::CutReplayFile(Path("does_not_exist"), source.footer, plan, dst); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("does not exist"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } + { + // A plan built against a different footer must not index past this one. + VTX::ReplayCutPlan stale = VTX::PlanCutChunks(source.footer, 0, 0); + stale.last_chunk = 42; + const auto r = VTX::CutReplayFile(src, source.footer, stale, dst); + EXPECT_FALSE(r.ok()); + EXPECT_NE(r.error.find("chunk index"), std::string::npos) << r.error; + EXPECT_FALSE(fs::exists(dst)); + } +} + +INSTANTIATE_TEST_SUITE_P(Formats, ReplayCutTest, + ::testing::Values(VTX::VtxFormat::FlatBuffers, VTX::VtxFormat::Protobuf), + [](const ::testing::TestParamInfo& info) { return FormatName(info.param); }); + +// =========================================================================== +// Plan resolution on a synthetic footer, no file involved. +// =========================================================================== +TEST(ReplayCutPlan, ResolvesRangesAgainstTheChunkIndex) { + VTX::FileFooter footer; + footer.total_frames = 100; + for (int32_t i = 0; i < 3; ++i) { + VTX::ChunkIndexEntry e; + e.chunk_index = i; + e.start_frame = i * 40; + e.end_frame = std::min(i * 40 + 39, 99); + e.chunk_size_bytes = 1000 + static_cast(i); + footer.chunk_index.push_back(e); + } + + const auto whole = VTX::PlanCutChunks(footer, 0, 2); + ASSERT_TRUE(whole.valid); + EXPECT_EQ(whole.first_frame, 0); + EXPECT_EQ(whole.last_frame, 99); + EXPECT_FALSE(whole.trims_head || whole.trims_tail); + EXPECT_EQ(whole.chunk_bytes, 3003u); + + const auto exact = VTX::PlanCutFrames(footer, 45, 45); + ASSERT_TRUE(exact.valid); + EXPECT_EQ(exact.first_chunk, 1); + EXPECT_EQ(exact.last_chunk, 1); + EXPECT_TRUE(exact.trims_head); + EXPECT_TRUE(exact.trims_tail); + EXPECT_EQ(exact.FrameCount(), 1); + + const auto boundary = VTX::PlanCutFrames(footer, 40, 79); + ASSERT_TRUE(boundary.valid); + EXPECT_FALSE(boundary.trims_head || boundary.trims_tail); // exactly chunk 1 + + EXPECT_FALSE(VTX::PlanCutFrames(footer, 50, 10).valid); + EXPECT_FALSE(VTX::PlanCutChunks(footer, 2, 1).valid); + EXPECT_FALSE(VTX::PlanCutFrames(VTX::FileFooter {}, 0, 0).valid); + EXPECT_EQ(VTX::ReplayCutPlan {}.FrameCount(), 0); +} diff --git a/tests/writer/test_replay_filter.cpp b/tests/transform/test_replay_filter.cpp similarity index 90% rename from tests/writer/test_replay_filter.cpp rename to tests/transform/test_replay_filter.cpp index 46683bb..97e1a50 100644 --- a/tests/writer/test_replay_filter.cpp +++ b/tests/transform/test_replay_filter.cpp @@ -17,23 +17,28 @@ #include #include -#include - #include "vtx_schema_generated.h" // complete fbsvtx/cppvtx types before the policy headers #include "vtx_schema.pb.h" #include "vtx/common/readers/schema_reader/schema_registry.h" #include "vtx/common/vtx_types.h" #include "vtx/reader/core/vtx_reader_facade.h" -#include "vtx/writer/core/vtx_replay_filter.h" +#include "vtx/transform/vtx_replay_filter.h" #include "vtx/writer/core/vtx_writer_facade.h" #include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" #include "vtx/writer/policies/formatters/protobuff_vtx_policy.h" +#include "util/replay_snapshot.h" #include "util/test_fixtures.h" namespace { + // Read-back helpers shared with the cut / framing tests. + using Snapshot = VtxTest::ReplaySnapshot; + using VtxTest::ExpectEntityEqual; + using VtxTest::ExpectSeekTableMatchesFile; + using VtxTest::ReadReplay; + constexpr int kFrames = 100; // 3 chunks: [0,39] [40,79] [80,99] constexpr int kChunkFrames = 40; constexpr float kFps = 60.0f; @@ -137,68 +142,6 @@ namespace { return f; } - struct Snapshot { - bool ok = false; - VTX::VtxFormat format = VTX::VtxFormat::Unknown; - VTX::FileHeader header; - VTX::ContextualSchema schema; - VTX::FileFooter footer; - std::vector frames; - }; - - Snapshot ReadReplay(const std::string& path) { - Snapshot s; - VTX::ReaderContext ctx = VTX::OpenReplayFile(path); - if (!ctx.Loaded()) { - ADD_FAILURE() << "open failed for " << path << ": " << ctx.GetError().message; - return s; - } - if (!ctx.WaitUntilReady()) { - ADD_FAILURE() << "load failed for " << path << ": " << ctx.GetReadyError().message; - return s; - } - s.format = ctx.format; - s.header = ctx->GetHeader(); - s.schema = ctx->GetContextualSchema(); - s.footer = ctx->GetFooter(); - const int32_t total = ctx->GetTotalFrames(); - s.frames.reserve(static_cast(total)); - for (int32_t i = 0; i < total; ++i) { - const VTX::Frame* frame = ctx->GetFrameSync(i); - if (!frame) { - ADD_FAILURE() << "frame " << i << " unreadable in " << path; - return s; - } - s.frames.push_back(*frame); - } - s.ok = true; - return s; - } - - void ExpectEntityEqual(const VTX::PropertyContainer& a, const VTX::PropertyContainer& b, const std::string& what) { - EXPECT_EQ(a.entity_type_id, b.entity_type_id) << what; - EXPECT_EQ(a.content_hash, b.content_hash) << what; - EXPECT_EQ(a.bool_properties, b.bool_properties) << what; - EXPECT_EQ(a.int32_properties, b.int32_properties) << what; - EXPECT_EQ(a.int64_properties, b.int64_properties) << what; - EXPECT_EQ(a.float_properties, b.float_properties) << what; - EXPECT_EQ(a.double_properties, b.double_properties) << what; - EXPECT_EQ(a.string_properties, b.string_properties) << what; - EXPECT_EQ(a.transform_properties, b.transform_properties) << what; - EXPECT_EQ(a.vector_properties, b.vector_properties) << what; - EXPECT_EQ(a.quat_properties, b.quat_properties) << what; - EXPECT_EQ(a.range_properties, b.range_properties) << what; - EXPECT_EQ(a.int32_arrays.data, b.int32_arrays.data) << what; - EXPECT_EQ(a.int32_arrays.offsets, b.int32_arrays.offsets) << what; - EXPECT_EQ(a.float_arrays.data, b.float_arrays.data) << what; - EXPECT_EQ(a.float_arrays.offsets, b.float_arrays.offsets) << what; - EXPECT_EQ(a.string_arrays.data, b.string_arrays.data) << what; - EXPECT_EQ(a.string_arrays.offsets, b.string_arrays.offsets) << what; - EXPECT_EQ(a.vector_arrays.data, b.vector_arrays.data) << what; - EXPECT_EQ(a.any_struct_properties.size(), b.any_struct_properties.size()) << what; - EXPECT_EQ(a.map_properties.size(), b.map_properties.size()) << what; - } - // The output bucket must be exactly the source bucket minus the entities `dropped` // rejects, in the original order, with every survivor unchanged. template @@ -279,31 +222,6 @@ namespace { } } - // Every seek-table entry must describe the bytes actually on disk: the length - // prefix equals chunk_size_bytes - 4 and the xxHash64 of the payload matches. - void ExpectSeekTableMatchesFile(const std::string& path, const VTX::FileFooter& footer) { - std::ifstream in(path, std::ios::binary); - ASSERT_TRUE(in) << path; - in.seekg(0, std::ios::end); - const uint64_t file_size = static_cast(in.tellg()); - ASSERT_FALSE(footer.chunk_index.empty()); - for (const VTX::ChunkIndexEntry& entry : footer.chunk_index) { - ASSERT_LE(entry.file_offset + entry.chunk_size_bytes, file_size) << "chunk " << entry.chunk_index; - in.seekg(static_cast(entry.file_offset)); - uint32_t payload_size = 0; - ASSERT_TRUE(in.read(reinterpret_cast(&payload_size), sizeof(payload_size))); - ASSERT_EQ(payload_size + sizeof(uint32_t), entry.chunk_size_bytes) << "chunk " << entry.chunk_index; - std::string payload(payload_size, '\0'); - ASSERT_TRUE(in.read(payload.data(), payload_size)); - EXPECT_EQ(XXH3_64bits(payload.data(), payload.size()), entry.checksum) << "chunk " << entry.chunk_index; - } - // Chunks are contiguous from the first chunk to the footer block. - for (size_t i = 1; i < footer.chunk_index.size(); ++i) { - EXPECT_EQ(footer.chunk_index[i].file_offset, - footer.chunk_index[i - 1].file_offset + footer.chunk_index[i - 1].chunk_size_bytes); - } - } - VTX::TimelineEvent MakeEvent(float game_time, const std::string& type, const std::string& label, const std::string& entity_id) { VTX::TimelineEvent ev; diff --git a/tests/util/replay_snapshot.h b/tests/util/replay_snapshot.h new file mode 100644 index 0000000..293fddd --- /dev/null +++ b/tests/util/replay_snapshot.h @@ -0,0 +1,109 @@ +#pragma once +// Read-back helpers shared by the replay rewrite tests (framing, cut, filter): +// open a .vtx, pull every frame, and check a footer's seek table against the +// bytes actually on disk through the shared framing definitions. + +#include + +#include +#include +#include +#include + +#include "vtx/common/vtx_replay_framing.h" +#include "vtx/common/vtx_types.h" +#include "vtx/reader/core/vtx_reader_facade.h" + +namespace VtxTest { + + struct ReplaySnapshot { + bool ok = false; + VTX::VtxFormat format = VTX::VtxFormat::Unknown; + VTX::FileHeader header; + VTX::ContextualSchema schema; + VTX::FileFooter footer; + std::vector frames; + }; + + /// Opens @p path and copies out header, schema, footer and every frame. + inline ReplaySnapshot ReadReplay(const std::string& path) { + ReplaySnapshot s; + VTX::ReaderContext ctx = VTX::OpenReplayFile(path); + if (!ctx.Loaded()) { + ADD_FAILURE() << "open failed for " << path << ": " << ctx.GetError().message; + return s; + } + if (!ctx.WaitUntilReady()) { + ADD_FAILURE() << "load failed for " << path << ": " << ctx.GetReadyError().message; + return s; + } + s.format = ctx.format; + s.header = ctx->GetHeader(); + s.schema = ctx->GetContextualSchema(); + s.footer = ctx->GetFooter(); + const int32_t total = ctx->GetTotalFrames(); + s.frames.reserve(static_cast(total)); + for (int32_t i = 0; i < total; ++i) { + const VTX::Frame* frame = ctx->GetFrameSync(i); + if (!frame) { + ADD_FAILURE() << "frame " << i << " unreadable in " << path; + return s; + } + s.frames.push_back(*frame); + } + s.ok = true; + return s; + } + + /// Field-by-field equality of two entities (scalars, math types, the common arrays, container counts). + inline void ExpectEntityEqual(const VTX::PropertyContainer& a, const VTX::PropertyContainer& b, + const std::string& what) { + EXPECT_EQ(a.entity_type_id, b.entity_type_id) << what; + EXPECT_EQ(a.content_hash, b.content_hash) << what; + EXPECT_EQ(a.bool_properties, b.bool_properties) << what; + EXPECT_EQ(a.int32_properties, b.int32_properties) << what; + EXPECT_EQ(a.int64_properties, b.int64_properties) << what; + EXPECT_EQ(a.float_properties, b.float_properties) << what; + EXPECT_EQ(a.double_properties, b.double_properties) << what; + EXPECT_EQ(a.string_properties, b.string_properties) << what; + EXPECT_EQ(a.transform_properties, b.transform_properties) << what; + EXPECT_EQ(a.vector_properties, b.vector_properties) << what; + EXPECT_EQ(a.quat_properties, b.quat_properties) << what; + EXPECT_EQ(a.range_properties, b.range_properties) << what; + EXPECT_EQ(a.int32_arrays.data, b.int32_arrays.data) << what; + EXPECT_EQ(a.int32_arrays.offsets, b.int32_arrays.offsets) << what; + EXPECT_EQ(a.float_arrays.data, b.float_arrays.data) << what; + EXPECT_EQ(a.float_arrays.offsets, b.float_arrays.offsets) << what; + EXPECT_EQ(a.string_arrays.data, b.string_arrays.data) << what; + EXPECT_EQ(a.string_arrays.offsets, b.string_arrays.offsets) << what; + EXPECT_EQ(a.vector_arrays.data, b.vector_arrays.data) << what; + EXPECT_EQ(a.any_struct_properties.size(), b.any_struct_properties.size()) << what; + EXPECT_EQ(a.map_properties.size(), b.map_properties.size()) << what; + } + + /// Every seek-table entry must describe the bytes on disk: the size prefix equals + /// chunk_size_bytes - prefix, the xxHash64 of the payload matches, and chunks are + /// contiguous from the first one onwards. + inline void ExpectSeekTableMatchesFile(const std::string& path, const VTX::FileFooter& footer) { + std::ifstream in(path, std::ios::binary); + ASSERT_TRUE(in) << path; + const uint64_t file_size = VTX::Framing::StreamSize(in); + ASSERT_FALSE(footer.chunk_index.empty()); + for (const VTX::ChunkIndexEntry& entry : footer.chunk_index) { + ASSERT_LE(entry.file_offset + entry.chunk_size_bytes, file_size) << "chunk " << entry.chunk_index; + char prefix[VTX::Framing::kSizePrefixSize]; + ASSERT_TRUE(VTX::Framing::ReadAt(in, entry.file_offset, prefix, sizeof(prefix))); + const uint32_t payload_size = VTX::Framing::DecodeSizePrefix(prefix); + ASSERT_EQ(VTX::Framing::ChunkSizeOnDisk(payload_size), entry.chunk_size_bytes) + << "chunk " << entry.chunk_index; + std::string payload(payload_size, '\0'); + ASSERT_TRUE(in.read(payload.data(), payload_size)); + EXPECT_EQ(VTX::Framing::PayloadChecksum(payload), entry.checksum) << "chunk " << entry.chunk_index; + } + for (size_t i = 1; i < footer.chunk_index.size(); ++i) { + EXPECT_EQ(footer.chunk_index[i].file_offset, + footer.chunk_index[i - 1].file_offset + footer.chunk_index[i - 1].chunk_size_bytes); + } + } + +} // namespace VtxTest diff --git a/tools/inspector/CMakeLists.txt b/tools/inspector/CMakeLists.txt index 6409e1c..a8223c5 100644 --- a/tools/inspector/CMakeLists.txt +++ b/tools/inspector/CMakeLists.txt @@ -29,10 +29,19 @@ target_include_directories(vtx_inspector PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include ) +# File > Repair Replay / Cut Replay / Filter Entities run through vtx_transform, which +# is only built when reader + writer are; fail at configure time rather than at link time. +if(NOT TARGET vtx_transform) + message(FATAL_ERROR + "vtx_inspector requires the vtx_transform module (File > Repair Replay / Cut Replay / Filter Entities). " + "Enable VTX_BUILD_TRANSFORM together with VTX_BUILD_READER and VTX_BUILD_WRITER, " + "or configure with -DBUILD_VTX_INSPECTOR=OFF.") +endif() + target_link_libraries(vtx_inspector PRIVATE vtx_tool_shared vtx_reader - vtx_writer # VTX::RepairReplayFile for File > Repair Replay + vtx_transform # VTX::RepairReplayFile / VTX::CutReplayFile / VTX::FilterReplayFile ) set_target_properties(vtx_inspector PROPERTIES diff --git a/tools/inspector/include/services/replay_cut_service.h b/tools/inspector/include/services/replay_cut_service.h deleted file mode 100644 index 15e3780..0000000 --- a/tools/inspector/include/services/replay_cut_service.h +++ /dev/null @@ -1,47 +0,0 @@ -#pragma once - -#include -#include - -#include "vtx/common/vtx_types.h" - -namespace VtxServices { - - // Resolved cut request. Interior chunks are always copied verbatim; when the - // requested range starts or ends inside a chunk (trims_head/trims_tail) that - // edge chunk is re-serialized with only the kept frames, and its seek-table - // entry (size, checksum, frame range) is regenerated. - struct ReplayCutPlan { - bool valid = false; - std::string error; // set when !valid - - int first_chunk = 0; // index into footer.chunk_index - int last_chunk = 0; - int first_frame = 0; // exact kept range, original frame numbering - int last_frame = 0; - bool trims_head = false; // first chunk starts before first_frame - bool trims_tail = false; // last chunk ends after last_frame - uint64_t chunk_bytes = 0; // on-disk bytes of the involved source chunks - }; - - // Cuts a new .vtx out of an existing flatbuffers replay: header block copied - // verbatim, whole chunks copied verbatim, partial edge chunks rewritten, and - // the footer rebuilt (frame numbering rebased to 0, time table sliced, - // duration recomputed). Timeline events are not carried over. - class ReplayCutService { - public: - // Exact cut: keeps [start_frame, end_frame] inclusive, trimming inside - // the edge chunks when the bounds do not fall on chunk boundaries. - static ReplayCutPlan PlanCutFrames(const VTX::FileFooter& footer, int start_frame, int end_frame); - - // Whole-chunk cut: keeps chunks [first_chunk, last_chunk] untouched. - static ReplayCutPlan PlanCutChunks(const VTX::FileFooter& footer, int first_chunk, int last_chunk); - - // Writes the cut to dest_path. Returns false and fills `error` on failure. - // dest_path must differ from source_path. When the plan trims an edge - // chunk, the source is reopened with the SDK reader to re-serialize it. - static bool ExecuteCut(const std::string& source_path, const VTX::FileFooter& footer, const ReplayCutPlan& plan, - const std::string& dest_path, std::string& error); - }; - -} // namespace VtxServices diff --git a/tools/inspector/include/windows/cut_replay_window.h b/tools/inspector/include/windows/cut_replay_window.h index 87b405c..3306a34 100644 --- a/tools/inspector/include/windows/cut_replay_window.h +++ b/tools/inspector/include/windows/cut_replay_window.h @@ -5,7 +5,8 @@ #include #include "gui/gui_layer.h" -#include "services/replay_cut_service.h" +#include "vtx/common/vtx_types.h" +#include "vtx/transform/vtx_replay_cut.h" class InspectorSession; @@ -43,8 +44,8 @@ class CutReplayWindow : public IGuiLayer { // Resolves the current inputs to an inclusive frame range; false when the // inputs cannot be parsed (error_out explains why). bool ResolveRequestedFrames(int& start_frame, int& end_frame, std::string& error_out) const; - void StartCut(const VtxServices::ReplayCutPlan& plan, const std::string& dest_path); - static Outcome RunCut(std::string source_path, VTX::FileFooter footer, VtxServices::ReplayCutPlan plan, + void StartCut(const VTX::ReplayCutPlan& plan, const std::string& dest_path); + static Outcome RunCut(std::string source_path, VTX::FileFooter footer, VTX::ReplayCutPlan plan, std::string dest_path); std::shared_ptr session_; diff --git a/tools/inspector/include/windows/filter_entities_window.h b/tools/inspector/include/windows/filter_entities_window.h index 01553bc..4037b28 100644 --- a/tools/inspector/include/windows/filter_entities_window.h +++ b/tools/inspector/include/windows/filter_entities_window.h @@ -10,7 +10,7 @@ #include "gui/gui_layer.h" #include "vtx/common/vtx_property_cache.h" -#include "vtx/writer/core/vtx_replay_filter.h" +#include "vtx/transform/vtx_replay_filter.h" class InspectorSession; diff --git a/tools/inspector/include/windows/repair_replay_window.h b/tools/inspector/include/windows/repair_replay_window.h index a17dfb3..0f0a50f 100644 --- a/tools/inspector/include/windows/repair_replay_window.h +++ b/tools/inspector/include/windows/repair_replay_window.h @@ -4,7 +4,7 @@ #include #include "gui/gui_layer.h" -#include "vtx/writer/core/vtx_replay_recovery.h" +#include "vtx/transform/vtx_replay_recovery.h" // Floating window that repairs a crashed .vtx from its ".recovery" sidecar via // VTX::RepairReplayFile. The user selects the .vtx (and, if it lives elsewhere, the diff --git a/tools/inspector/src/services/replay_cut_service.cpp b/tools/inspector/src/services/replay_cut_service.cpp deleted file mode 100644 index 6aa805b..0000000 --- a/tools/inspector/src/services/replay_cut_service.cpp +++ /dev/null @@ -1,328 +0,0 @@ -#include "services/replay_cut_service.h" - -#include -#include -#include -#include -#include -#include - -#include -#include - -#include "vtx/reader/core/vtx_reader_facade.h" -#include "vtx/writer/policies/formatters/flatbuffers_vtx_policy.h" - -namespace { - - constexpr double kTicksPerSecond = 10'000'000.0; - - // Streams [offset, offset+bytes) from `in` to `out` through a bounded buffer. - bool CopyRange(std::ifstream& in, std::ofstream& out, uint64_t offset, uint64_t bytes, std::string& error) { - constexpr size_t kBufferSize = 4 * 1024 * 1024; - std::vector buffer(std::min(bytes, kBufferSize)); - in.seekg(static_cast(offset)); - uint64_t remaining = bytes; - while (remaining > 0) { - const size_t step = static_cast(std::min(remaining, buffer.size())); - if (!in.read(buffer.data(), static_cast(step))) { - error = "Read failed at offset " + std::to_string(offset + (bytes - remaining)); - return false; - } - if (!out.write(buffer.data(), static_cast(step))) { - error = "Write failed"; - return false; - } - remaining -= step; - } - return true; - } - - // Mirror of ChunkedFileSink::CompressIfBeneficial with the writer defaults - // (skip tiny payloads, keep the raw payload when zstd does not shrink it). - std::string CompressIfBeneficial(std::string payload) { - constexpr int kCompressionLevel = 10; - if (payload.size() < 512) { - return payload; - } - const size_t max_size = ZSTD_compressBound(payload.size()); - std::string compressed(max_size, '\0'); - const size_t compressed_size = - ZSTD_compress(compressed.data(), max_size, payload.data(), payload.size(), kCompressionLevel); - if (ZSTD_isError(compressed_size) || compressed_size >= payload.size()) { - return payload; - } - compressed.resize(compressed_size); - return compressed; - } - - // Duration of the sliced range from a tick table (first..last nonzero), or a - // negative value when the table cannot answer. - double DurationFromTicks(const std::vector& ticks) { - int64_t first = 0; - int64_t last = 0; - for (const int64_t t : ticks) { - if (t != 0) { - if (first == 0) { - first = t; - } - last = t; - } - } - if (first == 0 || last < first) { - return -1.0; - } - return static_cast(last - first) / kTicksPerSecond; - } - - // Serializes frames [kept_lo, kept_hi] of the source into a fresh chunk - // payload (mirrors ChunkedFileSink::SaveChunk framing: length-prefixed, - // compressed when beneficial, checksummed on the on-disk bytes). - bool WriteRewrittenChunk(VTX::IVtxReaderFacade& reader, std::ofstream& out, int kept_lo, int kept_hi, - int new_chunk_index, int rebase_frame, VTX::ChunkIndexData& entry_out, - std::string& error) { - std::vector> frames; - frames.reserve(static_cast(kept_hi - kept_lo + 1)); - for (int frame = kept_lo; frame <= kept_hi; ++frame) { - const VTX::Frame* source_frame = reader.GetFrameSync(frame); - if (!source_frame) { - error = "Could not read frame " + std::to_string(frame) + " from the source replay."; - return false; - } - frames.push_back(std::make_unique(*source_frame)); - } - - std::string payload = - VTX::FlatBuffersVtxPolicy::SerializeChunk(frames, new_chunk_index, /*is_compressed=*/true); - payload = CompressIfBeneficial(std::move(payload)); - - entry_out.chunk_index = new_chunk_index; - entry_out.file_offset = static_cast(out.tellp()); - entry_out.chunk_size_bytes = static_cast(payload.size() + sizeof(uint32_t)); - entry_out.start_frame = kept_lo - rebase_frame; - entry_out.end_frame = kept_hi - rebase_frame; - entry_out.checksum = XXH3_64bits(payload.data(), payload.size()); - - const uint32_t payload_size = static_cast(payload.size()); - if (!out.write(reinterpret_cast(&payload_size), sizeof(payload_size)) || - !out.write(payload.data(), static_cast(payload.size()))) { - error = "Write failed for rewritten chunk " + std::to_string(new_chunk_index); - return false; - } - return true; - } - - VtxServices::ReplayCutPlan PlanForChunkSpan(const VTX::FileFooter& footer, int first_chunk, int last_chunk, - int first_frame, int last_frame) { - VtxServices::ReplayCutPlan plan; - plan.first_chunk = first_chunk; - plan.last_chunk = last_chunk; - plan.first_frame = first_frame; - plan.last_frame = last_frame; - plan.trims_head = footer.chunk_index[static_cast(first_chunk)].start_frame < first_frame; - plan.trims_tail = footer.chunk_index[static_cast(last_chunk)].end_frame > last_frame; - for (int i = first_chunk; i <= last_chunk; ++i) { - plan.chunk_bytes += footer.chunk_index[static_cast(i)].chunk_size_bytes; - } - plan.valid = true; - return plan; - } - -} // namespace - -namespace VtxServices { - - ReplayCutPlan ReplayCutService::PlanCutFrames(const VTX::FileFooter& footer, int start_frame, int end_frame) { - ReplayCutPlan plan; - if (footer.chunk_index.empty() || footer.total_frames <= 0) { - plan.error = "The loaded replay has no chunk index."; - return plan; - } - start_frame = std::clamp(start_frame, 0, footer.total_frames - 1); - end_frame = std::clamp(end_frame, 0, footer.total_frames - 1); - if (end_frame < start_frame) { - plan.error = "End of range is before its start."; - return plan; - } - - int first_chunk = -1; - int last_chunk = -1; - for (int i = 0; i < static_cast(footer.chunk_index.size()); ++i) { - const auto& entry = footer.chunk_index[static_cast(i)]; - if (first_chunk < 0 && start_frame <= entry.end_frame) { - first_chunk = i; - } - if (end_frame >= entry.start_frame) { - last_chunk = i; - } - } - if (first_chunk < 0 || last_chunk < first_chunk) { - plan.error = "The requested range does not overlap any chunk."; - return plan; - } - // Clamp the exact bounds into the covered span (start may fall in a gap - // between chunks; snap it to the first covered frame). - const int lo = std::max(start_frame, footer.chunk_index[static_cast(first_chunk)].start_frame); - const int hi = std::min(end_frame, footer.chunk_index[static_cast(last_chunk)].end_frame); - return PlanForChunkSpan(footer, first_chunk, last_chunk, lo, hi); - } - - ReplayCutPlan ReplayCutService::PlanCutChunks(const VTX::FileFooter& footer, int first_chunk, int last_chunk) { - ReplayCutPlan plan; - if (footer.chunk_index.empty()) { - plan.error = "The loaded replay has no chunk index."; - return plan; - } - const int max_chunk = static_cast(footer.chunk_index.size()) - 1; - first_chunk = std::clamp(first_chunk, 0, max_chunk); - last_chunk = std::clamp(last_chunk, 0, max_chunk); - if (last_chunk < first_chunk) { - plan.error = "End chunk is before start chunk."; - return plan; - } - return PlanForChunkSpan(footer, first_chunk, last_chunk, - footer.chunk_index[static_cast(first_chunk)].start_frame, - footer.chunk_index[static_cast(last_chunk)].end_frame); - } - - bool ReplayCutService::ExecuteCut(const std::string& source_path, const VTX::FileFooter& footer, - const ReplayCutPlan& plan, const std::string& dest_path, std::string& error) { - if (!plan.valid) { - error = plan.error.empty() ? "Invalid cut plan." : plan.error; - return false; - } - namespace fs = std::filesystem; - std::error_code ec; - if (fs::equivalent(fs::path(source_path), fs::path(dest_path), ec)) { - error = "Destination must be a different file than the source."; - return false; - } - - // Edge chunks that start/end inside the kept range must be re-serialized - // frame by frame; open a private reader for that (the caller's reader may - // be busy on the UI thread). - VTX::ReaderContext reader_context; - if (plan.trims_head || plan.trims_tail) { - reader_context = VTX::OpenReplayFile(source_path); - if (!reader_context.Loaded() || !reader_context.WaitUntilReady()) { - error = - "Could not reopen the source replay for edge-chunk rewriting: " + reader_context.GetError().message; - return false; - } - } - - std::ifstream in(source_path, std::ios::binary); - if (!in) { - error = "Could not open the source replay: " + source_path; - return false; - } - std::ofstream out(dest_path, std::ios::binary | std::ios::trunc); - if (!out) { - error = "Could not create the output file: " + dest_path; - return false; - } - - // 1) Header region, verbatim: everything before the first chunk of the - // ORIGINAL file (magic + length-prefixed header payload). The header - // carries no frame-dependent data; copying it keeps the schema and - // recording metadata bit-identical. - const uint64_t header_end = footer.chunk_index.front().file_offset; - if (!CopyRange(in, out, 0, header_end, error)) { - return false; - } - - // 2) Chunks. Whole chunks are copied verbatim; partial edge chunks are - // rebuilt with only the kept frames and get fresh index metadata. - std::vector seek_table; - seek_table.reserve(static_cast(plan.last_chunk - plan.first_chunk + 1)); - for (int i = plan.first_chunk; i <= plan.last_chunk; ++i) { - const auto& entry = footer.chunk_index[static_cast(i)]; - const int kept_lo = std::max(entry.start_frame, plan.first_frame); - const int kept_hi = std::min(entry.end_frame, plan.last_frame); - const int new_index = static_cast(seek_table.size()); - - if (kept_lo == entry.start_frame && kept_hi == entry.end_frame) { - VTX::ChunkIndexData rebased; - rebased.chunk_index = new_index; - rebased.file_offset = static_cast(out.tellp()); - rebased.chunk_size_bytes = entry.chunk_size_bytes; - rebased.start_frame = entry.start_frame - plan.first_frame; - rebased.end_frame = entry.end_frame - plan.first_frame; - rebased.checksum = entry.checksum; - seek_table.push_back(rebased); - if (!CopyRange(in, out, entry.file_offset, entry.chunk_size_bytes, error)) { - return false; - } - } else { - VTX::ChunkIndexData rewritten; - if (!WriteRewrittenChunk(*reader_context.reader, out, kept_lo, kept_hi, new_index, plan.first_frame, - rewritten, error)) { - return false; - } - seek_table.push_back(rewritten); - } - } - - // 3) Footer: time table sliced to the kept frames (tick values stay - // absolute; only frame indices are rebased), duration recomputed. - const auto slice_ticks = [&](const std::vector& ticks) { - std::vector sliced; - const size_t begin = std::min(static_cast(plan.first_frame), ticks.size()); - const size_t end = std::min(static_cast(plan.last_frame) + 1, ticks.size()); - sliced.reserve(end - begin); - for (size_t i = begin; i < end; ++i) { - sliced.push_back(static_cast(ticks[i])); - } - return sliced; - }; - const auto slice_frame_indices = [&](const std::vector& values) { - std::vector sliced; - for (const uint32_t value : values) { - const int frame = static_cast(value); - if (frame >= plan.first_frame && frame <= plan.last_frame) { - sliced.push_back(frame - plan.first_frame); - } - } - return sliced; - }; - - const std::vector game_times = slice_ticks(footer.times.game_time); - const std::vector created_utc = slice_ticks(footer.times.created_utc); - const std::vector gaps = slice_frame_indices(footer.times.gaps); - const std::vector segments = slice_frame_indices(footer.times.segments); - - VTX::SessionFooter session_footer; - session_footer.total_frames = plan.last_frame - plan.first_frame + 1; - double duration = DurationFromTicks(created_utc); - if (duration < 0.0) { - duration = DurationFromTicks(game_times); - } - if (duration < 0.0 && footer.total_frames > 0) { - duration = static_cast(footer.duration_seconds) * session_footer.total_frames / footer.total_frames; - } - session_footer.duration_seconds = std::max(duration, 0.0); - session_footer.game_times = &game_times; - session_footer.created_utc = &created_utc; - session_footer.gaps = &gaps; - session_footer.segments = &segments; - - // Written uncompressed; the reader sniffs the zstd magic and passes - // uncompressed payloads through. - const std::string footer_payload = VTX::FlatBuffersVtxPolicy::SerializeFooter(seek_table, session_footer); - const uint32_t footer_size = static_cast(footer_payload.size()); - const std::string magic = VTX::FlatBuffersVtxPolicy::GetMagicBytes(); - if (!out.write(footer_payload.data(), static_cast(footer_payload.size())) || - !out.write(reinterpret_cast(&footer_size), sizeof(footer_size)) || - !out.write(magic.data(), static_cast(magic.size()))) { - error = "Write failed while finishing the footer."; - return false; - } - out.flush(); - if (!out) { - error = "Flush failed for the output file."; - return false; - } - return true; - } - -} // namespace VtxServices diff --git a/tools/inspector/src/windows/cut_replay_window.cpp b/tools/inspector/src/windows/cut_replay_window.cpp index 9cb97a2..661eebd 100644 --- a/tools/inspector/src/windows/cut_replay_window.cpp +++ b/tools/inspector/src/windows/cut_replay_window.cpp @@ -184,15 +184,17 @@ bool CutReplayWindow::ResolveRequestedFrames(int& start_frame, int& end_frame, s } CutReplayWindow::Outcome CutReplayWindow::RunCut(std::string source_path, VTX::FileFooter footer, - VtxServices::ReplayCutPlan plan, std::string dest_path) { + VTX::ReplayCutPlan plan, std::string dest_path) { // Touches only its by-value arguments -- safe to run detached from the window. Outcome out; out.dest_path = dest_path; - out.ok = VtxServices::ReplayCutService::ExecuteCut(source_path, footer, plan, dest_path, out.error); + const VTX::ReplayCutResult result = VTX::CutReplayFile(source_path, footer, plan, dest_path); + out.ok = result.ok(); + out.error = result.error; return out; } -void CutReplayWindow::StartCut(const VtxServices::ReplayCutPlan& plan, const std::string& dest_path) { +void CutReplayWindow::StartCut(const VTX::ReplayCutPlan& plan, const std::string& dest_path) { phase_ = Phase::Running; spinner_tick_ = 0; outcome_ = Outcome {}; @@ -212,10 +214,6 @@ void CutReplayWindow::DrawContent() { range_initialized_ = false; return; } - if (session_->GetFormat() != VTX::VtxFormat::FlatBuffers) { - ImGui::TextColored(kWarn, "Cut currently supports flatbuffers (.vtx \"VTXF\") replays only."); - return; - } if (!range_initialized_) { ResetRangeToFullReplay(); range_initialized_ = true; @@ -273,15 +271,15 @@ void CutReplayWindow::DrawContent() { ImGui::Separator(); // Live plan preview. - VtxServices::ReplayCutPlan plan; + VTX::ReplayCutPlan plan; if (range_mode_ == 3) { - plan = VtxServices::ReplayCutService::PlanCutChunks(footer, chunk_start_, chunk_end_); + plan = VTX::PlanCutChunks(footer, chunk_start_, chunk_end_); } else { int requested_start = 0; int requested_end = 0; std::string input_error; if (ResolveRequestedFrames(requested_start, requested_end, input_error)) { - plan = VtxServices::ReplayCutService::PlanCutFrames(footer, requested_start, requested_end); + plan = VTX::PlanCutFrames(footer, requested_start, requested_end); } else { plan.error = input_error; } From 33dd6dafada2319ace6d2886d6c8e5ce14072b21 Mon Sep 17 00:00:00 2001 From: Alejandro Canela Date: Thu, 10 Sep 2026 15:54:32 +0200 Subject: [PATCH 4/4] fix(build): hide static-archive symbols in libvtx_writer.so (Linux shared) --- CHANGELOG.md | 1 + docs/BUILD.md | 1 + sdk/src/vtx_writer/CMakeLists.txt | 13 +++++++++++++ 3 files changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bd207f5..9a0f1b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **build**: Linux shared builds (`VTX_BUILD_SHARED=ON`) exported the IXWebSocket and mbedTLS symbols `libvtx_writer.so` links from static archives. A consumer that also links its own copy of those archives (`vtx_tests` does, for its in-process WebSocket server) ended up with one interposed set of their globals, constructed by both images and destroyed twice -- `free(): double free detected in tcache 2` at exit. Latent on `main` (the test executable happened to bind to the `.so`'s copy); surfaced by the link-order change that adding `vtx_transform` to the test executable caused. `libvtx_writer.so` now hides the symbols of every static archive it links (`--exclude-libs,ALL`), keeping a private copy exactly like the Windows DLL always did - **docs**: `FILE_FORMAT.md` described the 4 bytes before each chunk as a "marker / padding" and the trailing 4 bytes of the file as a "sentinel"; they are the chunk's size prefix and the repeated format magic - **reader**: `GetFooter().chunk_index[i].checksum` was always 0 on both backends -- the footer-to-native conversions copied every seek-table field except the xxHash64 checksum (the internal seek table used for chunk loading did carry it). Visible as `checksum: 0` for every chunk in the CLI's `chunks` command, and the inspector's cut copied 0 into the seek-table entries of verbatim chunks. The FlatBuffers footer conversion also dropped each timeline event's `entity_unique_id` and `location`. All three conversions now carry the fields; `ReplayFilterTest.DropByStructName` asserts the footer checksums of a freshly written file match the bytes on disk and `ReplayFilterTest.EventsCarriedAndPruned` reads events back with their entity id and location on both backends diff --git a/docs/BUILD.md b/docs/BUILD.md index b34cf19..488139a 100644 --- a/docs/BUILD.md +++ b/docs/BUILD.md @@ -243,6 +243,7 @@ Shared-build caveats: 1. Windows still uses `CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS`. 2. If your process also links another protobuf runtime, ABI duplication is your problem. 3. Installed Linux binaries rely on `$ORIGIN/../lib` RPATH. +4. On Linux, `libvtx_writer.so` hides the symbols of the static archives it links (IXWebSocket, mbedTLS, zstd, flatbuffers) with `--exclude-libs,ALL`, exactly as the Windows DLL does implicitly. A process can therefore link its own copy of those libraries without symbol interposition -- two visible copies of a global with a destructor would otherwise be constructed and destroyed twice (a double free at exit). Consumers get zstd/flatbuffers through `vtx_common`'s link interface, never through `vtx_writer`. ## Outputs diff --git a/sdk/src/vtx_writer/CMakeLists.txt b/sdk/src/vtx_writer/CMakeLists.txt index 41b535e..560ca2f 100644 --- a/sdk/src/vtx_writer/CMakeLists.txt +++ b/sdk/src/vtx_writer/CMakeLists.txt @@ -46,6 +46,19 @@ if(BUILD_SHARED_LIBS AND WIN32) target_link_libraries(vtx_writer PRIVATE $) endif() +# Linux + shared build: IXWebSocket and mbedTLS are linked into this .so from static +# archives. GNU ld would export their symbols with default visibility, so a consumer +# that links its own copy of the same archives (vtx_tests does, for its in-process +# WebSocket server) ends up with ONE interposed set of their globals -- constructed +# by both images and destroyed twice: `free(): double free detected` at exit. +# Hiding every archive keeps the .so self-contained with a private copy, which is +# what the Windows DLL already does (a DLL never exports the symbols of static +# libraries it links). Nothing here is public API: consumers get zstd/flatbuffers +# through vtx_common's link interface, never through vtx_writer. +if(BUILD_SHARED_LIBS AND UNIX AND NOT APPLE) + target_link_options(vtx_writer PRIVATE "LINKER:--exclude-libs,ALL") +endif() + # ============================================================================== # 3. VS filters # ==============================================================================