Skip to content

feat(writer): FilterReplayFile rewrites a replay without selected entities (bucket/struct/id/property rules); inspector File > Filter Entities - #48

Open
onur-zenos wants to merge 2 commits into
mainfrom
feat/replay-entity-filter
Open

onur-zenos wants to merge 2 commits into
mainfrom
feat/replay-entity-filter

Conversation

@onur-zenos

Copy link
Copy Markdown
Contributor

Summary

Rewrite a .vtx replay into a new file without selected entities. The cut (#46) slices a frame range; this filters what is in every frame. Every frame survives, the header block is copied verbatim, every chunk is re-serialized with the surviving entities, and the footer keeps total_frames, duration and the per-frame time table. Only the seek table changes.

SDK: VTX::FilterReplayFile

sdk/include/vtx/writer/core/vtx_replay_filter.h, next to RepairReplayFile.

  • ReplayFilterSpec: Drop (blacklist: entities matching any rule go) or Keep (whitelist: only matching entities stay); rules combine as a union.
  • Rule kinds: Bucket (schema bucket name glob, or the bucket index as digits), UniqueId glob, StructName glob, Property(struct, field, value-glob) where the struct may be empty for "any struct declaring that scalar field" (Bool/Int32/Int64/Float/Double/String). Optional ASCII case-insensitive matching. Glob = * and ?.
  • A dropped bucket is emptied in every frame; its slot stays, because bucket index is positional and the schema names buckets by index. Nested containers inside a kept entity are never touched. The schema is never pruned.
  • Unresolvable rules fail before anything is written: unknown struct or field, non-scalar field, empty pattern, bucket rule matching no schema bucket (the error lists the available names).
  • Chunks keep their frame ranges and their per-chunk zstd choice (sniffed from the source), get fresh xxHash64 checksums; the footer's time table, gaps and segments are carried as-is. Timeline events ride along through the new SessionFooter::events (both footer serializers) and are pruned by default to entities that still exist.
  • Progress callback with cancel (the partial destination is removed). ReplayFilterResult reports frames, chunks, entities seen/kept/dropped, drops per struct and per bucket, event counts, sizes, elapsed.
  • ReplayFilterMatcher (Matches / Keeps / Apply per bucket index, BucketMatches, BucketName) and GlobMatch expose the rule engine for previews on in-memory frames.
  • Both backends; the output keeps the source format. vtx_writer now links vtx_reader privately; the entry point is compiled only when VTX_BUILD_READER is on.

Inspector: File > Filter Entities

Buckets and structs checklists with current-frame counts, unique id globs (one per line), property rules (struct / field / value glob), case-insensitive and event-pruning toggles. A live preview shows what the rules keep per bucket on the current frame and reports unresolvable rules before anything runs. The rewrite runs on a worker thread with a chunk progress bar and Cancel, then reports frames, kept/dropped entities, per-bucket and per-struct drops, event counts and sizes.

Reader fix found on the way

GetFooter().chunk_index[i].checksum was always 0 on both backends: the footer-to-native conversions copied every seek-table field except the checksum (the internal seek table used for chunk loading had it). Visible as checksum: 0 for every chunk in the CLI's chunks command; the cut copied 0 into verbatim entries. The FlatBuffers footer conversion also dropped each event's entity_unique_id and location. All three conversions now carry the fields.

Tests and verification

  • tests/writer/test_replay_filter.cpp: 27 tests. ReplayFilterTest.* on both backends (struct / unique id / property / numeric / case-insensitive / bucket-by-name / bucket-by-index-and-glob / keep-bucket rules, keep-nothing, events carried and pruned, error paths, cancel) plus ReplayFilterGlob and ReplayFilterMatcher unit tests. Full suite green (456 tests, Windows Release static, samples on).
  • Real capture (FlatBuffers, 2811 frames, 10 chunks, 80374 entities): ten rule sets rewritten and independently verified frame by frame. Survivors bitwise identical to the source (real captures carry NaN floats), seek table matches the bytes on disk, header block byte-identical, and the CLI reports identical header, schema, footer, chunk ranges, events and full per-frame time table. Files written from the inspector are byte-identical to the SDK output for the same rules.

Docs: new "Filtering entities into a new replay" section in docs/SDK_API.md; CHANGELOG under Unreleased.

…ities; inspector File > Filter Entities

- VTX::FilterReplayFile(src, dst, spec, progress): drop (blacklist) or keep
  (whitelist) entities by unique id glob, struct name glob or scalar property
  value (struct.field resolved through the file's schema; empty struct = any
  struct with that field). Every frame survives: the header block is copied
  verbatim, every chunk is re-serialized with the surviving entities (same frame
  ranges, same per-chunk compression, fresh xxHash64 checksums) and the footer
  keeps total_frames, duration and the per-frame time table; only the seek table
  changes. Timeline events are carried through the new SessionFooter::events
  (both policies) and pruned to entities that still exist. Progress callback
  with cancel; ReplayFilterResult reports kept/dropped, per-struct drops, event
  counts and sizes. ReplayFilterMatcher / GlobMatch expose the rule engine for
  previews. vtx_writer links vtx_reader privately; the entry point is compiled
  only when VTX_BUILD_READER is on.
- tools/inspector: File > Filter Entities window -- struct checklist with
  current-frame counts, unique id globs, property rules, case-insensitive and
  event-pruning toggles, live preview on the current frame, worker thread with
  chunk progress and Cancel, result summary.
- reader: GetFooter() now carries chunk_index[i].checksum on both backends
  (was always 0) and, for FlatBuffers, each event's entity_unique_id and
  location.
- tests: ReplayFilterTest (both backends), ReplayFilterGlob, ReplayFilterMatcher.
  Docs: SDK_API.md section, CHANGELOG.
…the inspector

- ReplayFilterRule::Bucket(pattern): selects every entity of the buckets whose
  schema name matches the glob, or of the bucket at that index when the pattern
  is digits only (also for buckets the schema does not name). Dropping a bucket
  empties it in every frame; the slot stays because bucket index is positional.
  A bucket rule that matches no schema bucket is an error listing the available
  names. ReplayFilterMatcher queries now take the bucket index (Matches / Keeps /
  Apply) and expose BucketMatches / BucketName; ReplayFilterResult adds
  dropped_by_bucket.
- tools/inspector: Filter Entities gets a Buckets checklist (index, name,
  current-frame count) above the struct list; the outcome lists drops per bucket.
- tests: DropByBucketName, KeepBucketCombinesWithOtherRules, BucketByIndexAndGlob
  (both backends, three-bucket schema) and BucketRulesResolveAgainstSchemaBuckets.
  Docs and CHANGELOG updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant