Skip to content

Add selective multi-mip tile decoding - #269

Open
jsperrier1 wants to merge 1 commit into
AcademySoftwareFoundation:mainfrom
alejandro-arango-epicgames:selective_multi_mip_tile_decode
Open

Add selective multi-mip tile decoding#269
jsperrier1 wants to merge 1 commit into
AcademySoftwareFoundation:mainfrom
alejandro-arango-epicgames:selective_multi_mip_tile_decode

Conversation

@jsperrier1

Copy link
Copy Markdown
Contributor

Add selective multi-mip tile decoding

Adds one decoder entry point that decodes a chosen subset of tiles, from one or more frames of an
access unit, into a bounded tile-major output buffer:

OAPV_EXPORT int oapvd_decode_selective_multi_mips(oapvd_t did, oapv_bitb_t *bitb,
                                                  oapv_multi_mip_decode_t *multi_mip_decode,
                                                  oapvm_t mid, oapvd_stat_t *stat);

This is the narrowed replacement for #228. Everything else that draft proposed has since landed
upstream in other forms, or is deferred to a follow-up (see below).

Why a dedicated entry point is still needed

oapvd_decode_frame() already accepts part_tile_idxs, so it can decode a subset of tiles. What
it cannot express is the output shape: it writes into a scanline-strided oapv_imgb_t sized to
the whole picture. A partial-tile decode of a 15360x8640 frame therefore still requires a 506 MB
output buffer even when only 16 tiles are wanted.

A viewport-driven client wants storage proportional to its tile budget, not to the frame's
tile count. Decoding the same 16 tiles of mip 0 on a 15360x8640 4:2:2 10-bit frame:

output buffer required
oapvd_decode_frame() with part_tile_idxs 506.2 MiB (full-frame oapv_imgb_t)
this entry point 4 MiB (16 slots x 256 KiB)

Both figures are just the layout arithmetic: a 15360x8640 4:2:2 10-bit picture is 506.2 MiB, and a
256x256 tile of the same format is 256 KiB. The ratio grows with frame size and shrinks with the
selection, which is the point - the buffer follows the client's budget instead of the picture.

What it adds

  • oapv_imgb_t optional tile-major layouttiled_layout, num_tile_cols, num_tile_rows,
    tile_size, tile_w[], tile_h[], tile_stride[]. Off for a zero-initialised struct, so the
    scanline path is untouched.
  • oapv_mip_request_t — one requested mip level: which tiles, where to put them, and
    per-request status plus frame metadata filled by the decoder.
  • oapv_mip_request_t::tile_dst_slots — optional per-tile destination slot. Without it a tile
    lands at its natural (row * num_tile_cols + col) position; with it the caller routes each tile
    to a physical slot, which is what allows an output buffer sized to a tile budget rather than to
    the tile count.
  • oapv_multi_mip_decode_t — the array of requests for one call.

The thread pool gains oapv_tpool_atomic_inc() to hand work items to competing workers.

The only existing code this touches is oapvd_info_tile(): it and the new path derived tile
offsets from the frame header's sizes with the same running sum, differing only in the bound they
checked, so both now call a shared dec_derive_tile_offsets(). Everything else is additive.

What it does not support

Interleaved chroma (OAPV_CF_PLANAR2) is rejected with OAPV_ERR_UNSUPPORTED_COLORSPACE. A tile's
destination is described per component - tile_w[c], tile_h[c], tile_stride[c], and a[c]
biased to the component's intra-tile offset - which cannot express two components sharing a plane
on alternating samples. oapvd_decode_frame() remains the route for that layout.

The output buffer's colour space and the bitstream's chroma_format_idc must agree, as they must
for oapvd_decode_frame(): the component shifts come from the former and the component count from
the latter. A mismatch returns OAPV_ERR_INVALID_ARGUMENT rather than writing chroma at the wrong
coordinates.

How tiles are located

From the frame header's tile sizes — the same derivation oapvd_info_tile() (#260) reports to
callers planning a selection. One oapvd_vlc_frame_header_ex() pass yields both the frame header
the decoder needs and every tile's size, so no tile has to be visited to find the next one. That
property is what keeps a selective decode from touching bytes it does not need.

Frames whose header does not carry tile sizes are rejected with OAPV_ERR_UNSUPPORTED rather than
silently falling back to a chain walk.

Why one call across mips, rather than one call per mip

A viewport does not request the same number of tiles at every level. Looking at a planar plate
edge-on under perspective gives a pyramid - a few levels with tens of tiles, then a tail of levels
with a handful or one. A captured example, 117 tiles over six levels:

mip 0 1 2 3 4 5
tiles 27 31 35 18 5 1

So a viewport touches six levels per frame, and splitting that into six calls pays each call's
fixed cost six times. Same tiles decoded either way, warm, median of 5:

threads one call six calls penalty
8 6.66 ms 7.85 ms +18%
16 3.89 ms 5.60 ms +44%
32 2.93 ms 4.86 ms +66%

The penalty grows with thread count, but not for the reason one might assume. Repeating the same
A/B with one tile per level - six tiles total, so almost no decoding to parallelise - isolates the
per-call cost:

threads one call six calls delta per extra call
8 0.88 ms 2.54 ms 1.66 ms 0.33 ms
16 0.90 ms 2.56 ms 1.66 ms 0.33 ms
32 0.79 ms 2.55 ms 1.76 ms 0.35 ms

That delta is the same ~1.7 ms the real workload shows, so the gap is almost entirely fixed
per-call cost rather than lost parallelism across levels. The cost per call is roughly constant
while the decode itself gets faster with more threads, which is why the percentage grows: at 32
threads six calls' overhead is comparable to the entire decode.

Each call creates and destroys a sync object, makes its own allocations, walks the access unit's
PBU chain from the start, and dispatches then joins the worker pool. Batching pays that once.

Being clear about what this does and does not show: most of the gap is what this implementation
spends per call, and a leaner per-call path would narrow it - no new API is needed for that. What
cannot be avoided is that a call must dispatch and drain the pool before it returns, so some cost
per call is inherent to any shape where the caller asks one level at a time.

Measured on 16 physical cores, so the 32-thread row is hyperthreaded. Warm on purpose: the
mechanism under test is per-call overhead, and cold-cache noise swamps it.

Selectivity

Bytes paged in vs bytes in the selected tiles, cold cache, 4 mips of the 16K pyramid, measured as
a resident-page delta over a memory-mapped access unit:

selection tiles needed paged in amplification
2x2 / mip 16 212 KiB 264 KiB 1.24x
4x4 / mip 64 815 KiB 888 KiB 1.09x
8x8 / mip 232 2933 KiB 3052 KiB 1.04x
12x12 / mip 436 5367 KiB 5520 KiB 1.03x

The residual is 4 KiB page granularity at tile boundaries, so it shrinks as the selection grows.
Tile counts are below the nominal NxN per mip because the block is clamped to each level's grid.

Correctness

On a 15360x8640, 4:2:2 10-bit, 10-level pyramid with 256x256 tiles (mip 0 is a 60x34 = 2040 tile
grid), every selected tile is sample-identical to an independent full-frame
oapvd_decode_frame() of the same level:

  • all ten levels, interior tiles and clipped edge tiles;
  • both tile routings (natural position and tile_dst_slots);
  • at one thread and at eight.

Also verified on a conventional 1024x512 pyramid, and ctest passes 18/18.

The two rejections above are covered too: a PLANAR2 output buffer returns
OAPV_ERR_UNSUPPORTED_COLORSPACE, a 4:4:4 buffer handed a 4:2:2 stream returns
OAPV_ERR_INVALID_ARGUMENT, and a correctly matched buffer is unaffected.

oapvd_info_tile() still reports offsets identical to an independent walk of the tile chain
across all 2745 tiles of the 16K asset, so sharing the derivation did not change its behaviour.

Input

bitb follows oapvd_decode()'s convention: addr points at the access unit's signature, past
the leading 4-byte length field, and ssize is the access unit size; bsize, when non-zero, is
the capacity behind addr. The decoder only reads through addr and copies no tile bytes before
decoding them, so a caller may hand it a memory-mapped file and have the decoder touch only the
pages the selected tiles occupy.

Worth noting for a future cleanup: oapv_bitb_t::addr is void*, not const void*, so a
read-only mapping needs a cast at the call site. Every other decode entry point has the same
wart, so this change does not add one.

Metadata

mid is optional, exactly as it is for oapvd_decode(): pass a container to collect the access
unit's metadata, or NULL to skip it.

Skipping is the cheaper path, and deliberately so. Metadata is written after the frames, so
collecting it means walking to the end of the access unit; with NULL the walk stops at the last
requested mip. On a 27-tile selection from the 16K pyramid that difference is 90 resident pages
versus 97 — small, but it is the kind of thing a viewport client doing this every frame would
rather not pay for metadata it already has.

Verified by appending a metadata_cll() payload after the frames of the 16K asset: it round-trips
through oapvm_get_all() with the right group ID, type and values, tile output stays
sample-identical, and an access unit carrying no metadata yields zero payloads rather than an
error.

Not in this PR

Deliberately left out, to keep this reviewable as one idea. Happy to open any of them as
follow-ups:

  • a logging callback API;
  • CPU trace callbacks;
  • an encoder option to write a mip chain into each access unit;
  • input prefetch for mapped access units (platform-specific, off by default, and measured neutral
    in warm playback, so it does not belong in the load-bearing change).

Decodes a chosen subset of tiles from one or more frames of an access unit
into a bounded, tile-major output buffer, reading only the bytes those tiles
occupy.

oapvd_decode_frame() cannot express this. It writes into a scanline-strided
oapv_imgb_t sized to the whole picture, so decoding 16 tiles of a 15360x8640
frame still needs a 506 MiB output buffer where a tile budget needs 4 MiB. A
viewport-driven client wants storage proportional to what it asked for, not to
the frame's tile count.

Tile locations come from the frame header's tile sizes, the same derivation
oapvd_info_tile() reports to callers planning a selection: one header pass
yields both the frame header and every tile's size, so no tile has to be
visited to find the next one. That derivation is now shared between the two,
which is the only existing code this touches.

- oapv_imgb_t gains an optional tile-major layout, off for zero-initialised
  structs, so the existing scanline path is unchanged. It describes each
  component separately, so interleaved chroma is rejected rather than written
  as if planar.
- oapv_mip_request_t::tile_dst_slots lets a caller route each tile to a
  physical slot, which is what allows a bounded resident-tile cache.
- The output buffer's colour space and the bitstream's chroma_format_idc must
  agree, as they must for oapvd_decode_frame(): the component shifts come from
  the former and the component count from the latter.
- 'mid' is optional, as it is for oapvd_decode(): pass a container to collect
  the access unit's metadata, or NULL to skip it. Metadata follows the frames,
  so collecting it means walking to the end of the access unit, whereas
  otherwise the walk stops at the last requested mip.

Verified on a 15360x8640 10-level mip pyramid with 256x256 tiles, whose mip 0
is a 60x34 = 2040 tile grid: every selected tile is sample-identical to an
independent full-frame oapvd_decode_frame() of the same level, across all ten
levels, for interior and clipped edge tiles, under both tile routings and at
one and eight threads. Read amplification is 1.09x at 64 tiles and 1.04x at
232 tiles. A metadata_cll payload appended after the frames round-trips
through the optional container. Rejections are covered too: a PLANAR2 buffer
returns OAPV_ERR_UNSUPPORTED_COLORSPACE and a 4:4:4 buffer on a 4:2:2 stream
returns OAPV_ERR_INVALID_ARGUMENT. ctest 18/18.

Signed-off-by: Jean Perrier <jean.perrierr@xa.epicgames.com>
@jsperrier1
jsperrier1 force-pushed the selective_multi_mip_tile_decode branch from a173777 to 2d71707 Compare August 20, 2026 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant