Add selective multi-mip tile decoding - #269
Open
jsperrier1 wants to merge 1 commit into
Open
Conversation
Decodes a chosen subset of tiles from one or more frames of an access unit into a bounded, tile-major output buffer, reading only the bytes those tiles occupy. oapvd_decode_frame() cannot express this. It writes into a scanline-strided oapv_imgb_t sized to the whole picture, so decoding 16 tiles of a 15360x8640 frame still needs a 506 MiB output buffer where a tile budget needs 4 MiB. A viewport-driven client wants storage proportional to what it asked for, not to the frame's tile count. Tile locations come from the frame header's tile sizes, the same derivation oapvd_info_tile() reports to callers planning a selection: one header pass yields both the frame header and every tile's size, so no tile has to be visited to find the next one. That derivation is now shared between the two, which is the only existing code this touches. - oapv_imgb_t gains an optional tile-major layout, off for zero-initialised structs, so the existing scanline path is unchanged. It describes each component separately, so interleaved chroma is rejected rather than written as if planar. - oapv_mip_request_t::tile_dst_slots lets a caller route each tile to a physical slot, which is what allows a bounded resident-tile cache. - The output buffer's colour space and the bitstream's chroma_format_idc must agree, as they must for oapvd_decode_frame(): the component shifts come from the former and the component count from the latter. - 'mid' is optional, as it is for oapvd_decode(): pass a container to collect the access unit's metadata, or NULL to skip it. Metadata follows the frames, so collecting it means walking to the end of the access unit, whereas otherwise the walk stops at the last requested mip. Verified on a 15360x8640 10-level mip pyramid with 256x256 tiles, whose mip 0 is a 60x34 = 2040 tile grid: every selected tile is sample-identical to an independent full-frame oapvd_decode_frame() of the same level, across all ten levels, for interior and clipped edge tiles, under both tile routings and at one and eight threads. Read amplification is 1.09x at 64 tiles and 1.04x at 232 tiles. A metadata_cll payload appended after the frames round-trips through the optional container. Rejections are covered too: a PLANAR2 buffer returns OAPV_ERR_UNSUPPORTED_COLORSPACE and a 4:4:4 buffer on a 4:2:2 stream returns OAPV_ERR_INVALID_ARGUMENT. ctest 18/18. Signed-off-by: Jean Perrier <jean.perrierr@xa.epicgames.com>
jsperrier1
force-pushed
the
selective_multi_mip_tile_decode
branch
from
August 20, 2026 23:58
a173777 to
2d71707
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add selective multi-mip tile decoding
Adds one decoder entry point that decodes a chosen subset of tiles, from one or more frames of an
access unit, into a bounded tile-major output buffer:
This is the narrowed replacement for #228. Everything else that draft proposed has since landed
upstream in other forms, or is deferred to a follow-up (see below).
Why a dedicated entry point is still needed
oapvd_decode_frame()already acceptspart_tile_idxs, so it can decode a subset of tiles. Whatit cannot express is the output shape: it writes into a scanline-strided
oapv_imgb_tsized tothe whole picture. A partial-tile decode of a 15360x8640 frame therefore still requires a 506 MB
output buffer even when only 16 tiles are wanted.
A viewport-driven client wants storage proportional to its tile budget, not to the frame's
tile count. Decoding the same 16 tiles of mip 0 on a 15360x8640 4:2:2 10-bit frame:
oapvd_decode_frame()withpart_tile_idxsoapv_imgb_t)Both figures are just the layout arithmetic: a 15360x8640 4:2:2 10-bit picture is 506.2 MiB, and a
256x256 tile of the same format is 256 KiB. The ratio grows with frame size and shrinks with the
selection, which is the point - the buffer follows the client's budget instead of the picture.
What it adds
oapv_imgb_toptional tile-major layout —tiled_layout,num_tile_cols,num_tile_rows,tile_size,tile_w[],tile_h[],tile_stride[]. Off for a zero-initialised struct, so thescanline path is untouched.
oapv_mip_request_t— one requested mip level: which tiles, where to put them, andper-request
statusplus frame metadata filled by the decoder.oapv_mip_request_t::tile_dst_slots— optional per-tile destination slot. Without it a tilelands at its natural
(row * num_tile_cols + col)position; with it the caller routes each tileto a physical slot, which is what allows an output buffer sized to a tile budget rather than to
the tile count.
oapv_multi_mip_decode_t— the array of requests for one call.The thread pool gains
oapv_tpool_atomic_inc()to hand work items to competing workers.The only existing code this touches is
oapvd_info_tile(): it and the new path derived tileoffsets from the frame header's sizes with the same running sum, differing only in the bound they
checked, so both now call a shared
dec_derive_tile_offsets(). Everything else is additive.What it does not support
Interleaved chroma (
OAPV_CF_PLANAR2) is rejected withOAPV_ERR_UNSUPPORTED_COLORSPACE. A tile'sdestination is described per component -
tile_w[c],tile_h[c],tile_stride[c], anda[c]biased to the component's intra-tile offset - which cannot express two components sharing a plane
on alternating samples.
oapvd_decode_frame()remains the route for that layout.The output buffer's colour space and the bitstream's
chroma_format_idcmust agree, as they mustfor
oapvd_decode_frame(): the component shifts come from the former and the component count fromthe latter. A mismatch returns
OAPV_ERR_INVALID_ARGUMENTrather than writing chroma at the wrongcoordinates.
How tiles are located
From the frame header's tile sizes — the same derivation
oapvd_info_tile()(#260) reports tocallers planning a selection. One
oapvd_vlc_frame_header_ex()pass yields both the frame headerthe decoder needs and every tile's size, so no tile has to be visited to find the next one. That
property is what keeps a selective decode from touching bytes it does not need.
Frames whose header does not carry tile sizes are rejected with
OAPV_ERR_UNSUPPORTEDrather thansilently falling back to a chain walk.
Why one call across mips, rather than one call per mip
A viewport does not request the same number of tiles at every level. Looking at a planar plate
edge-on under perspective gives a pyramid - a few levels with tens of tiles, then a tail of levels
with a handful or one. A captured example, 117 tiles over six levels:
So a viewport touches six levels per frame, and splitting that into six calls pays each call's
fixed cost six times. Same tiles decoded either way, warm, median of 5:
The penalty grows with thread count, but not for the reason one might assume. Repeating the same
A/B with one tile per level - six tiles total, so almost no decoding to parallelise - isolates the
per-call cost:
That delta is the same ~1.7 ms the real workload shows, so the gap is almost entirely fixed
per-call cost rather than lost parallelism across levels. The cost per call is roughly constant
while the decode itself gets faster with more threads, which is why the percentage grows: at 32
threads six calls' overhead is comparable to the entire decode.
Each call creates and destroys a sync object, makes its own allocations, walks the access unit's
PBU chain from the start, and dispatches then joins the worker pool. Batching pays that once.
Being clear about what this does and does not show: most of the gap is what this implementation
spends per call, and a leaner per-call path would narrow it - no new API is needed for that. What
cannot be avoided is that a call must dispatch and drain the pool before it returns, so some cost
per call is inherent to any shape where the caller asks one level at a time.
Measured on 16 physical cores, so the 32-thread row is hyperthreaded. Warm on purpose: the
mechanism under test is per-call overhead, and cold-cache noise swamps it.
Selectivity
Bytes paged in vs bytes in the selected tiles, cold cache, 4 mips of the 16K pyramid, measured as
a resident-page delta over a memory-mapped access unit:
The residual is 4 KiB page granularity at tile boundaries, so it shrinks as the selection grows.
Tile counts are below the nominal NxN per mip because the block is clamped to each level's grid.
Correctness
On a 15360x8640, 4:2:2 10-bit, 10-level pyramid with 256x256 tiles (mip 0 is a 60x34 = 2040 tile
grid), every selected tile is sample-identical to an independent full-frame
oapvd_decode_frame()of the same level:tile_dst_slots);Also verified on a conventional 1024x512 pyramid, and
ctestpasses 18/18.The two rejections above are covered too: a PLANAR2 output buffer returns
OAPV_ERR_UNSUPPORTED_COLORSPACE, a 4:4:4 buffer handed a 4:2:2 stream returnsOAPV_ERR_INVALID_ARGUMENT, and a correctly matched buffer is unaffected.oapvd_info_tile()still reports offsets identical to an independent walk of the tile chainacross all 2745 tiles of the 16K asset, so sharing the derivation did not change its behaviour.
Input
bitbfollowsoapvd_decode()'s convention:addrpoints at the access unit's signature, pastthe leading 4-byte length field, and
ssizeis the access unit size;bsize, when non-zero, isthe capacity behind
addr. The decoder only reads throughaddrand copies no tile bytes beforedecoding them, so a caller may hand it a memory-mapped file and have the decoder touch only the
pages the selected tiles occupy.
Worth noting for a future cleanup:
oapv_bitb_t::addrisvoid*, notconst void*, so aread-only mapping needs a cast at the call site. Every other decode entry point has the same
wart, so this change does not add one.
Metadata
midis optional, exactly as it is foroapvd_decode(): pass a container to collect the accessunit's metadata, or NULL to skip it.
Skipping is the cheaper path, and deliberately so. Metadata is written after the frames, so
collecting it means walking to the end of the access unit; with NULL the walk stops at the last
requested mip. On a 27-tile selection from the 16K pyramid that difference is 90 resident pages
versus 97 — small, but it is the kind of thing a viewport client doing this every frame would
rather not pay for metadata it already has.
Verified by appending a
metadata_cll()payload after the frames of the 16K asset: it round-tripsthrough
oapvm_get_all()with the right group ID, type and values, tile output stayssample-identical, and an access unit carrying no metadata yields zero payloads rather than an
error.
Not in this PR
Deliberately left out, to keep this reviewable as one idea. Happy to open any of them as
follow-ups:
in warm playback, so it does not belong in the load-bearing change).