[kyzer1023]

What changed

On Windows, AMD screen recordings were always routed to the FFmpeg h264_amf fallback. That path reads each frame back to the CPU, converts RGBA to NV12, then encodes. This PR enables the native Media Foundation D3D11 texture path for AMD screen recording, using GPU video-processor conversion and the MF H.264 MFT without CPU readback.

AMD camera recording retains its existing FFmpeg fallback. Camera setup describes a decoded frame, while capture can still deliver compressed MJPEG buffers that the native texture uploader cannot handle. Keeping that fallback preserves MJPEG decoding.

AMD-only adjustments in cap-enc-mediafoundation:

• BT.709 limited-range signalling: the input and output media types and the D3D11 video-processor color spaces are set explicitly, so the AMD MFT output matches the FFmpeg path (color_range=tv, color_space=bt709). Without this, decoded colors drifted, e.g. source RGB (125,62,93) decoded as (127,68,91). With it, the MF output decodes as (122,60,90), the same as the fallback.
• Bounded drain on stop: AMD's MFT holds queued input when the stream stops, and the existing flush threw away the tail (16 of 120 frames in a constant-frame probe). On AMD, stop now sends END_OF_STREAM + COMMAND_DRAIN and waits for METransformDrainComplete, with an ENCODER_OPERATION_TIMEOUT guard. Other vendors keep the current stop behavior.

Screen recording still falls back to FFmpeg when MF init fails or software encoding is preferred (should_force_software()). An ignored hardware test (amd_muxer_tries_textures_and_preserves_ffmpeg_fallback) checks native selection, fallback after an injected MF init failure, and the explicit software preference. crates/recording/examples/mf-texture-benchmark.rs is a reusable benchmark that compares the two paths.

Non-overlap with #2940 / #2999

This PR only changes encoder selection and the AMD MF texture path for non-fragmented MP4. The instant-mode and fragmented M4S paths are untouched. It does not include any of the readback or fragmented AMF work from #2940 or #2999.

Benchmarks

Hardware: AMD Ryzen 7 8745HS / Radeon 780M (driver 32.0.21030.2001), Windows, rustc 1.88.0, release profile.
Method: synthetic moving RGBA gradients from 8 pre-created D3D11 textures, 300 frames per trial at 30 fps timestamps (unpaced), run order alternated between paths. Each run did 6 trials: trial 0 was a warmup and is excluded, leaving 5 measured trials per path per resolution. Wall time covers readback/conversion/encode/mux/drain per input frame. CPU is process user+kernel time per input frame. Init, texture generation, trailer, and decode verification are not timed.

cargo run -p cap-recording --example mf-texture-benchmark --release -- 1920 1080 300 6 <out-dir>
cargo run -p cap-recording --example mf-texture-benchmark --release -- 3840 2160 300 6 <out-dir>

Resolution	Path	Wall ms/frame (median)	Wall ms/frame (mean)	CPU ms/frame (mean)
1920x1080	FFmpeg AMF fallback	3.412	3.510	2.958
1920x1080	MF texture	4.164	3.792	1.083
3840x2160	FFmpeg AMF fallback	12.502	12.144	13.333
3840x2160	MF texture	7.719	7.783	1.760

• 4K: median wall 1.62x faster (mean 1.56x), mean CPU 7.57x lower (-86.8%).
• 1080p: median wall 0.82x (slower; mean 0.93x), mean CPU 2.73x lower (-63.4%). Wall time at 1080p is worse here. The CPU savings are what this path buys at that resolution.

Every output file decoded all 300 frames, and every decoded frame passed a synthetic BT.709 luma check. ffprobe on the MF output: 30/1 fps, 10.000 s, has_b_frames=0, color_range=tv, color_space=bt709.

Test plan

• cargo fmt --all
• cargo check -p cap-recording -p cap-enc-mediafoundation --lib
• cargo clippy -p cap-recording -p cap-enc-mediafoundation --lib --example mf-texture-benchmark -- -D warnings (re-run after rebasing onto current main)
• cargo test -p cap-recording --release --lib output_pipeline::win::tests -- --include-ignored --test-threads=1 (3 passed on AMD 780M)
• cargo test -p cap-recording --release --test hardware_compat -- --test-threads=1 (13 passed, 7 ignored)
• cargo test -p cap-recording --release --test hardware_compat test_amd_amf_encoding -- --ignored --test-threads=1
• One-frame release smoke test on both paths; benchmark above
• Not tested: live camera capture (existing AMD fallback retained), Intel/NVIDIA hardware (behavior unchanged), instant/fragmented modes (unchanged)

Confidence Score: 5/5

The PR appears safe to merge, with no new findings.

Summary

Enables native AMD texture encoding for Windows screen recordings while keeping the camera fallback.

• AMD screen recordings can use the Media Foundation texture encoder.
• Wayland display queries wait longer for busy compositors.
• Hyprland screenshot checks use the monitor's reported size.
• Linux camera CI loads video modules and headers before running.

_{Reviews (8) · Last reviewed commit: "chore: retry CI after hosted runner disc..." · Reviewed by Greptile}

[vercel]

@kyzer1023 is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

[kyzer1023]

@greptileai review

[richiemcilroy]

Thanks for this, the BT.709 and drain work is well investigated. A few things need fixing before we can take it:

1. Stopping after a pause longer than 5 seconds fails on AMD. While paused, FramePacer repeats nothing, so get_frame blocks and then returns None on stop. The idle clock reset only happens in the Some branch, so the first check_health() after entering drain reports Timeout, flushes the tail and fails the recording (h264.rs around 680-687). Please reset the health monitor's output clock when entering drain, or skip check_health while draining and rely on the drain deadline.
2. If the drain deadline is hit (h264.rs around 582-591), please log a warning and fall through to the normal END_OF_STREAM / END_STREAMING / FLUSH shutdown instead of returning EncoderUnhealthy. Today AMD always goes through FFmpeg and stop never fails this way, so a slow drain should not turn into a failed recording.
3. check_pixels in mf-texture-benchmark reads pixel (1000, 500) regardless of the frame size, so smaller runs panic or compare the wrong pixel. Please pick coordinates inside the frame and use the same ones for the expected value.
4. Minor: cap_frame_converter treats both 0x1002 and 0x1022 as AMD. Please match that (or reuse its vendor mapping) for is_amd.

This also replaces the AMD to FFmpeg routing we added on purpose, so we will need to test it on real AMD hardware (studio screen, studio camera and instant camera-only, including pause then stop) and confirm NVIDIA and Intel are unaffected before merging. Once the fixes are in, it needs a 5/5 from Greptile plus green CI to merge.

[kyzer1023]

Addressed all four on 1b22812b:

1. Pause >5s then stop — on AMD drain start, reset the health-monitor idle clock via notify_input_stall_recovered(), and skip check_health while draining so only the drain deadline applies.
2. Drain deadline — logs a tracing::warn! and falls through to the normal END_OF_STREAM / END_STREAMING / FLUSH shutdown instead of returning EncoderUnhealthy.
3. check_pixels — picks x/y inside the decoded frame (capped at 1000/500 and half-size, centred in a 16px cell) and uses the same coords for expected color and the read. Smoke at 640×480 passed at (324, 244).
4. is_amd — now matches!(adapter.VendorId, 0x1002 | 0x1022), matching cap_frame_converter::GpuVendor::from_id.

640×480×60 smoke on Radeon 780M: texture path 60 in/out/decoded, healthy=true. Requesting Greptile recheck on the new head.

[kyzer1023]

@greptileai review

[kyzer1023]

@greptileai review

[kyzer1023]

@richiemcilroy Ready for another look — Greptile clean and CI green on 6a1c312e5 (your four points + clippy drain-timeout collapse + rustfmt). Ignoring Vercel auth fails.

[kyzer1023]

@greptileai review

[kyzer1023]

@greptileai review

[greptile-apps]

Comments Outside Diff

These findings could not be posted inline.

•  Resume loses texture encoding crates/recording/src/output_pipeline/win.rs:854 ▶

  The AMD camera bypass calls fallback(None), which sets the shared encoder_preferences flag. Screen and camera recording share that flag across segments. After a user pauses and resumes a screen recording with a camera, WindowsMuxer sees the flag and skips the native texture encoder for every remaining segment. This brings back CPU readback and conversion, losing the performance benefit added here.

  Keep the intentional camera fallback from changing the screen encoder preference.

[kyzer1023]

@greptileai review

[kyzer1023]

@greptileai review

[kyzer1023]

@greptileai review

[richiemcilroy]

Thanks! The code fixes look in. Before we drop the AMD-to-FFmpeg routing we still need runs on real AMD hardware beyond the 780M (studio screen, studio camera, instant camera-only, pause then stop), plus a check that NVIDIA and Intel are unaffected. 1080p is also slower on wall time (0.82x) here.