[Addin]

Summary

This branch reduces per-frame CPU/GPU work in the camera capture path and fixes capture/format issues found while profiling.

Windows / Media Foundation

• Wrap MF buffers in a single refcounted AVBufferRef with stack plane metadata instead of one Vec/Arc per plane.
• Replace the per-frame Arc<Mutex<IMFMediaBuffer>> with a direct COM-backed CameraBuffer; every consumer uses a consistent 2D/contiguous lock policy.
• Cache the D3D11 upload texture and immediate context, and use UpdateSubresource instead of creating a texture per frame.
• Upload directly into the hardware encoder's video-processor input texture, skipping the extra CopyResource; VideoProcessorBlt still performs conversion/scaling.
• Fix DirectShow/Media Foundation capture issues: shared media types across the callback worker, MTA initialization, bounded callback queues, drop counters, and NV21 to NV12 conversion.
• Infer contiguous camera-buffer strides from the buffer length so padded DirectShow/MF rows keep the correct chroma offset.
• Restore the non-Windows CameraBuffer import gate.

macOS / VideoToolbox

• Reuse an IOSurface-backed CVPixelBufferPool for VideoToolbox output.
• Revert the camera-ffmpeg zero-copy wrapper so preview/FFmpeg consumers copy instead of holding CVPixelBuffer locks for the AVFrame lifetime.
• Fix the VideoToolbox pool attribute compile error by using cv::PixelFormat::to_cf_number() for the pool attributes.
• Remove the unused CVPixelBufferRef alias so macOS Clippy stays warning-free.

Benchmarks

Added frame-copy-bench and camera-upload-bench.

1080p NV12, release, median of 3 runs:

Benchmark	Before	After	Improvement	Speedup
Zero-copy frame wrap	0.002 ms / ~559k fps	0.001 ms / 936k fps	67.4% higher throughput	1.67×
Preview copy	1.015 ms / 985 fps	0.881 ms / 1,136 fps	13.2% lower latency	1.15×
Row copy	0.980 ms / 1,020 fps	0.878 ms / 1,139 fps	10.4% lower latency	1.12×
Buffer clone	0.000 ms	0.000 ms	0.0%	1.00×
Bytes copy	2.074 ms	1.862 ms	10.2% lower latency	1.11×
Cached upload	0.733 ms / 1,364 fps	0.825 ms / 1,212 fps	−12.6% (run variance)	0.89×
Uncached upload	0.956 ms / 1,046 fps	0.948 ms / 1,055 fps	0.8% lower latency	1.01×
Direct encoder-input upload	1.053 ms / 949 fps	0.884 ms / 1,131 fps	16.0% lower latency	1.19×
Cached upload + CopyResource (old two-step)	1.359 ms / 736 fps	1.205 ms / 830 fps	11.3% lower latency	1.13×

Direct encoder-input upload versus the old two-step path: 1.359 → 0.884 ms/frame, 35.0% lower latency, 1.54× speedup. Upload timings vary by roughly ±20% run-to-run on this host; the table uses medians, with the best cached upload at 0.700 ms and best direct upload at 0.818 ms.

Test plan

• cargo clippy --workspace --exclude cap-desktop --all-features --locked -- -D warnings (Windows/MSVC) ΓÇö passed.
• cargo fmt --all --check ΓÇö passed.
• CI-equivalent Windows test set: cap-project, cap-timestamp, cap-enc-ffmpeg, cap-camera-effects, cap-recording lib (724 passed), cap-rendering, cap-audio, cap-export, cap-editor, cap-render-farm, cap-camera-ffmpeg (5 passed, including padded NV12 stride), cap-enc-mediafoundation, cap-utils (79 passed), cap CLI record tests (13 passed), scap-targets, and WASAPI loopback.
• scripts/test-recording-reliability.py and scripts/test-capbench.py ΓÇö passed (18 and 17 tests, with expected skips).
• Instant-mode DASH scenarios: 61/61 pass with CAP_EXPORT_FORCE_SOFTWARE_ENCODER=1; the local NVIDIA NVENC path fails 3 segment-count tests, which is hardware-encoder-specific and not caused by this branch.

Notes

• A full macOS build was not possible on the Windows development host. The non-Windows CameraBuffer import is now cfg(windows)-gated; macOS CI will validate the build.
• The stricter local --all-targets clippy (not the CI clippy command) still reports two pre-existing test-only lints in cap-export and cap-recording; neither is touched by this branch.

Confidence Score: 5/5

The reviewed changes appear safe to merge; no new actionable issue was established.

Summary

This PR reduces camera-frame copies and reuses upload resources.

• Windows recording keeps camera frames in their original buffers for conversion.
• Windows camera callbacks hand samples to bounded worker queues.
• Windows camera recording reuses upload and encoder textures.
• VideoToolbox conversion reuses output pixel buffers on macOS.

_{Reviews (5) · Last reviewed commit: "fix(camera): remove unused VideoToolbox ..." · Reviewed by Greptile}

[vercel]

@Addin is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

[Addin]

Addressed the two current-head Greptile findings:

• frame-converter: VideoToolbox pool attributes now use format.to_cf_number().as_ref().
• camera-ffmpeg: contiguous camera buffers infer their stride from the buffer length, preserving padded DirectShow/MF row layouts; added a padded NV12 regression test (cap-camera-ffmpeg now 5 tests).

The other three comments were left on the pre-rebase stacked head and reference the Windows readback commits (win_fragmented_m4s MF path and scap-ffmpeg mapped frames), which are not part of this camera-only PR. Those belong to #2940.

[Addin]

@greptileai review

[Addin]

@greptileai review

[Addin]

Hi @richiemcilroy can I have a review on this, or should this be closed?

[richiemcilroy]

Thanks! Can you resolve the conflicts with main? Then we'll take a look.

[Addin]

@greptileai review

[Addin]

@richiemcilroy This is ready, review is 5/5.

[richiemcilroy]

Thanks! Can you rebase on main to clear the conflicts? Then we'll take a look.

[Addin]

Rebased onto current main (3ca736247) and force-pushed.

Conflict notes: combined main's MJPEG scaled-IDCT work with the frame-conversion/stats refactor in camera-ffmpeg, and kept main's encoder-fallback reporting inside the camera-encode closure. Local cargo fmt --all -- --check and Windows checks for cap-recording (lib + examples), cap-camera-ffmpeg --all-targets, and cap-verify --tests pass. macOS/VideoToolbox still needs CI validation.

@greptileai review
@richiemcilroy could you take a look when you have a moment?

[Addin]

The GPUI Desktop build jobs were failing because the separate apps/desktop-gpui/Cargo.lock was stale after the rebase (cannot update the lock file ... because --locked was passed). I regenerated that lockfile for the new path-crate dependencies and pushed it.

Verified with cargo metadata --manifest-path apps/desktop-gpui/Cargo.toml --locked. The new head is a269deea4.