Skip to content

perf(media-server): GPU/NVENC farm support, lifecycle hardening and faster verify/upload - #2935

Open
richiemcilroy wants to merge 15 commits into
mainfrom
perf/media-server-gpu-farm
Open

richiemcilroy wants to merge 15 commits into
mainfrom
perf/media-server-gpu-farm

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Gets the media server ready to run on our own AWS boxes (g6.2xlarge with an L4, or plain CPU nodes) and makes it a lot faster there. Every speed change is opt-in through env vars, so the current Railway deploy behaves the same apart from the lifecycle fixes listed below.

What changed

Opt-in (default off, no change on Railway)

  • CAP_MEDIA_VIDEO_ENCODER=nvenc: encodes with h264_nvenc on the GPU. Defaults are p1, cq = crf + 8, no lookahead, and spatial AQ, which match x264 medium on file size and VMAF across our fixtures. If NVENC fails for any reason that isn't a cancel, the job falls back to libx264. Tuning knobs: CAP_NVENC_PRESET, CAP_NVENC_CQ_OFFSET, CAP_NVENC_LOOKAHEAD, CAP_NVENC_SPATIAL_AQ, CAP_NVENC_TEMPORAL_AQ, CAP_NVENC_GPU_DECODE_CODECS.
  • HEVC and AV1 sources decode on the GPU (cuvid with resize, then scale_cuda) when NVENC is on. H.264 and VP9 still decode on the CPU, with every core when NVENC is active.
  • MEDIA_SERVER_DOWNLOAD_CONCURRENCY / MEDIA_SERVER_DOWNLOAD_PART_MB: parallel ranged source downloads.
  • MEDIA_SERVER_UPLOAD_CONCURRENCY: uploads multipart output parts in parallel (default 1, max 16).
  • MEDIA_SERVER_PROCESS_DECODE_THREADS, MEDIA_SERVER_VERIFY_DECODE_THREADS, MEDIA_SERVER_VALIDATE_DECODE_THREADS: decode thread counts.
  • MEDIA_SERVER_SHUTDOWN_GRACE_MS: on SIGTERM, stop taking new jobs, let in-flight work finish, and return 503 from /health while draining (default 0, same shutdown as today).
  • MEDIA_SERVER_PUBLIC_STATUS=false: puts job status reads behind the secret (public by default, same as today).
  • MEDIA_SERVER_IDLE_TIMEOUT_SECONDS

On by default

  • Each instance gets its own temp dir with a lease file. Dirs whose lease has gone stale (instance crashed, so no cleanup ran) are swept every 5 minutes. Before this, a kill -9 left the temp files on disk for good.
  • Long jobs send heartbeats, so a healthy job can extend its lifetime up to a 4 h hard cap. Main still kills anything at 1 h.
  • A cancelled job now kills its ffmpeg and deletes its temp files right away instead of running to the end.
  • /video/verify-recording no longer computes the SHA-256 of every decoded raw frame, because that route throws the result away. The full -xerror decode still runs, and the hash still runs whenever source evidence is passed in for comparison.
  • /health includes videoEncoder.active. Runtime counters and NVENC attempt details only appear with x-media-server-secret, and they contain no job or video IDs. URLs are stripped from NVENC fallback reasons.

Benchmarks

Median time from accept to the final webhook, with the same fixtures and S3 bucket. The GPU node is a g6.2xlarge (L4, 8 vCPU); Railway is production media.cap.so, run gently with one job at a time.

Job Railway GPU node
iPhone HEVC 4K, 5 min 215.8 s 26.0 s
Verify recording, 5 min 94.4 s 27.2 s
Mux segments, 5 min 96.7 s 35.5 s
VP9, 5 min 65.2 s 30.9 s
4K H.264, 5 min 54.1 s 37.2 s
Edit (3 cuts), 5 min 45.9 s 12.5 s
Verify recording, 1 min 14.4 s 6.1 s

20-minute recordings on the GPU node:

  • verify: 295 → 101 s
  • 4K process: 243 → 138 s
  • mux: 167 → 133 s
  • multipart upload of 1.15 GB: 11 → 2.3 s

No job failed in any run. Lifecycle was tested on the GPU node:

  • cancel mid-encode
  • SIGTERM drain finishing a 20-minute 4K job
  • kill -9 followed by the orphan sweep
  • a 24-job mixed burst across two nodes, where back-pressure returned 503s and every job still completed

Testing

  • New unit and integration tests for NVENC args and fallback, the GPU decode path, lifecycle and drain, the temp sweep, concurrent multipart upload, and skipping the verify hash.
  • The Docker build passes, including the full Dockerfile test list.
  • tsc reports the same 7 errors as main and none in touched files.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the follow-up changes address the earlier findings without introducing a new blocking issue.

Summary

Adds opt-in GPU encoding and faster transfers, plus job cleanup and shutdown improvements.

  • Video jobs can use NVENC and selected GPU decoding when enabled.
  • Media files can download and upload in parallel when configured.
  • The media server drains work and stops cancelled jobs promptly.
  • Each server instance keeps its temporary files in a leased directory.
  • Recording checks can skip hashing decoded frames when no source evidence is compared.
  • Status endpoints can hide job details and show runtime counters to secret holders.

Reviews (2) · Last reviewed commit: "fix(media-server): restore the audio str..." · Reviewed by Greptile

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 1 other finding in Capy.

Open in Capy

Comment thread apps/media-server/src/lib/media-video.ts

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

Comment thread apps/media-server/src/lib/parallel-download.ts Outdated
Comment thread apps/media-server/src/lib/parallel-download.ts
Comment thread apps/media-server/src/lib/media-video.ts
Comment thread apps/media-server/src/lib/media-video.ts
Comment thread apps/media-server/src/lib/media-transfer.ts Outdated
@vercel
vercel Bot temporarily deployed to Preview – cap-status October 22, 2026 15:05 Inactive
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Oct 22, 2026 3:23pm UTC

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

Open in Capy

Comment thread apps/media-server/src/routes/audio.ts Outdated
@vercel
vercel Bot temporarily deployed to Preview – cap-status October 22, 2026 15:23 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

This branch was successfully deployed

1 active and 1 inactive deployments
Preview – cap-web — 98c2cbd9 Deployed Oct 22, 2026 by vercel[bot]
Preview – cap-status — 98c2cbd9 Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant