AIPerf stage 9: qualify all TRTMC audio-language question-answering models #1676

Request type

New capability

Problem and use case

The inspected Qwen3-Omni path is a text-only Thinker pipeline. Transcription or a multimodal model name does not prove audio-conditioned question answering.

Proposed outcome

Qualify all supported models in this group for one audio-language understanding workload through the existing native serving entrypoint.

• Workload: One prerecorded audio clip plus a question produces an answer, using fixed audio-QA pairs.
• Client / endpoint: AIPerf chat; POST /v1/chat/completions with audio content.
• Rollout / prerequisites: Model-group rollout after stage 8. Only an existing supported native audio-QA capability is eligible; model/capability onboarding is outside this epic.

Existing-model scope — owner clarification

This epic integrates AIPerf with models and model specs already supported on upstream main. “Whole group” means every existing model/spec for this workload, rather than adding checkpoints that an architecture might also support.

• Reuse existing checkpoint identities, normal full-model profiles/manifests, build settings, references and correctness thresholds. Do not add model families, checkpoints, model specs/manifests, precision/parallel variants or new numerical gates to enlarge the group.
• HTTP/media adapters, AIPerf input/workload configuration and regression fixes needed for existing models are in scope. Keep model semantics family-owned and existing correctness criteria unchanged.
• Missing full-model implementations or native audio/video capabilities are recorded as unsupported/outside this integration. Do not implement a new encoder, model graph, temporal path or native Task to fill the inventory. L0/partial profiles do not require full-model onboarding in this epic.
• Refresh against current main only to include independently existing supported specs. Architectural compatibility, model cards and upstream model availability do not establish an in-scope TRTMC model.

Group-wide scope and completion (revised)

Cover every TRTMC-supported checkpoint in this model group for the workload below, not just one pilot. Refresh the inventory against github/main at implementation and before completion. Use family-owned manifests, actual build/runtime behavior, and public Task capabilities together; a model name, broad support declaration, or an inherited C++ interface alone does not establish the checkpoint's supported workload.

Use one normal full-model profile per distinct checkpoint and workload. Record all additional precision/quantization/TP/CP variants separately; do not count variants as extra models or claim untested configurations. L0/partial/random/regression smoke profiles do not establish full-model coverage. Checkpoints with only partial implementation remain explicit gaps, not silent omissions. Only independently existing supported specs on main join the refreshed inventory; do not add models or specs as part of this integration.

• Publish the exhaustive per-checkpoint inventory with checkpoint/profile/revision, native Task, supported controls, and any explicit capability gap; reconcile it with current main before closing.
• Every supported checkpoint passes its unchanged owning-family correctness criteria on the exact served bundle, followed by a real published pip-installed AIPerf run through trtmc-server for this one workload.
• Check meaningful complete output contents and native/HTTP identity for every model, with separately marked warmup, failures, client/Task timing, and workload-appropriate throughput.
• Publish a result row and reproducible command for every checkpoint. A pilot, source-only check, HTTP fixture, skipped test, unrun model, unsupported adapter, or unresolved failure cannot close this group-wide task.
• Preserve family ownership and existing quality criteria; fix real defects without introducing model behavior in shared serving code or relaxing gates.

Implementation scope:

1. Audit existing specs and real native audio-plus-question capabilities. Reuse an already-supported model/spec if one exists; otherwise report the stage as blocked by absent existing capability. Do not add models, specs, audio encoders, orchestration or native Task contracts.
2. Establish family correctness on actual audio-conditioned results before adding native-server transport decoding and the AIPerf run.
3. Use fixed audio-question-answer pairs that exercise audio understanding; record audio properties and processing configuration.

Acceptance criteria for every model in the group:

• The issue records the chosen model, profile, checkpoint, precision, and actual native audio capability; conditional support is not reported as already implemented.
• Audio-conditioned answers satisfy the owning family's correctness/quality contract and demonstrate the selected QA behavior rather than transcription alone.
• A real pip-installed AIPerf run sends audio plus questions through the native endpoint and returns correlated answers with visible errors and separate client/Task timing.
• Reuse existing serving/qualification mechanics; keep all model semantics, orchestration, and correctness family-owned, with capability-based Task dispatch.
• Account for inputs, outputs/artifacts, failures, and warmup by request identity. Keep client end-to-end latency separate from public_task_call_wall; exclude build/load/startup/warmup from steady-state timing.
• Record tested package versions, checkpoint/profile, actual bundle, dataset, endpoint/mode, hardware, effective precision, settings, commands, and public evidence. Distinguish planned, implemented, family-validated, and AIPerf-qualified status.
• Provide one concise reproducible example using published pip-installed AIPerf and links to existing build guidance; an AIPerf source checkout is not a runtime dependency.

Model and target details

Inventory baseline: TRTMC fd79ad26ca71a11b3ec086a7daaaead03e6c933f. Candidate models/profiles for this group:

No existing full-model audio-question-answering profile was found in this baseline. Audit all native audio-conditioned models (including Qwen3-Omni, Phi-4 multimodal and speech-session families) for an actual audio-plus-question -> text-answer contract. Record unsupported input paths explicitly. If no existing supported spec provides this behavior, keep a visible unsupported/blocked status and report the gap. Implementing an audio-QA model, adding a spec or extending model capability requires separate owner-authorized work. A transcription-only or text-only result is not audio-QA coverage.

Use the public native Task runtime path. Record the actual supported target GPU and software stack during implementation/qualification; no new hardware validation or benchmark is claimed by this planning issue. Keep existing numerical criteria unchanged.

Alternatives considered

Reuse apps/perf_serving protocol/media/recording mechanics and apps/aiperf_qual qualification instead of adding a third server or benchmark harness. Deliver this stage on its own short-lived branch from current github/main.

Follow-ups outside this initial stage: Streaming audio QA requires genuine native incremental support and its own qualification.

Additional context

• Parent epic: Epic: expand TRTMC AIPerf model and workload coverage #1667
• AIPerf docs/tutorials/audio.md
• Native server
• Existing qualification harness

Research baseline: TRTMC 0bccb5e93af0e5ad346aaf44e404396f54b2e7de and AIPerf e1daadd10faa395100384358131d784d6858c5ae; published AIPerf 0.13.0. Recheck the tested published version and native capabilities when implementing. Existing source/fixtures/reference-only runs do not establish end-to-end TRTMC qualification. Keep internal logs, artifacts, and runner details private.

Submission checks

• I searched open and closed issues and found no duplicate request.
• I removed secrets, private/internal evidence, personal paths, and restricted artifacts.

Group completion policy revised at the owner's request: all supported checkpoints in this group must be accounted for and qualified; one successful pilot is insufficient. Existing pilot results are retained as historical evidence.