AIPerf stage 4: qualify all TRTMC single-image object detection models #1671

Stage 4 delivered — PR #1753 merged as 9643800b3b5a951382c87ae1ebfbbeb24b1a8090. All ten catalogued detector checkpoints are now qualified for this workload. The other nine retain their original results and tested heads in PR #1718.

Deformable DETR's exact FP16/TP1 bundle passed its unchanged owning checkpoint case, the dynamic geometry regression, 100 original-image native/HTTP comparisons, existing corpus gates and published AIPerf 0.13.0: 50 measured requests plus five warmups, 463 complete boxes, zero errors or missing joins. On the 100-image selection, native/reference mAP@[.50:.95] is 46.8307% / 46.8189% and mAP@.50 is 62.8337% / 62.6492%; both maximum-drop gates remain 0.02. YOLOS retains its original owning reference contract and separately reported corpus metrics because it has no existing dataset gate.

Measured on one shared GB300 at concurrency 1; these are short integration measurements, not capacity or performance gates. The PR contains all ten result rows, checkpoint/bundle identities, original measurement attribution, environment and reproducible commands. Usage guide. Rebuild older Deformable DETR bundles for the corrected aspect-preserving resize and dynamic TensorRT plan. Existing numerical criteria, public Task API and ABI are unchanged.

All expected checks passed on tested head cf93a041d8c0660f2e3e3ad0a4e4008b70cfd26c, including Stable/Dev Community CPU, the selected real GPU case, confirmed GPU deletion/independent cleanup, automated review and the protected internal premerge gate. Each CPU lane reports 1,550 Python passes, two existing skips and 262 native passes. There are no unresolved review conversations. The benchmark container is stopped and generated evidence is preserved outside source commits.

Request type

New capability

Problem and use case

AIPerf's image_retrieval endpoint is a bounding-box detection interface. TRTMC needs the matching adapter backed by real native detections to qualify this group.

Proposed outcome

Qualify all supported models in this group for one object detection workload through the existing native serving entrypoint.

• Workload: One image per request produces labelled bounding boxes.
• Client / endpoint: AIPerf image_retrieval; NIM-shaped POST /v1/infer.
• Rollout / prerequisites: Model-group rollout after stage 3.

Existing-model scope — owner clarification

This epic integrates AIPerf with models and model specs already supported on upstream main. “Whole group” means every existing model/spec for this workload, rather than adding checkpoints that an architecture might also support.

• Reuse existing checkpoint identities, normal full-model profiles/manifests, build settings, references and correctness thresholds. Do not add model families, checkpoints, model specs/manifests, precision/parallel variants or new numerical gates to enlarge the group.
• HTTP/media adapters, AIPerf input/workload configuration and regression fixes needed for existing models are in scope. Keep model semantics family-owned and existing correctness criteria unchanged.
• Missing full-model implementations or native audio/video capabilities are recorded as unsupported/outside this integration. Do not implement a new encoder, model graph, temporal path or native Task to fill the inventory. L0/partial profiles do not require full-model onboarding in this epic.
• Refresh against current main only to include independently existing supported specs. Architectural compatibility, model cards and upstream model availability do not establish an in-scope TRTMC model.

Group-wide scope and completion (revised)

Cover every TRTMC-supported checkpoint in this model group for the workload below, not just one pilot. Refresh the inventory against github/main at implementation and before completion. Use family-owned manifests, actual build/runtime behavior, and public Task capabilities together; a model name, broad support declaration, or an inherited C++ interface alone does not establish the checkpoint's supported workload.

Use one normal full-model profile per distinct checkpoint and workload. Record all additional precision/quantization/TP/CP variants separately; do not count variants as extra models or claim untested configurations. L0/partial/random/regression smoke profiles do not establish full-model coverage. Checkpoints with only partial implementation remain explicit gaps, not silent omissions. Only independently existing supported specs on main join the refreshed inventory; do not add models or specs as part of this integration.

• Publish the exhaustive per-checkpoint inventory with checkpoint/profile/revision, native Task, supported controls, and any explicit capability gap; reconcile it with current main before closing.
• Every supported checkpoint passes its unchanged owning-family correctness criteria on the exact served bundle, followed by a real published pip-installed AIPerf run through trtmc-server for this one workload.
• Check meaningful complete output contents and native/HTTP identity for every model, with separately marked warmup, failures, client/Task timing, and workload-appropriate throughput.
• Publish a result row and reproducible command for every checkpoint. A pilot, source-only check, HTTP fixture, skipped test, unrun model, unsupported adapter, or unresolved failure cannot close this group-wide task.
• Preserve family ownership and existing quality criteria; fix real defects without introducing model behavior in shared serving code or relaxing gates.

Implementation scope:

1. Add the NIM wire-format adapter using public detection Task outputs and generic image transport/recording mechanics.
2. Preserve correct coordinates and normalization, confidence values, and family-owned labels.
3. Reuse the owning family's detection correctness and dataset qualification rather than moving model semantics into serving code.

Acceptance criteria for every model in the group:

• Bounding-box values, labels, and coordinates satisfy the unchanged family correctness/mAP criteria on the selected images and bundle.
• A real pip-installed AIPerf run exercises /v1/infer and verifies meaningful detection contents with input/output correlation.
• Report image latency and images/sec separately from Task-call time; record image sizes and processing configuration.
• The coverage matrix accurately names object detection and includes every document/table-layout checkpoint that actually implements the matching Task, with its own labels and quality contract. Unrelated document-understanding Tasks are separate workloads.
• Reuse existing serving/qualification mechanics; keep all model semantics, orchestration, and correctness family-owned, with capability-based Task dispatch.
• Account for inputs, outputs/artifacts, failures, and warmup by request identity. Keep client end-to-end latency separate from public_task_call_wall; exclude build/load/startup/warmup from steady-state timing.
• Record tested package versions, checkpoint/profile, actual bundle, dataset, endpoint/mode, hardware, effective precision, settings, commands, and public evidence. Distinguish planned, implemented, family-validated, and AIPerf-qualified status.
• Provide one concise reproducible example using published pip-installed AIPerf and links to existing build guidance; an AIPerf source checkout is not a runtime dependency.

Model and target details

Inventory reconciled at merged main 9643800b3b5a951382c87ae1ebfbbeb24b1a8090. All ten canonical checkpoints are qualified for single-image object detection: nine in merged PR #1718 and Deformable DETR in merged PR #1753. Original measurement heads and bundle identities are preserved; earlier models were not remeasured on the new head. Original links below retain their inventory provenance:

Preserve each family's real label space, scores, coordinates, preprocessing and suppression. Different label numbering is recorded, not overwritten to look like YOLO. Document/table-layout models join this group only where an actual matching native Task exists; do not silently claim that bounding boxes qualify unrelated page semantics.

Use the public native Task runtime path. Record the actual supported target GPU and software stack during implementation/qualification; completed hardware validation and benchmark evidence are recorded in the merged PRs linked above. Keep existing numerical criteria unchanged.

Alternatives considered

Reuse apps/perf_serving protocol/media/recording mechanics and apps/aiperf_qual qualification instead of adding a third server or benchmark harness. Deliver this stage on its own short-lived branch from current github/main.

Include document-layout checkpoints in this inventory when they expose the matching object-detection Task, and preserve their family-owned label and quality contracts. Other document-understanding Tasks remain separate workloads.

Additional context

• Parent epic: Epic: expand TRTMC AIPerf model and workload coverage #1667
• Pilot profile
• AIPerf docs/tutorials/nim-image-retrieval.md
• Native server
• Existing qualification harness

Research baseline: TRTMC 0bccb5e93af0e5ad346aaf44e404396f54b2e7de and AIPerf e1daadd10faa395100384358131d784d6858c5ae; published AIPerf 0.13.0. Recheck the tested published version and native capabilities when implementing. Existing source/fixtures/reference-only runs do not establish end-to-end TRTMC qualification. Keep internal logs, artifacts, and runner details private.

Submission checks

• I searched open and closed issues and found no duplicate request.
• I removed secrets, private/internal evidence, personal paths, and restricted artifacts.

Historical pilot delivery (revised group scope remains open)

Delivered to main by PR #1713; merge commit 90afeece5c210c7c12ed9f19b061c40318862986. All expected checks, including Stable/Dev Community validation and cleanup plus the protected internal premerge gate, passed on tested head 57a992a38ad7e9173bbccf135a2ebe4349f3f4fc, based on 0f341f1cb60ff40c0dd11bf903e4c03d64ab1534. Both review conversations are resolved. This issue is closed by the merged PR.

The actual YOLO11n FP16 bundle passes the unchanged owning-family checkpoint/reference case and 100-image COCO gates: native/reference mAP@[.50:.95] 39.4663%/39.5096%; mAP@.50 50.7374%/50.7170%; existing maximum drops remain 0.02/0.02. All 100 direct HTTP/native verification calls and all 55 published AIPerf responses preserve complete boxes, numeric class IDs, scores and normalized coordinates; 325 boxes were checked in the client responses, with zero errors or missing metric joins.

The pip-installed AIPerf 0.13.0 run has 50 measured requests and five warmups, concurrency 1: HTTP p50/p95 15.83/20.13 ms, public Task p50/p95 7.52/9.29 ms, 60.17 images/sec. Build/load/startup, verification and warmup are excluded. Native serving tests: 7 passed; Python serving/transport: 178 passed without skips; source quality: 307 passed; website inventory/production build passed. Dataset/checkpoint/package/bundle provenance, image dimensions, commands, and the three-step usage guide are in the PR. Other detectors, document/table layout, other input modes and capacity sweeps remain unqualified.

Both image-preparation review findings are fixed and resolved at the current head: blocking image inspection/file writes run off the event loop, pixel limits precede full decoding, damaged pixel streams fail at upload, and cancellation regressions verify safe cleanup. The final accuracy and pip-client runs were repeated on this clean head and its rebuilt bundle.

Group completion policy revised at the owner's request: all supported checkpoints in this group must be accounted for and qualified; one successful pilot is insufficient. Existing pilot results are retained as historical evidence.

Group qualification update — PR #1718

Nine original detectors have completed selected owning-family correctness checks, complete native/HTTP corpus verification, and 55 responses each from pip-installed AIPerf 0.13.0 (five warmups, 50 measured requests, concurrency 1). RT-DETR's benchmark label metadata is corrected in 68e30ad9e238d8fa2bfa3bc381de8d8832827f0a from sparse COCO category IDs to the checkpoint's actual contiguous 80-class index space. The same 100 saved native/reference outputs now give meaningful mAP@[.50:.95] 53.6812%/53.7066% and pass both unchanged two-percentage-point drop gates. The original zero/zero report is retained; no model output or numerical threshold changed. YOLOS corpus metrics are reporting-only because its family has no existing dataset gate.

Main subsequently added Deformable DETR. It has not been qualified in this campaign; this issue remains open and is not closed by PR #1718. The benchmark container is stopped and results preserved at the owner's request. Current-head CI and merge are also pending.