Skip to content

Fix Mistral SUT model ID formatting - #1661

Open
kaulastudies wants to merge 1 commit into
mlcommons:mainfrom
kaulastudies:fix/mistral-model-id-1659
Open

kaulastudies wants to merge 1 commit into
mlcommons:mainfrom
kaulastudies:fix/mistral-model-id-1659

Conversation

@kaulastudies

Copy link
Copy Markdown

Summary

Fixes Mistral SUT model ID formatting by separating ModelBench's canonical SUT identity from the model ID sent to Mistral's API.

Dynamic Mistral definitions keep the canonical ModelBench UID:

mistral/<model>:mistral

but MistralSUTFactory.make_sut() now sends only the model component (<model>) to Mistral's model_info() lookup and to MistralAISut.

This prevents native Mistral model IDs such as mistral-medium-3 from being rewritten to mistral/mistral-medium-3, which returns 404 from the Mistral API.

Tests

The regression test verifies both sides explicitly:

  • ModelBench UID still preserves the maker: foo/bar:mistral
  • Mistral API/runtime model ID is only bar
  • model_info() is called with bar

Validation

  • Mistral SUT factory tests: 3 passed
  • Related SUT factory tests: 13 passed
  • Black: passed
  • MyPy: passed
  • git diff --check: passed

Closes #1659

@kaulastudies
kaulastudies requested a review from a team as a code owner September 15, 2026 05:50
@kaulastudies
kaulastudies deployed to Scheduled Testing September 15, 2026 05:51 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix mistral SUT model id formatting

1 participant