L07 reports 306 entries in the content repo whose provenance comment names gpt-5-6-mini as the model that wrote them. All three routes in the route file request gpt-5. Nothing in the pipeline asks for a mini model at any point.
How it surfaced
L07 checks that the model named in a provenance comment is one of the models a route was configured to ask for. It was written as a plumbing check, on the theory that a mismatch would mean a threading bug in answered_by. It found a real one instead, and incidentally proves the threading works: the model name reaching the memory is the name the server reported, not the name we sent.
What it means
Three possibilities and they need to be told apart before anything is re-run.
- The server silently substitutes a cheaper model under load. If so,
gpt-5 in the route file is a request rather than a guarantee, and every quality measurement taken so far has an unknown fraction of mini in it.
- One route is misconfigured on the server side and always answers as mini.
- The name is reported inconsistently by the endpoint and the entries are
gpt-5 after all.
What to do
Found by pydocvi audit --only L07.
L07reports 306 entries in the content repo whose provenance comment namesgpt-5-6-minias the model that wrote them. All three routes in the route file requestgpt-5. Nothing in the pipeline asks for a mini model at any point.How it surfaced
L07checks that the model named in a provenance comment is one of the models a route was configured to ask for. It was written as a plumbing check, on the theory that a mismatch would mean a threading bug inanswered_by. It found a real one instead, and incidentally proves the threading works: the model name reaching the memory is the name the server reported, not the name we sent.What it means
Three possibilities and they need to be told apart before anything is re-run.
gpt-5in the route file is a request rather than a guarantee, and every quality measurement taken so far has an unknown fraction of mini in it.gpt-5after all.What to do
doctorshould fail when the answering model is not the requested one, rather than only checking that a completion comes back at all.Found by
pydocvi audit --only L07.