Skip to content

306 committed entries were written by a model nobody asked for #37

Description

@tamnd

L07 reports 306 entries in the content repo whose provenance comment names gpt-5-6-mini as the model that wrote them. All three routes in the route file request gpt-5. Nothing in the pipeline asks for a mini model at any point.

How it surfaced

L07 checks that the model named in a provenance comment is one of the models a route was configured to ask for. It was written as a plumbing check, on the theory that a mismatch would mean a threading bug in answered_by. It found a real one instead, and incidentally proves the threading works: the model name reaching the memory is the name the server reported, not the name we sent.

What it means

Three possibilities and they need to be told apart before anything is re-run.

  1. The server silently substitutes a cheaper model under load. If so, gpt-5 in the route file is a request rather than a guarantee, and every quality measurement taken so far has an unknown fraction of mini in it.
  2. One route is misconfigured on the server side and always answers as mini.
  3. The name is reported inconsistently by the endpoint and the entries are gpt-5 after all.

What to do

  • Group the 306 by route and by run id. If they are one route, it is configuration. If they are spread across all three and clustered in time, it is substitution under load.
  • Ask one route directly for its model name outside a batch and compare.
  • Decide whether doctor should fail when the answering model is not the requested one, rather than only checking that a completion comes back at all.
  • Re-translate the 306 once the cause is known, since they cost nothing to redo and their quality is unmeasured.

Found by pydocvi audit --only L07.

Metadata

Metadata

Assignees

No one assigned

    Labels

    auditThe deterministic check cataloguebugSomething is wrongfleetSpends model calls through the fleet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions