Policy for models absent from model_cost.json, and for free/promotional endpoints

[ypollak2]

Hi RouterArena maintainers,

Following #166, a question about the other side of model identity: models that are absent from model_cost/model_cost.json entirely, and specifically how promotional or preview endpoints should be handled.

What happens today

A prediction naming a model with no price entry is not scored as an expensive route — it is dropped before evaluation. Running the harness locally on 5,273 full-split rows generated by stealth/ox-alpha:

ERROR - Error converting model name 'stealth/ox-alpha' to universal name:
        Model name stealth/ox-alpha not found in universal_names or mapping
INFO  - Total: 5273 | Evaluated: 0 | Skipped: 5273 | Failed: 0
ERROR - Fatal error: No entries have valid cost values.
        Cannot compute RouterArena score without cost data.

Every row is skipped, then compute_router_metrics refuses to score. In a submission that routes to a mix of priced and unpriced models, those queries are effectively graded as failed inference. That is a defensible default, and we are not reporting it as a bug — the behaviour is safe. The question is what the intended path is for adding a model.

The concrete case, with the awkward part stated up front

stealth/ox-alpha is currently served on OpenRouter at $0 / 1M tokens in and out (1M context). We measured it on the full split under the standard zero-shot \boxed{} contract:

	stealth/ox-alpha	gemini-3-flash-preview
Accuracy (full split)	78.18%	78.44%
Accuracy, external corpus (2,600 items)	78.85%*	78.85%*
Mean output tokens	256	97
Empty responses	3.9%	0%

_{*measured on an identical 52-item subsample; the 2,600-item run is still in progress at the time of writing.}

So it is a genuinely capable model. But a $0 price entry would let any submission that routes to it saturate the cost term, and that advantage would be an artifact of a temporary promotion rather than of routing quality. Given #166 was precisely about a submission benefiting from a price that no longer reflected what was really being served, we would rather raise this than quietly exploit it — and we are not asking for a $0 entry.

What we are asking

1. Is there a documented process for requesting a model be added to model_cost.json? A3M Router submission - what is the official listing process? #138 asks a related question about listing routers; we could not find the equivalent for models.

2. What is the intended policy for free / promotional / preview endpoints? Options we can see, with no preference beyond wanting the rule to be stated:

   • exclude them until they carry a standing commercial price;
   • admit them at a reference price for their capability tier rather than at $0, so cost reflects the compute rather than the promotion;
   • admit them at $0 but flag such entries on the leaderboard as promotionally priced;
   • admit them only once a price is published, and re-grade if it changes.

3. Should the checker warn on unpriced model names at submission time? Right now the failure surfaces only at evaluation, as a skip and then a fatal error, which is easy to misread as a broken artifact. Fix Grok 4.1 Fast → Grok 4.3 mispricing on the leaderboard (#166) #172 added a preflight warning for retired/redirected slugs — the unpriced case seems like a natural neighbour to it.

Happy to open a PR for either the checker warning or a documentation note, whichever is more useful. And happy to be told the answer is simply "not eligible" — that is a perfectly good answer and we will design around it.

Thanks for the work on the benchmark.