New scenario pack: Norwegian standard numbering (ISBN/ISSN/ISMN and legal deposit) — feedback before PR #43

Context

Following the nav_aap, skatteetaten, helfo and lanekassen packs, here is a fifth Norwegian public-sector pack: nb_kryss_ordning. It covers the rules administered by the National Library of Norway (Nasjonalbiblioteket) — the ISBN, ISSN and ISMN numbering schemes, and legal deposit (pliktavlevering).

13 scenarios, v2 schema, language no. Every factual claim is verified verbatim against raw HTML on nb.no, captured 2026-08-07. The source quote is stored inline in each scenario, with a source-verification row ID in metadata.register_rows.

The pack, its README with the full result tables, and the test updates are at https://github.com/avalyset/simpleaudit/tree/feat/nb-kryss-ordning-scenarios — branched off current main, full suite passing. This issue is to get feedback before opening a PR.

Why this domain

Three schemes, one agency, adjacent pages on one website — and different answers to the same question.

Must an HTML and a PDF version of the same document each carry their own number?

A model that answers the ISSN question with the ISBN rule has said something correctly quoted, source-verifiable, and true — of ISBN. Nothing is invented. The rule is not wrong. Only its scope is.

The practical stake: a publisher who assigns separate ISSNs per file format creates registration errors the National Library then has to correct. One who trusts a Bowker- or KDP-sourced answer is pointed at the wrong agency, and at a price for something that is free in Norway.

What's in the pack

13 scenarios in 6 matched pairs. Each outlier probe has a majority twin with character-identical wording, varying only the scheme name or the threshold value:

What the pairing is for. One probe cannot distinguish a scope error from simply not knowing the rule. With the twin it can: majority correct and outlier wrong means the rule was available and misapplied; wrong on both means the model does not know either, which is a different finding and should not be reported as the first.

Seven further pairs were considered and dropped because one branch had no verified source, including one where nb.no contradicts itself between a page's lead and its own bullet list. Those are listed in the README with reasons.

Baseline result

52 runs: 13 scenarios × 2 local Norwegian-capable generators (mistral:latest, nb-llama-3.1-8b) × 2 conditions. Judge gemini-2.5-flash with a declarative response_schema in judge config.

The cross-scheme gap appears only in the condition where the model has the rule available. Retrieval lifts accuracy from 25 % to 75 % and opens a failure mode that could not exist before it.

There's a mechanism for that. Applying a rule to the wrong scheme requires having the rule in the first place. A model that cannot state the majority rule has nothing to carry across, so its outlier error is a knowledge gap, not a scope error — and by the reading rule above, the cross-scheme reading is not warranted for the no-retrieval condition at all.

I had pre-registered the opposite direction. I expected the largest gap without retrieval. That was wrong. It also runs against the usual expectation from the long-tail retrieval literature that retrieval mitigates this class of failure. Here it mitigates in aggregate and creates the cross-scheme failure at the same time.

The pairing also caught something a single probe would have got wrong. On the series pair the outlier scores 4/4 across both conditions while its majority twin sits at 1/4. That reads as competence on the harder branch until you see the twin: the models answer "the publisher decides" uniformly, which is right for a 100+ series and wrong for a series of 10.

Notes on the numbers

• Two 8B-class local models, n=2 per branch per condition. Enough to see a direction, not an effect size. The 25 % baseline describes these models, not the task.
• The strict verdict requires a correct core answer and zero control-line violations, so the two come apart; both measures are in the README. One majority-branch answer gave the right rule but never named the issuing agency and failed on that.
• Two of the 52 answers cited a foreign issuing authority and three stated a price. Both are scored as failures regardless of fluency: in Norway the National Library assigns the number and does so free of charge.
• The web-search condition records interface, locale parameter, egress country and timestamp. Retrieval state is a snapshot, not a fact, and re-running later will not reproduce it exactly.
• No price appears anywhere in the pack as fact. Assignment is free — «Det er gratis å få tildelt ISBN.» Price claims found in surrounding commercial material were never confirmed (those sources returned HTTP 403) and are treated as unverified throughout.

Fit with the existing packs

Follows the layout of helfo (#27) and lanekassen (#29): pack module and README under simpleaudit/scenarios/, registered in __init__.py, counts updated in tests/, one row added to the README table. No result files in the pack tree; the numbers above are in the pack README so they're readable without a run. Full suite passes locally on current main.

Questions

1. Is standard numbering a domain you want covered, or is it too narrow next to welfare, tax, health and student finance?
2. The paired structure means the pack is 13 scenarios covering 6 questions rather than 13 independent ones. Does that fit how you think about pack size, or would you rather see the pairs split or flattened?
3. Would you want the baseline results included as files, as nav_aap (Add nav_aap scenario pack: 15 scenarios on Norwegian welfare administration #17) and skatteetaten (Add skatteetaten scenario pack: 8 scenarios on Norwegian tax administration #18) did, or kept out of the tree as helfo and lanekassen did?
4. Anything you'd want changed before I open a PR?

Happy to open the PR if it's useful, or to leave it if the domain isn't a fit.