During COVID-19, SPI-M-O, the modelling subgroup of the UK's Scientific Advisory Group for Emergencies (SAGE), wrote its consensus statements with verbal probability terms such as "likely", "highly likely" and "realistic possibility". The terms come from the probability yardstick of the Professional Head of Intelligence Assessment (PHIA), which gives each one a numeric range: "likely", for example, means 55–75%. This study asks whether the statements came true as often as their terms said. When SPI-M-O wrote "likely", was it right about 55–75% of the time?
Each statement is stripped of its probability term and shown to independent reviewers as a bare proposition. They judge, with hindsight, whether it turned out correct, without seeing the term. Grouping their verdicts by the withheld term gives the calibration.
The survey is ready to deploy. It has 309 items: 291 statements from February 2020 to February 2022 and 18 hidden quality-control items.
NOTEBOOK.md has a dated entry for each update to a line of inquiry, with the intent, what was decided or changed, and what remains uncertain. Read it before the issues and pull requests behind each entry, which have the full detail. AGENTS.md sets out the conventions for the notebook, issues and pull requests.
scripts/ pipeline code (run from repo root)
data/answer_key/ sealed answer-key checksums (the only tracked data)
NOTEBOOK.md dated entry per update to a line of inquiry; start here
AGENTS.md conventions for issues, pull requests and the notebook
ANALYSIS_PLAN.md pre-specified analysis plan (work in progress)
README.md
LICENSE
The source documents, generated propositions, survey file and audit tables are not in the repository. Everything under data/ apart from data/answer_key/CHECKSUMS.txt and data/answer_key/README.md is git-ignored: most of it is regenerated by the pipeline, and some is withheld until coding is complete (see Sharing and embargo).
data/manifest.json holds the provenance record; git add -f it if you want it tracked.
The scripts in scripts/ are run from the repository root and read and write paths under data/. In rough order:
list_statements.pylists the SPI-M-O consensus statements and their attachments through the gov.uk search and content APIs, intodata/consensus_with_attachments.json. It uses the search API because the monthly SAGE collections miss about 12% of the statements, all of August 2021 among them.ANALYSIS_PLAN.mdsets out the scope.download.pydownloads the PDFs of the statements (rate-limited, with an identifying User-Agent) and extracts their text withpdftotext -layout(poppler 26.01; other versions can place text differently). It writesdata/pdfs/,data/txt/anddata/manifest.json, the provenance record with SHA-256 checksums.scan.py,claims.py,classify.py,tense.py,make_frame.pyanddedupe_frame.pyextract candidate yardstick statements, classify them by resolvability (whether a statement can be judged true or false at all), and filter and deduplicate them into the coding frame,data/coding_frame_dedup.csv.build_lineage.py,select_new_claims.pyandbuild_strip_payload.pylink each claim in the frame to its candidates, select the candidates outside the frame for a second run, and build the claim payloads for both stripping runs.wf_strip.jsandwf_strip_new.jsare multi-agent workflows, run with the payloads as theirargs. They strip the probability wording to leave a bare, self-contained proposition, split compound statements, and check each proposition against its source with an adversarial verifier. Their output,data/strip_results.jsonanddata/strip_results_new.json, cannot be regenerated exactly and is archived with the answer key (below).build_propositions.py,finalise_propositions.py(the edits made on review) andbuild_new_props.pyassemble the propositions.scenario_detailing.pyrenders the decisions on named model scenarios (for example omicron A–F), recorded indata/scenario_decisions.json, intodata/scenario_detailing_review.mdfor review. Each decision states the claim that distinguishes the scenario, rewords a reference to spell out an R value or concept, drops a figure caption, or labels a reference too vague to resolve.build_master.pyapplies them.build_master.pymerges everything, with the quality-control items fromdata/controls_key.json, intodata/propositions_master.csv, the full item set with the band and resolvability class of each item.build_survey_master.pyproduces the survey file and the item map (below).build_ledger.pywrites the audit trail (below).
textnorm.py holds the sentence normalisation that the matching scripts share. exclusions.py lists the items dropped by a scope rule or on review. The gen_jisc, gen_native, build_sheet*, build_final, build_survey_final, rclaims and contexts scripts and the wf_annotate*.js workflows are exploratory or superseded, and kept for provenance.
data/annotations.json and data/annotations_v2.json, read by the superseded build_sheet* scripts, come from two runs of the wf_annotate*.js workflows, in which an LLM judged whether each claim in the coding frame was resolvable. Both failed validation, and the study uses neither. The two runs agreed on fewer than half of the claims (Cohen's kappa 0.18) and their judgements followed the wording of the prompt. An audit also found that most "resolvable" judgements in the second run overstated what could be checked. They are kept, untracked, in data/archive_ai_runs/ as the evidence for that finding.
Four inputs cannot be recreated by any script. They are kept as data and, because each reveals bands or answers, sealed with the answer key (data/answer_key/CHECKSUMS.txt):
strip_results.jsonandstrip_results_new.json, the output of the two stripping runs (16 and 28 July 2026). Both ran in the Claude Code workflow runner, with every agent on the model aliassonnet(Claude Sonnet), as set in the scripts. The alias does not pin a model version, so a rerun can give different output.scenario_decisions.json, the decisions on scenario references, made by hand.controls_key.json, the quality-control items and their answers.
The first stripping run also included the first version of the quality-control items, which were replaced before coding (ANALYSIS_PLAN.md §9). Its archived results still contain them.
build_survey_master.py writes data/SPI-M_Yardstick_Review_FINAL.json, which imports into JISC Online Surveys (onlinesurveys.jisc.ac.uk). The survey has 310 pages: an introduction, then one page per item with the statement as a blockquote, its exact date, an assessment (correct, incorrect or cannot tell) and a confidence rating (high, medium or low). Items are titled "Item N", and no probability term or band appears anywhere.
Deploy it to independent reviewers, and make sure none of them sees another's responses until all have finished. For the analysis, export the responses from JISC and join them to data/survey_item_map.csv on the question, which gives each item its band and resolvability class.
build_lineage.py and build_ledger.py record what happened to every candidate statement between extraction and the survey:
data/lineage_candidates.csv: whether each candidate was included or excluded, and why.data/audit_ledger.csv: the transformations and final status of every proposition.
The reviewers are also the co-authors and know the study design, which ANALYSIS_PLAN.md records and which can be shared. Until they have finished coding, they must not see the answer key: the probability band of each item and the identity of the quality-control items.
The bands appear throughout the working files (survey_item_map.csv, propositions_master.csv, audit_ledger.csv and others). For that reason the repository tracks only the sealed answer key in data/answer_key/ and ignores the rest of data/. The scripts read bands only from the ignored files. Because earlier versions of the scripts contained claims with their bands, the reviewers' agreement covers the history of the repository as well as the answer key.
data/answer_key/ holds the SHA-256 checksums of the answer key, committed as proof of pre-registration, and a README recording the agreement of the reviewers not to open the key until coding is done. The plaintext answer key is added after coding, and anyone can then check it against the committed checksums. A checksum in a repository the team controls is not tamper-proof on its own; for that, register the checksums externally, for example on OSF (see data/answer_key/README.md).
The code in scripts/ is released under the MIT Licence; see LICENSE.
The SPI-M-O consensus statements and SAGE minutes are © Crown copyright, published on gov.uk under the Open Government Licence v3.0. The statements and the propositions derived from them adapt that material under that licence:
Contains public sector information licensed under the Open Government Licence v3.0.
Source: SPI-M-O consensus statements and SAGE minutes, gov.uk (Scientific Advisory Group for Emergencies). The PHIA probability yardstick, against which the verbal terms in the statements are scored, is defined in the Professional Development Framework for All-Source Intelligence Assessment (Professional Head of Intelligence Assessment, 2019).
Candidate statements were extracted from the source documents and rewritten into blinded propositions with LLM assistance and adversarial verification against the source. The prompts are in the workflow scripts (scripts/wf_strip.js, wf_strip_new.js and wf_annotate*.js), exactly as run, apart from one worked example in wf_strip.js that stays a placeholder until coding is complete (ANALYSIS_PLAN.md §9). The pipeline and audit trail are in this repository. Suggested wording for the methods section:
Candidate statements were extracted and rewritten into blinded propositions with LLM assistance and adversarial verification against the source; code and prompts are available at [DOI].
Replace [DOI] with an archived snapshot (for example on Zenodo or OSF) taken at submission, so that the citation stays stable.