Skip to content
epiforecastsPublic

About

Calibration of the probability terms in SPI-M-O COVID-19 consensus statements

Resources

Code of conduct

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

59 Commits

Folders and files

Repository files navigation

SPI-M yardstick calibration

During COVID-19, SPI-M-O, the modelling subgroup of the UK's Scientific Advisory Group for Emergencies (SAGE), wrote its consensus statements with verbal probability terms such as "likely", "highly likely" and "realistic possibility". The terms come from the probability yardstick of the Professional Head of Intelligence Assessment (PHIA), which gives each one a numeric range: "likely", for example, means 55–75%. This study asks whether the statements came true as often as their terms said. When SPI-M-O wrote "likely", was it right about 55–75% of the time?

Each statement is stripped of its probability term and shown to independent reviewers as a bare proposition. They judge, with hindsight, whether it turned out correct, without seeing the term. Grouping their verdicts by the withheld term gives the calibration.

The survey is ready to deploy. It has 309 items: 291 statements from February 2020 to February 2022 and 18 hidden quality-control items.

Start here

NOTEBOOK.md has a dated entry for each update to a line of inquiry, with the intent, what was decided or changed, and what remains uncertain. Read it before the issues and pull requests behind each entry, which have the full detail. AGENTS.md sets out the conventions for the notebook, issues and pull requests.

Repository contents

scripts/          pipeline code (run from repo root)
data/answer_key/  sealed answer-key checksums (the only tracked data)
NOTEBOOK.md       dated entry per update to a line of inquiry; start here
AGENTS.md         conventions for issues, pull requests and the notebook
ANALYSIS_PLAN.md  pre-specified analysis plan (work in progress)
README.md
LICENSE

The source documents, generated propositions, survey file and audit tables are not in the repository. Everything under data/ apart from data/answer_key/CHECKSUMS.txt and data/answer_key/README.md is git-ignored: most of it is regenerated by the pipeline, and some is withheld until coding is complete (see Sharing and embargo).

data/manifest.json holds the provenance record; git add -f it if you want it tracked.

Pipeline

The scripts in scripts/ are run from the repository root and read and write paths under data/. In rough order:

  1. list_statements.py lists the SPI-M-O consensus statements and their attachments through the gov.uk search and content APIs, into data/consensus_with_attachments.json. It uses the search API because the monthly SAGE collections miss about 12% of the statements, all of August 2021 among them. ANALYSIS_PLAN.md sets out the scope.
  2. download.py downloads the PDFs of the statements (rate-limited, with an identifying User-Agent) and extracts their text with pdftotext -layout (poppler 26.01; other versions can place text differently). It writes data/pdfs/, data/txt/ and data/manifest.json, the provenance record with SHA-256 checksums.
  3. scan.py, claims.py, classify.py, tense.py, make_frame.py and dedupe_frame.py extract candidate yardstick statements, classify them by resolvability (whether a statement can be judged true or false at all), and filter and deduplicate them into the coding frame, data/coding_frame_dedup.csv.
  4. build_lineage.py, select_new_claims.py and build_strip_payload.py link each claim in the frame to its candidates, select the candidates outside the frame for a second run, and build the claim payloads for both stripping runs.
  5. wf_strip.js and wf_strip_new.js are multi-agent workflows, run with the payloads as their args. They strip the probability wording to leave a bare, self-contained proposition, split compound statements, and check each proposition against its source with an adversarial verifier. Their output, data/strip_results.json and data/strip_results_new.json, cannot be regenerated exactly and is archived with the answer key (below). build_propositions.py, finalise_propositions.py (the edits made on review) and build_new_props.py assemble the propositions.
  6. scenario_detailing.py renders the decisions on named model scenarios (for example omicron A–F), recorded in data/scenario_decisions.json, into data/scenario_detailing_review.md for review. Each decision states the claim that distinguishes the scenario, rewords a reference to spell out an R value or concept, drops a figure caption, or labels a reference too vague to resolve. build_master.py applies them.
  7. build_master.py merges everything, with the quality-control items from data/controls_key.json, into data/propositions_master.csv, the full item set with the band and resolvability class of each item.
  8. build_survey_master.py produces the survey file and the item map (below).
  9. build_ledger.py writes the audit trail (below).

textnorm.py holds the sentence normalisation that the matching scripts share. exclusions.py lists the items dropped by a scope rule or on review. The gen_jisc, gen_native, build_sheet*, build_final, build_survey_final, rclaims and contexts scripts and the wf_annotate*.js workflows are exploratory or superseded, and kept for provenance.

data/annotations.json and data/annotations_v2.json, read by the superseded build_sheet* scripts, come from two runs of the wf_annotate*.js workflows, in which an LLM judged whether each claim in the coding frame was resolvable. Both failed validation, and the study uses neither. The two runs agreed on fewer than half of the claims (Cohen's kappa 0.18) and their judgements followed the wording of the prompt. An audit also found that most "resolvable" judgements in the second run overstated what could be checked. They are kept, untracked, in data/archive_ai_runs/ as the evidence for that finding.

Archived inputs

Four inputs cannot be recreated by any script. They are kept as data and, because each reveals bands or answers, sealed with the answer key (data/answer_key/CHECKSUMS.txt):

  • strip_results.json and strip_results_new.json, the output of the two stripping runs (16 and 28 July 2026). Both ran in the Claude Code workflow runner, with every agent on the model alias sonnet (Claude Sonnet), as set in the scripts. The alias does not pin a model version, so a rerun can give different output.
  • scenario_decisions.json, the decisions on scenario references, made by hand.
  • controls_key.json, the quality-control items and their answers.

The first stripping run also included the first version of the quality-control items, which were replaced before coding (ANALYSIS_PLAN.md §9). Its archived results still contain them.

The survey

build_survey_master.py writes data/SPI-M_Yardstick_Review_FINAL.json, which imports into JISC Online Surveys (onlinesurveys.jisc.ac.uk). The survey has 310 pages: an introduction, then one page per item with the statement as a blockquote, its exact date, an assessment (correct, incorrect or cannot tell) and a confidence rating (high, medium or low). Items are titled "Item N", and no probability term or band appears anywhere.

Deploy it to independent reviewers, and make sure none of them sees another's responses until all have finished. For the analysis, export the responses from JISC and join them to data/survey_item_map.csv on the question, which gives each item its band and resolvability class.

Audit trail

build_lineage.py and build_ledger.py record what happened to every candidate statement between extraction and the survey:

  • data/lineage_candidates.csv: whether each candidate was included or excluded, and why.
  • data/audit_ledger.csv: the transformations and final status of every proposition.

Sharing and embargo

The reviewers are also the co-authors and know the study design, which ANALYSIS_PLAN.md records and which can be shared. Until they have finished coding, they must not see the answer key: the probability band of each item and the identity of the quality-control items.

The bands appear throughout the working files (survey_item_map.csv, propositions_master.csv, audit_ledger.csv and others). For that reason the repository tracks only the sealed answer key in data/answer_key/ and ignores the rest of data/. The scripts read bands only from the ignored files. Because earlier versions of the scripts contained claims with their bands, the reviewers' agreement covers the history of the repository as well as the answer key.

data/answer_key/ holds the SHA-256 checksums of the answer key, committed as proof of pre-registration, and a README recording the agreement of the reviewers not to open the key until coding is done. The plaintext answer key is added after coding, and anyone can then check it against the committed checksums. A checksum in a repository the team controls is not tamper-proof on its own; for that, register the checksums externally, for example on OSF (see data/answer_key/README.md).

Licence and attribution

The code in scripts/ is released under the MIT Licence; see LICENSE.

The SPI-M-O consensus statements and SAGE minutes are © Crown copyright, published on gov.uk under the Open Government Licence v3.0. The statements and the propositions derived from them adapt that material under that licence:

Contains public sector information licensed under the Open Government Licence v3.0.

Source: SPI-M-O consensus statements and SAGE minutes, gov.uk (Scientific Advisory Group for Emergencies). The PHIA probability yardstick, against which the verbal terms in the statements are scored, is defined in the Professional Development Framework for All-Source Intelligence Assessment (Professional Head of Intelligence Assessment, 2019).

Code and data availability

Candidate statements were extracted from the source documents and rewritten into blinded propositions with LLM assistance and adversarial verification against the source. The prompts are in the workflow scripts (scripts/wf_strip.js, wf_strip_new.js and wf_annotate*.js), exactly as run, apart from one worked example in wf_strip.js that stays a placeholder until coding is complete (ANALYSIS_PLAN.md §9). The pipeline and audit trail are in this repository. Suggested wording for the methods section:

Candidate statements were extracted and rewritten into blinded propositions with LLM assistance and adversarial verification against the source; code and prompts are available at [DOI].

Replace [DOI] with an archived snapshot (for example on Zenodo or OSF) taken at submission, so that the citation stays stable.

About

Calibration of the probability terms in SPI-M-O COVID-19 consensus statements

Resources

Code of conduct

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages