Make release Experiments opt-in

docs/guide.md

@@ -2007,8 +2007,12 @@ cargo test
 
 ## Releasing
 
-Before pushing anything, the release script runs `thirdshift experiment
---release <commit>` using this checkout's runner and private Eval suites.
+The release script skips Experiments by default, printing a note. Opt in with
+`scripts/release.sh --with-experiment 0.4.0`, adding `--with-experiment` again
+on each resume that should run an Experiment.
+
+With `--with-experiment`, before pushing anything, the script runs
+`thirdshift experiment --release <commit>` using this checkout's runner and private Eval suites.
 It evaluates the content commit beneath the version-only bump, so reruns use
 the same Experiment even before a bump branch exists. It pins that commit in a
 local Git ref before starting, so a merge to `main` during interruption does
@@ -2019,18 +2023,15 @@ intervals and asks `release anyway? [y/N]`, including without `--review`.
 Only `y` or `yes` continues; EOF or any other answer stops. Pending borderline
 and decisive Scores are listed with their recheck commands and polled until
 you have rechecked them. No touched suite or a last release with no Trial flag
-prints a note and carries on. `--no-experiment` skips the release Experiment
-outright, on the first run and on every resume, and prints a line saying so:
-use it only when an Experiment can't give a Verdict worth its Trials, such as
-while every Trial is known to be void. An interrupted invocation resumes its saved
+prints a note and carries on. An interrupted invocation resumes its saved
 Trials. A pushed bump branch is checked again before opening its PR, and an
 open or merged bump PR before merging or tagging. An already pushed tag only
 resumes the release workflow. Python 3 is needed to read the Experiment results.
 
 If `main` gains different content after the Candidate was selected, the script
-refuses to push or merge that release under its old Experiment; choose a fresh
-release version to measure current `main`. It also checks the merged tree before
-tagging, so a concurrent merge cannot publish unmeasured content.
+refuses to push or merge that release; choose a fresh release version for current
+`main`. It also checks the merged tree before tagging, so a concurrent merge
+cannot publish different content. These checks apply even without an Experiment.
 
 From a clone of this repo, signed in to `gh` and the User config's default Harness, run the release script with the new version:
 

scripts/release.sh

@@ -5,7 +5,7 @@
 # workflow, and the script waits for that workflow run to publish the GitHub
 # Release.
 #
-# Usage: scripts/release.sh [--review] [--no-experiment] <version>
+# Usage: scripts/release.sh [--review] [--with-experiment] <version>
 #
 # Run it in a clone of this repo, signed in to gh and the default Harness. It
 # works from origin/main in a temporary worktree, so the branch checked out
@@ -24,13 +24,13 @@
 # didn't succeed. It waits for that tip's run to appear and finish, following
 # main if it moves and naming each move. The version bump starts from the tip
 # whose CI succeeded, with the requested version checked against that tip.
-# Then an Experiment compares that content commit with its last release tag,
-# selecting touched suites. A pass carries on; pending Scores are listed and
-# waited for; fail/inconclusive Verdicts show intervals and require an override.
+# With --with-experiment, an Experiment compares that content commit with its
+# last release tag, selecting touched suites. A pass carries on; pending Scores
+# are listed and waited for; fail/inconclusive Verdicts show intervals and
+# require an override.
 # Untouched suites or a Baseline with no Trial flag need no Experiment.
-# --no-experiment skips the release Experiment outright, on a fresh run and on
-# every resume, and says so; use it only when an Experiment can't give a
-# Verdict worth its Trials.
+# Experiments are skipped by default, on a fresh run and on every resume,
+# with a progress note. Supply --with-experiment on each invocation to run one.
 #
 # If a run is interrupted, running it again with the same version carries on
 # from the first step not yet done: it opens the PR for a pushed branch, waits
@@ -74,17 +74,17 @@ poll_seconds=${RELEASE_POLL_SECONDS:-15}
 
 main() {
 	review=
-	no_experiment=
+	with_experiment=
 	while :; do
 		case ${1-} in
 		--review) review=1 ;;
-		--no-experiment) no_experiment=1 ;;
+		--with-experiment) with_experiment=1 ;;
 		*) break ;;
 		esac
 		shift
 	done
 	if [ "$#" -ne 1 ]; then
-		echo "usage: release.sh [--review] [--no-experiment] <version>" >&2
+		echo "usage: release.sh [--review] [--with-experiment] <version>" >&2
 		exit 2
 	fi
 	version=$1
@@ -262,8 +262,8 @@ commit_bump() {
 # Use this checkout's runner and private Eval suites, as for summary_agent.
 wait_for_experiment() {
 	local snapshot status outcome id answer
-	if [ -n "$no_experiment" ]; then
-		progress "skipping the release Experiment (--no-experiment) for ${1:0:7}"
+	if [ -z "$with_experiment" ]; then
+		progress "skipping the release Experiment (use --with-experiment to run it) for ${1:0:7}"
 		return
 	fi
 	progress "checking release Experiment for ${1:0:7}"

site/architecture/index.html

@@ -216,7 +216,7 @@ <h1>Architecture</h1>
       <p>Trial completion removes Harness caches from each attempt's Session home, retaining native transcripts and Session evidence. Once grading and all rechecks finish, the Experiment removes every attempt's repository copy. Paused and interrupted Experiments retain repositories for resume. Requests, results, traces, reports, Grader packets and Scores stay readable; pruning failures only warn.</p>
       <p>Each Trial, code Grader and model Grader gets its own <code>systemd-run --user --scope</code> with <code>MemoryHigh</code>, <code>MemoryMax</code>, <code>MemorySwapMax</code> and <code>OOMPolicy=kill</code>. Its process raises <code>oom_score_adj</code> before starting work. The <a href="/docs/#user-config">User config</a> defaults are 1 GiB, 2 GiB and 1 GiB; scopes require a reachable user manager with lingering enabled for logged-out work.</p>
       <p>The Experiment owns recheck history beside its immutable Grader Scores. <code>thirdshift experiments <wbr>&lt;id&gt; --json</code> exposes pending borderline and decisive Scores. <code>thirdshift recheck <wbr>&lt;id&gt; &lt;trial&gt; pass|fail --reason '…'</code> takes the Experiment's exclusive lock and atomically saves the history, recalculated Verdict and pending list. It also updates the latest invocation's Command outcome and Inbox group, preserving its completion time. The Verdict waits until the list is empty; flipping either pass/fail or a wrong-finding flag can make a Score decisive. Waiting rechecks place that invocation in Needs you; a confirmed pass moves it to Done.</p>
-      <p><code>thirdshift experiment <wbr>--release <wbr>&lt;commit&gt;</code> compares the content commit with the last release tag on touched suites. The release script waits for rechecks and asks permission for a fail or inconclusive Verdict before pushing. Reruns reuse the Experiment, including before opening a pushed branch's PR. It refuses different main content before pushing or merging and checks the merged tree before tagging, so the release cannot contain unmeasured changes. No touched suite or a last release without the Trial flag carries on with a note. <a href="/docs/#releasing">Releasing</a>.</p>
+      <p><code>thirdshift experiment <wbr>--release <wbr>&lt;commit&gt;</code> compares the content commit with the last release tag on touched suites. The release script skips Experiments by default. Add <code>--with-experiment</code> to <code>scripts/release.sh</code> on each invocation to run one: it waits for rechecks and asks permission for a fail or inconclusive Verdict before pushing. Reruns with the flag reuse the Experiment, including before opening a pushed branch's PR. It refuses different main content before pushing or merging and checks the merged tree before tagging, including without an Experiment. No touched suite or a last release without the Trial flag carries on with a note. <a href="/docs/#releasing">Releasing</a>.</p>
       <p>Each Ticket run's own process saves its <strong>Run metrics</strong> at completion in its Command record, including Ticket runs and Base fixes: outcome, Repairs, CI rounds, Unaddressed findings by Standards, Spec and Security, stage times against their Usual ranges, and minutes by stage and in total. Unaddressed findings counts retain Delivery's final PR body and the Security review's count, so later edits do not change them. Each Session stage keeps the review's Standards and Spec finding counts from its summary, with missing counts left missing. <code>thirdshift show &lt;command&gt; --json</code> includes these retained facts. The Console reads the same details, showing each Ticket run's available metrics and marking unavailable signals as Missing. <code>thirdshift metrics &lt;command&gt; [--json]</code> reads a Ticket run with its children; without a Command it groups weekly trends by repository, Harness, requested Model and Effort, Ticket run kind and session kind. Week and repository filters narrow that view. Historical metrics are computed from Command records without writing back, and missing signals stay missing. No combined score is produced.</p>
       <p>Run metrics and Experiments share one Session usage module for tokens by observed Model and Harness-reported cost. Its normalized-event and retained-span readers resolve call amendments and partial totals, retaining fields a total omits. Input includes cache reads and writes. Reported Session cost wins; otherwise each Model's reported cost replaces its call costs. The metrics reader separately counts tool errors, permission denials, killed work and repeated tool calls within that session. Completion metrics sum these by session kind, and weekly Session trends split by the Harness version from each trace's resource. Saved completion metrics stay fixed. Historical metrics also read converted Session traces, leaving missing traces and unreported measurements missing.</p>
       <p>Command details opens with the same subject and issue title as the Inbox, before Stop and Dismiss. The subject links to the issue and long titles wrap. Unknown titles say “(title unavailable)”; Commands without an issue show only their Inbox subject. The recorded issue title comes from the same details JSON as <code>thirdshift show &lt;command&gt; --json</code>.</p>

site/docs/index.html

@@ -1691,7 +1691,11 @@
 </div></details>
 <details class="fold"><summary><h2>Releasing</h2></summary>
 <div class="manual-content" id="releasing">
-<p>Before pushing anything, the release script runs <code>thirdshift experiment --release &lt;commit&gt;</code> using this checkout&#39;s runner and private Eval suites.
+<p>The release script skips Experiments by default, printing a note. Opt in with
+<code>scripts/release.sh --with-experiment 0.4.0</code>, adding <code>--with-experiment</code> again
+on each resume that should run an Experiment.</p>
+<p>With <code>--with-experiment</code>, before pushing anything, the script runs
+<code>thirdshift experiment --release &lt;commit&gt;</code> using this checkout&#39;s runner and private Eval suites.
 It evaluates the content commit beneath the version-only bump, so reruns use
 the same Experiment even before a bump branch exists. It pins that commit in a
 local Git ref before starting, so a merge to <code>main</code> during interruption does
@@ -1702,17 +1706,14 @@
 Only <code>y</code> or <code>yes</code> continues; EOF or any other answer stops. Pending borderline
 and decisive Scores are listed with their recheck commands and polled until
 you have rechecked them. No touched suite or a last release with no Trial flag
-prints a note and carries on. <code>--no-experiment</code> skips the release Experiment
-outright, on the first run and on every resume, and prints a line saying so:
-use it only when an Experiment can&#39;t give a Verdict worth its Trials, such as
-while every Trial is known to be void. An interrupted invocation resumes its saved
+prints a note and carries on. An interrupted invocation resumes its saved
 Trials. A pushed bump branch is checked again before opening its PR, and an
 open or merged bump PR before merging or tagging. An already pushed tag only
 resumes the release workflow. Python 3 is needed to read the Experiment results.</p>
 <p>If <code>main</code> gains different content after the Candidate was selected, the script
-refuses to push or merge that release under its old Experiment; choose a fresh
-release version to measure current <code>main</code>. It also checks the merged tree before
-tagging, so a concurrent merge cannot publish unmeasured content.</p>
+refuses to push or merge that release; choose a fresh release version for current
+<code>main</code>. It also checks the merged tree before tagging, so a concurrent merge
+cannot publish different content. These checks apply even without an Experiment.</p>
 <p>From a clone of this repo, signed in to <code>gh</code> and the User config&#39;s default Harness, run the release script with the new version:</p>
 <pre><code class="language-sh">scripts/release.sh 0.4.0
 </code></pre>

site/how-it-works/index.html

@@ -240,7 +240,7 @@ <h1>How thirdshift works</h1>
       <p>Proposed Eval tasks appear on the Experiments list and, for its suites, on each Experiment page. Approve one there or with <code>thirdshift evals approve &lt;suite&gt; &lt;task&gt; --json</code> to change its state in its suite, preserving its inputs. Approval starts no work and affects future Experiments; existing Experiments keep their fixed Eval tasks on resume. Unapproved proposals never run.</p>
       <p>Each Trial and each Grader runs in its own memory-capped scope and raises its OOM score to give way before the factory. The <a href="/docs/#user-config">User config</a> defaults to 1 GiB for throttling, 2 GiB for killing and 1 GiB of swap per scope. Enable lingering once with <code>loginctl enable-linger &lt;user&gt;</code>; an Experiment stops before dispatch if it cannot reach the systemd user manager.</p>
       <p>A Verdict stays waiting on rechecks until you have checked every borderline Score and every decisive Score, whose flip would change the Verdict. The pending list gives Trial IDs for <code>thirdshift recheck <wbr>&lt;id&gt; &lt;trial&gt; pass|fail --reason '…'</code>. Your latest decision feeds the calculation; the Grader's Score and your timestamped history remain. Each recheck updates the pending list and the latest invocation's Command outcome and Inbox group. A confirmed pass moves from Needs you to Done.</p>
-      <p><code>thirdshift experiment <wbr>--release <wbr>&lt;commit&gt;</code> compares touched suites with the last release tag. The release script waits for rechecks before pushing and asks whether to release on a fail or inconclusive Verdict. Reruns reuse the Experiment, including before opening a pushed branch’s PR. If main gains different content, the script refuses to publish it under the old Candidate's Experiment; choose a fresh release version. No touched suite or a last release with no Trial flag carries on with a note. <a href="/docs/#releasing">Release checks</a>.</p>
+      <p><code>thirdshift experiment <wbr>--release <wbr>&lt;commit&gt;</code> compares touched suites with the last release tag. The release script skips Experiments by default. Add <code>--with-experiment</code> to <code>scripts/release.sh</code> on each invocation to run one: it waits for rechecks before pushing and asks whether to release on a fail or inconclusive Verdict. Reruns with the flag reuse the Experiment, including before opening a pushed branch’s PR. If main gains different content, the script refuses to publish it; choose a fresh release version. No touched suite or a last release with no Trial flag carries on with a note. <a href="/docs/#releasing">Release checks</a>.</p>
       <p><a href="/docs/#architect-runs-every-few-minutes">Architect runs every few minutes</a> and <a href="/docs/#security-runs-on-a-schedule-with-fixing-allowed">Security runs on a schedule, with fixing allowed</a>, started by your operating system.</p>
       <p>Next: <a href="/prompts/">inspect every prompt and skill</a>, see <a href="/#how-a-run-works">a real Ticket run's recorded output</a>, see <a href="/architecture/">the architecture in six figures</a>, use the <a href="/docs/">manual for installation, commands and configuration</a>, or <a href="/videos/">watch the explainer videos</a>.</p>
       </div>