diff --git a/PLAN.md b/PLAN.md index 5397035..32f0f2a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -59,10 +59,10 @@ This milestone is the product. If it is not visible on screen, this is just anot **Done when:** the repo is public, the video is recorded, and the site points at both. - [x] Sanitisation review, second pass, by someone other than the author. Read in the week of 17 to 23 Aug 2026 by an observability engineer who knows the pattern this composite is drawn from, which is a stronger pass than a cold reader can give: someone who never saw the source material can apply the rules but cannot confirm an absence. Nothing identifiable found. Scope recorded rather than assumed, because the tree grew in four steps and only the dates say what was in it: eleven corpus documents landed 16 Aug, `DRAFT-2601` on 17 Aug, the five distractors on 18 Aug in `3706dd2`, and both `docs/samples/` directories on 24 Aug. Seventeen, which is the count `README.md` and M6 carry, and the twelve M1 records as read by the first pass are the 16 Aug eleven plus `DRAFT-2601`. The whole corpus was in the tree by 18 Aug and the read ran to 23 Aug, so the corpus was covered in full and the samples postdate the read. That gap is deliberate rather than outstanding: the samples are model output, the model only ever saw synthetic fixtures and the synthetic corpus, so it cannot emit source material it never had, and its one real exposure is naming a vendor in a causal role. The reproducible half of the pass is a sweep anyone can re-run and it is clean: no email addresses anywhere in tracked files, no routable IP other than loopback, no external hostname beyond `api.anthropic.com`, `github.com`, `genai.com.au` and `www.w3.org`, and four vendor mentions all in capability tables or market positioning rather than a causal role -- [x] Repo moved to the org. Already there: `prime-vector/response-engineering`, which is the standards home the brand decision put it under. **Still private**, and that is the separate condition in this milestone's done-when rather than a leftover of this item. Nothing in the tree gates going public: the sanitisation second pass is done and recorded above, and the mechanical sweep in that item is the part a reader can re-run. What does gate it is outside the tree: the pull request refs GitHub retains, recorded in the third-pass sweep item below, keep the repository private until GitHub Support confirms their purge +- [x] Repo moved to the org. Already there: `prime-vector/response-engineering`, which is the standards home the brand decision put it under. **Still private**, and that is the separate condition in this milestone's done-when rather than a leftover of this item. Nothing in the tree gates going public: the sanitisation second pass is done and recorded above, and the mechanical sweep in that item is the part a reader can re-run. What gated it was outside the tree: the pull request refs GitHub retains, recorded in the third-pass sweep item below, and that is settled by publishing from a fresh repository rather than waiting on a purge - [x] Four minute walkthrough recorded. 4:37, uploaded unlisted on 7 Oct 2026: https://youtu.be/OTdecUh1C98. Over the four minute name on purpose, for the reason `docs/walkthrough-script.md` gives: the length is where the spoken copy and a phrased read came out, and words were not cut to a clock. The shot list's timings are cut from the recorded takes - [ ] README readable by someone with no context -- [x] Mechanical sanitisation sweep, third pass, re-run 10 Oct 2026 against `main` at `a18f46f` (`fc7dfb7` after the history rewrite of 11 Oct, see `docs/history-rewrite-2026-10-11.txt`), 141 tracked files, after the tree grew again in October: agent traces, the OpenTelemetry document, the model split and its measurements, the objectives. Clean on every axis the second pass recorded, stated as what each grep found rather than as a count: no email address in any tracked file; no IP beyond loopback; the external hostnames are the four named above plus `youtu.be`, the walkthrough link added on 7 Oct, and no other; no vendor is named in a causal role, which is rule 2 and the only thing a sweep for product names can attest, since no list of names is closed and where a name sits is a reading, not a grep. Three axes the second pass did not record were swept as well: the two record directories, `evals/results/` and `docs/samples/`, name the model provider and model ids, which is what a record of a run must carry, and no other vendor; no tracked file carries a rate, revenue figure, client name or pipeline detail; and the added and removed lines of every commit on every branch were searched for addresses and routable IPs, finding only the SSH form of the GitHub remote. Untracked files: none. What a mechanical sweep cannot do is confirm an absence of source material, which is why the second pass was a read by someone who knows the pattern, and nothing added since then was derived from anything but the generators and the model. Separately, and outside what any sweep of tracked files can see, the pre-publication review of 19 Oct 2026 found that GitHub keeps a head ref for every pull request, and 54 of the 58 still pointed at pre-rewrite commits carrying the personal address and the model-version trailers, fetchable by hash as well. Those refs cannot be deleted by the repository owner; a purge request went to GitHub Support and the repository stays private until they confirm. A sweep of the tree says nothing about what the host retains. +- [x] Mechanical sanitisation sweep, third pass, re-run 10 Oct 2026 against `main` at `a18f46f` (`fc7dfb7` after the history rewrite of 11 Oct, see `docs/history-rewrite-2026-10-11.txt`), 141 tracked files, after the tree grew again in October: agent traces, the OpenTelemetry document, the model split and its measurements, the objectives. Clean on every axis the second pass recorded, stated as what each grep found rather than as a count: no email address in any tracked file; no IP beyond loopback; the external hostnames are the four named above plus `youtu.be`, the walkthrough link added on 7 Oct, and no other; no vendor is named in a causal role, which is rule 2 and the only thing a sweep for product names can attest, since no list of names is closed and where a name sits is a reading, not a grep. Three axes the second pass did not record were swept as well: the two record directories, `evals/results/` and `docs/samples/`, name the model provider and model ids, which is what a record of a run must carry, and no other vendor; no tracked file carries a rate, revenue figure, client name or pipeline detail; and the added and removed lines of every commit on every branch were searched for addresses and routable IPs, finding only the SSH form of the GitHub remote. Untracked files: none. What a mechanical sweep cannot do is confirm an absence of source material, which is why the second pass was a read by someone who knows the pattern, and nothing added since then was derived from anything but the generators and the model. Separately, and outside what any sweep of tracked files can see, the pre-publication review of 19 Oct 2026 found that GitHub keeps a head ref for every pull request, and 54 of the 58 still pointed at pre-rewrite commits carrying the personal address and the model-version trailers, fetchable by hash as well. Those refs cannot be deleted by the repository owner, so on 21 Oct 2026 the development repository, its pull requests included, was renamed `prime-vector/response-engineering-archive` and kept private, and this repository was created under the original name from `main` alone, with no pull request refs to retain. Pull request numbers cited before that date, in this tree and in commit messages, refer to the archive, although GitHub links a `#n` in a commit message to this repository's pull request of that number. The twelve commits of 19 to 21 Oct that had carried a model-version co-author trailer were rewritten to the generic one first. That rewrite also drops the signature GitHub puts on a merge it makes, so every commit from the merge of #55 on 11 Oct onward has a new hash, 33 in all, with messages otherwise unchanged and the tree identical. A sweep of the tree says nothing about what the host retains. - [x] Walkthrough shot 3: the architecture still shows the Scribe filing the timeline, which it stopped doing in PR #42. Decided 11 Oct 2026: the recording stays and `docs/walkthrough-script.md` carries a note under shot 3 saying the still predates #42 (PR #54). The narration never mentions the filing, so nothing spoken became false; the re-rendered still is on disk for the next cut ### M6: Precedent Analyst @@ -632,7 +632,7 @@ happened to try one. - [x] Named for what they are. Agent traces are OpenTelemetry spans over OTLP with the GenAI semantic conventions; `docs/agent-traces.md` states the span model from the runner's own module and live files, the README says the local sink is one OTLP destination among any, and the incident telemetry interface keeps the two words apart (PR #45) - [x] Held to objectives. `checks/agent_traces_slo.yaml` is the only place a target lives: six per-run objectives that are contracts, one rolling budget on the driver's pass rate, and p95 latency ceilings per stage and end to end, set on the eight-run baseline of 9 Oct 2026. The aggregate evaluates against it and `demos/agent_traces_report.py --baseline` renders each objective met or missed across runs with one row per run and a sparkline per stage; `demos/agent_traces_baseline.sh` regenerates `runs/baseline.html` (PR #46) - [x] A model refusal read as a transport failure. Sonnet 5.5 returns the mitigation directive as `stop_reason: refusal` with no content blocks, and the streaming patch reported that stream as cut: `PROVIDER_ERROR`, an abandoned attempt in the trace, and a signature `demos/retry.sh` would retry once and pay for twice. Found 9 Oct 2026, fixed 10 Oct: the patch returns a refusal whole, with its `stop_reason` and `stop_details`, and the shared post helper fails naming it on both transports, since unstreamed the tool-loop path had been returning the empty reply as a final answer. `checks/stream_probe.py` holds the recorded events and both transports, and reads the retry signature out of `demos/retry.sh` to hold the two apart; `checks/retry_probe.sh` holds the wrapper to one attempt. Observed live through `oa_run` on 10 Oct: exit 1, the refusal named, one attempt, no tool call. The trace signature is unchanged, `PROVIDER_ERROR` with `execution_failed`; the name is on stderr -- [x] A CI probe for `demos/traced.sh`: `checks/traced_probe.sh`, 10 Oct 2026, runs the real wrapper through a disposable tree with stub drivers and a stub `oa` posting canned OTLP spans, and holds its routing per mode, the expectations it constructs, the exporter environment, the sink lifecycle and the exit order to what the script claims; 176 assertions, each naming its reason, the exit order pinned pairwise, and the EXIT trap held to killing a sink that never became ready; the hand mutations of the wrapper that each failed a named case are recorded on the PR, not in the tree. The sink attribute allowlist is decided against, see Open decisions +- [x] A CI probe for `demos/traced.sh`: `checks/traced_probe.sh`, 10 Oct 2026, runs the real wrapper through a disposable tree with stub drivers and a stub `oa` posting canned OTLP spans, and holds its routing per mode, the expectations it constructs, the exporter environment, the sink lifecycle and the exit order to what the script claims; 176 assertions, each naming its reason, the exit order pinned pairwise, and the EXIT trap held to killing a sink that never became ready; the hand mutations of the wrapper that each failed a named case are recorded on PR #51, which is in the private development archive, not in this repository. The sink attribute allowlist is decided against, see Open decisions - [ ] One trace per pipeline rather than per stage, waiting on OA issue #116 (TRACEPARENT) ---