diff --git a/.gitignore b/.gitignore index dfe8a59b..583fe264 100644 --- a/.gitignore +++ b/.gitignore @@ -17,3 +17,6 @@ sentinel_flow.json # ChatGPT collaboration binding (local only) .chatgpt-collaboration.json + +# Phase 0 recon raw captures (conversation ids/titles — local only) +captures/ diff --git a/docs/research/chatgpt-web-contract-2026-09.md b/docs/research/chatgpt-web-contract-2026-09.md new file mode 100644 index 00000000..93b7c0e7 --- /dev/null +++ b/docs/research/chatgpt-web-contract-2026-09.md @@ -0,0 +1,223 @@ +# ChatGPT Web Contract — Delta Report, 2026-09-20 + +Phase 0 of the upstream reconnaissance gate. Compares the upstream behavior +assumed at `master` (`497527d`, 2026-07-12) against live observation of +ChatGPT Web on 2026-09-20. + +## Method and environment + +- Evidence: passive capture only. No tab was created, closed, or navigated; + no message was sent; no UI control was clicked. The bridge was running + throughout (port 8080) and was left untouched. +- Environment: Chrome 153.0.8010.48 via CDP on 127.0.0.1:9222, logged-in + session, 26 `chatgpt.com` tabs discovered (several frozen by Chrome's + memory saver; those are excluded from DOM evidence). +- Backend reads: same-origin GETs from within a live tab, replicating the + repo's own fetch shape (`Authorization: Bearer` token from + `/api/auth/session`). No token values, message text, or credentials are + retained in the capture files. +- Raw evidence: `captures/phase0/passive-20260920-224425.json`, + `captures/phase0/probe2-projections.json`, + `captures/phase0/probe3-dom-details.json`, + `captures/phase0/probe4-ui-census.json`, + `captures/phase0/probe5-model-sidebar.json` (local only; gitignored). +- User-provided screenshots (2026-09-20): composer "+" menu open; sidebar + with the More flyout (Images/Health/GPTs); the effort chip hover tooltip + ("Thinking effort — Ctrl+Shift+M"); the effort menu open (slider, "High >", + a locked position beyond); and the model list showing "GPT-5.6 Sol ✓" and + "GPT-5.5 — Leaving on October 14". Used as evidence for open-menu + contents, which cannot be captured passively (verified: no menu markup + exists in the closed state — probe6). +- Probe tool: `scripts/phase0_passive_recon.py` (re-runnable). + +## Contract matrix + +| Contract | July assumption (master) | Current observation (2026-09-20) | Class | Impact | +| --- | --- | --- | --- | --- | +| Composer | `div[role="textbox"]#prompt-textarea` / `.ProseMirror` (`chatgpt_dom.py:76`) | Both hit on every live tab; `div#prompt-textarea.ProseMirror[contenteditable][role=textbox]`; textarea fallback absent | UNCHANGED | type/verify | +| Send control | `button[aria-label*="Send" i]` + testid fallback + `form:has(#prompt-textarea) button[type=submit]` broad (`chatgpt_dom.py:85-94`) | With empty composer, no send button exists in DOM at all — consistent with July comments ("no data-testid=send-button; affordances are composer-plus-btn and dictation"). `composer-plus-btn` observed. Send-time resolution not yet re-observed | UNCERTAIN (provisionally unchanged) | send safety | +| Attachments entry | UI upload workflow (PR #49 open) | `composer-plus-btn` opens a merged tools/connectors/files menu (user screenshot; menu closed at capture time): Add photos & files (Upload from computer), Add from library, Create image, Web search, Deep research, Sketch, plus connectors GitHub, OpenAI Platform, ClickUp; footer "Type to search plugins, files, folders & skills" | CHANGED (menu scope) | PR #49 = first entry; more UI states to avoid | +| Message containers | `[data-message-author-role=user|assistant]` (`cdp_driver.py:1500-1509`) | Present; counts readable; `[data-testid^="conversation-turn"]` matches mounted turns | UNCHANGED | counting, TurnAnchor DOM guard | +| Message attributes | `data-message-id` on message nodes | Present, **plus new**: `data-message-model-slug` (observed `gpt-5-6-thinking` on a tab whose effort selector reads "High" — consistent with the runtime slug encoding reasoning mode; confirmation pending a controlled send), `data-turn-start-message` | NEW | model identity — new stable signal | +| Generating state | `.result-thinking` / `.result-streaming` / `.markdown` lifecycle (docs/reverse-engineering-notes.md) | `.markdown` present on settled messages; thinking/streaming classes require an active generation — not observed passively | PENDING-CONTROLLED | completion detector | +| Fresh-chat route | `/` | `https://chatgpt.com/` tab live, composer present | UNCHANGED (form only; navigation behavior untested) | navigation | +| Existing chat | `/c/{id}` | Live tabs use it; id extraction (`/c/` split, `backend_client.py:233-235`) still parses | UNCHANGED (form only) | navigation | +| Project chat | `/g/g-p-…/c/{id}` | Live tabs use it, incl. `/g/…/project?tab=sources` | UNCHANGED (form only) | navigation | +| Auth session | `/api/auth/session` → keys incl. `accessToken` | Status 200, same key set. **Cookies-only fetches to `/backend-api/conversation/{id}` return 404 (masked); Bearer required** — repo already always sends it | UNCHANGED (bearer requirement confirmed) | token refresh | +| Conversation projection | GET `/backend-api/conversation/{id}` → `mapping`, `current_node`, rich metadata (`backend_projection.py`) | 200 with Bearer on plain + project conversations; `current_node` present; top-level keys incl. `gizmo_id`, `default_model_slug`, `conversation_template_id`, `atlas_mode_enabled` | UNCHANGED (structurally) | TurnAnchor primary path | +| Long-chat DOM | All historical turns mounted; `querySelectorAll` counts reflect conversation length | **Sectioned loading confirmed**: Web2API project conversation `6a8149d3` has 314 backend mapping nodes (5 user, 129 assistant, 179 tool) but only 5 turns mounted (2 user + 3 assistant) in an idle, loaded tab. Plain conversation `6aaa4c07`: 142 backend nodes vs 1+1 mounted in run-1 sample | **CHANGED / highest priority** | count baselines, stale-return race | +| Models | `MODEL_MAP` (`api_server.py:36`): gpt-5.5…gpt-5 + legacy aliases | Account `/backend-api/models` returns 6: `auto`, `gpt-5-3-mini`, `gpt-5-5-mini`, `gpt-5-5`, `gpt-5-6-mini` ("GPT-5.6 Luna"), `gpt-5-6` ("GPT-5.6 Luna"). `gpt-5-2/5-1/5` absent from list. Runtime slug `gpt-5-6-thinking` observed in DOM but not in models list. Picker (user screenshot) titles the active model **"GPT-5.6 Sol"** while /models titles `gpt-5-6` "Luna" — naming split unresolved, do not hardcode either. **"GPT-5.5 — Leaving on October 14"**: `gpt-5-5` family has a dated deprecation (2026-10-14) | CHANGED | model selection; `gpt-5.5*` aliases expire 2026-10-14 | +| Model picker | `#model-selector-btn` / `button[aria-label*="Model"]` / `[data-testid*="model"]` (`cdp_driver.py:1122`) | **RESTRUCTURED (screenshots + probe6)**: composer chip = `button.__composer-pill[aria-haspopup="menu"]`, text shows current effort ("High"); hover tooltip "Thinking effort — Ctrl+Shift+M". Open, it becomes a Radix menu: effort **slider** ("High >") with a **locked position beyond** (plan-gated tier, unidentified), and a model list nested behind the effort level: "GPT-5.6 Sol ✓" selected, "GPT-5.5 — Leaving on October 14". Separate per-turn `button[aria-label="Switch model"]` hover buttons also exist on conversation tabs. July's selector matches none of the composer chip | CHANGED | `select_model` finds no legacy picker (non-fatal False on all surfaces today); model+effort selection is nested behind one chip | +| Model identity | one-dimensional slug (`MODEL_MAP` → picker) | **Two-dimensional**: (model slug, reasoning effort). Live pairing: effort "High" + account model gpt-5-6 ↔ runtime slug `gpt-5-6-thinking` in `data-message-model-slug`. Effort slider positions not yet enumerated (open-menu capture pending) | CHANGED | selection, per-turn model verification | +| Large paste | composer text at any length | Threshold behavior (≥~5,000 chars → attachment per third-party reporting) requires an active paste test | PENDING-CONTROLLED | type_message, send-ack, TurnAnchor fallback | +| Custom GPTs | active capability (`list_gpts`, `chat_with_gpt`) | Retirement claim found only in unofficial, mutually contradictory sources; no official confirmation located. Sidebar census (live DOM + user screenshot): top-level items are New chat, Library, Scheduled, **Plugins**, **Sites (NEW badge)**, More flyout → Images / Health / **GPTs**. GPTs and Plugins coexist today; GPTs reachable only via the More flyout | UNVERIFIED RUMOR (retirement); surface: GPTs demoted to flyout | MCP roadmap only | +| User-message identity | outgoing conversation request carries client message id (IdentityListener) | Requires a send to observe | PENDING-CONTROLLED | TurnAnchor primary | + +## Key findings + +1. **Sectioned loading is real and changes the DOM-count contract.** The + backend graph is the only complete representation of a conversation; the + DOM now holds only the tail. The delta-based send baseline + (`cdp_driver.py:1484`) still functions if mounting is stable when the + baseline is read, but (a) any code treating DOM counts as conversation + length is now wrong, and (b) the baseline read can race progressive + mounting right after navigation — the retry/fail-closed logic guards JS + failure, not a cleanly-read-but-still-rising count. The 50/100/200-turn + fixture test exists to size this. +2. **A new stable model-identity signal exists: `data-message-model-slug` + on assistant message containers.** Observed `gpt-5-6-thinking`. This is + exactly the "better semantic signal" the audit was looking for — it can + verify model selection per-turn without trusting the picker UI, and it + reveals runtime slugs (`gpt-5-6-thinking`) absent from `/backend-api/models`. + The live evidence pairs `gpt-5-6-thinking` with an effort selector + reading "High", suggesting that the runtime slug may encode reasoning + mode; a controlled model/effort send matrix is required before defining + the mapping. +3. **Model selection is now two-dimensional, effort-first, and nested.** + One composer chip ("Thinking effort", Ctrl+Shift+M) opens a slider with + the model list attached behind the current effort level. A correction to + an earlier pass of this report: the passive probes concluded "the picker + is gone" because no picker markup exists in the closed DOM — the user's + screenshots show it does exist behind interaction. This is exactly the + open-menu blind spot flagged in the method notes. Consequences for the + bridge: `select_model` finds no legacy picker on any surface (non-fatal + False — best-effort design tolerates it), model choice rides on the + effort chip, and per-turn "Switch model" buttons exist separately on + conversations. Any future model control should target the + `__composer-pill` chip + `data-message-model-slug` verification rather + than the legacy picker selectors. +4. **Dated deprecation: the gpt-5-5 family leaves on 2026-10-14.** The + picker advertises "GPT-5.5 — Leaving on October 14" while `MODEL_MAP` + still exposes `gpt-5.5` / `gpt-5.5-thinking` and `/models` still lists + `gpt-5-5` / `gpt-5-5-mini`. Requests mapped to gpt-5-5 after that date + will fall to best-effort default. Alias migration (gpt-5.5 → gpt-5-6 or + removal with a clear error) belongs on the roadmap with the date + attached. +5. **The models surface moved past `MODEL_MAP`.** This account serves + GPT-5.6 and 5.6-mini; `MODEL_MAP` has no 5.6 entries, and its + `gpt-5-2/5-1/5` targets no longer appear in the account's list. + Both 5.6 names now appear on this same account — the picker shows + "GPT-5.6 Sol" as active while `/models` titles `gpt-5-6` "Luna" — + so neither name can be treated as the universal one. +6. **Projection and auth contracts held.** The TurnAnchor primary path's + backend shape is unchanged; bearer-authenticated fetches behave as in + July. Unauthenticated (cookie-only) fetches 404 rather than 401 — + relevant only if auth-expiry detection ever relies on status codes from + unauthenticated calls (the repo does not). +7. **Composer and message-container selectors held.** No reactive patching + needed there today. Sidebar/branding surfaces moved instead: Plugins and + Sites are first-class nav items, GPTs lives under the More flyout, and + the composer "+" menu now merges files, tools, and connectors. + +## Pending controlled-capture checklist (next session) + +Blocking for issue #51 design: + +1. Send boundary with content in the composer: does + `button[aria-label*="Send" i]` resolve; does the `form:has()` fallback; + Send→Stop transition during generation; click-then-CDP-interrupt + behavior. +2. Generation-state class cycle on a live response (`.result-thinking`, + `.result-streaming`, `.markdown` fill, copy-button appearance). +3. Paste threshold: >5,000-char paste — attachment conversion, composer + text state, send-ack implications, TurnAnchor fallback text-match + implications. +4. Model selection (revised scope): the model list and slider shape are now + screenshot-known; remaining — enumerate the effort slider positions and + the locked tier, capture the open menu's DOM structure (Radix menu off + the `__composer-pill` chip), enumerate a per-turn "Switch model" menu, + and confirm the effort→runtime-slug mapping (`gpt-5-6` + High → + `gpt-5-6-thinking`?) with a controlled send reading + `data-message-model-slug`. + +Long-chat and navigation suites: + +5. 50/100/200-turn fixtures in a disposable project: baseline-race test + (read count immediately after navigation, watch it rise), tail-stability + test, projection-vs-DOM at each size. +6. Navigation recertification: `/`, `/c/{id}`, project routes, back/forward, + SPA normalization (PR #50's requirement). + +Attachments (blocks PR #49 rehabilitation only): file select, upload states, +removal, multi-file, failures. + +## Explicitly not tested in this pass + +- Nothing requiring interaction: no send, no click, no paste, no picker + open, no navigation. All send-boundary, generation-state, paste, and + picker rows above are pending, not negative, findings. +- No long-conversation fixture was created; the sectioned-loading numbers + come from existing conversations (314-node and 142-node samples). +- Frozen tabs (~several of 26) were excluded; their DOM state is unknown. +- Single account, single plan, one capture session — rollout variance + unaddressed. Re-run `scripts/phase0_passive_recon.py` before each phase + that depends on this contract. + +## Baseline freeze and execution order (post-freeze addendum, 2026-09-21) + +The passive baseline above is frozen as the Phase 0 passive snapshot; this +addendum records decisions made after the freeze and does not modify the +body. Classification: passive reconnaissance COMPLETE; current contract +snapshot FROZEN; controlled reconnaissance NOT COMPLETE. + +Provisional invariant promoted from the evidence (binding until the +controlled model/effort matrix tests it): + +> Configured/catalog model identity (`/backend-api/models`, `MODEL_MAP`) +> ≠ selected UI identity (picker label) +> ≠ observed turn execution identity (`data-message-model-slug`). + +The pending controlled-capture queue is reordered: the confirmed +sectioned-loading delta gates the send-acknowledgment evidence model, so +the baseline-race experiment runs first. + +1. Long-chat baseline-race experiment (50/100/200 turns; after navigation, + sample URL, readyState, composer presence, user/assistant counts, and + mounted turn IDs at 100–250 ms until stable; determine whether + composer-ready precedes mounted-turn-set-stable). +2. Send boundary / issue #51 mutation experiment, with four independent + evidence questions: did Runtime.evaluate reach Chrome; did the JS event + sequence execute; did React accept the submission; did the backend + create the intended user node. Outcome-unknown states reconcile only, + never resend. +3. Generation-state lifecycle (thinking/streaming/markdown/copy-button). +4. Model × effort send matrix (records picker label, effort state, /models + entry, outgoing request fields, DOM slug, backend metadata per turn; + defines ModelSelection(base_model, reasoning_effort) only after + evidence). +5. Large-paste conversion threshold. +6. Navigation recertification. +7. Attachment lifecycle. + +Design direction recorded for #51 (not implemented): evolve send +acknowledgment from DOM-count delta to turn-identity evidence — outgoing +request UUID observed, matching user message ID, backend graph contains the +submitted user node — with the DOM-count delta retained only as a secondary +hint. Sectioned loading makes DOM population unsuitable as +conversation-history authority, so waiting for DOM-count stability is +explicitly rejected as the fix. + +Experiment 1A executed 2026-09-21 under this authorization: the mounting +race was NOT observed (0/5 cold-load runs — atomic hydration commit brings +composer + mounted window up together; ~6 s readyState-to-hydration dead +zone documented). Results and scope: +`experiment-1a-mounting-race.md` (same directory). Experiment 1B (one +normal controlled send, disposable project) is next authorized and had not +been executed at the time of this addendum. + +Experiment 1B executed 2026-09-21 (after the paragraph above was written): +outcome class A — outgoing-UUID capture (+139 ms) precedes every DOM +acknowledgment signal (+437 ms), and the captured UUID equals the backend +node id exactly. The send path is now POST `/backend-api/f/conversation` +with prepare + sentinel wrapping; backend clock skew measured at ≈ −40 ms +(June's +5.1 s constant stale); `.result-thinking`/`.result-streaming` +absent during a thinking-model generation. Details: +`experiment-1b-send-timeline.md`. Matrix-row deltas recorded here, frozen +body unchanged: Generating state → PENDING-CONTROLLED; negative evidence: +`.result-thinking` / `.result-streaming` absent in 1/1 controlled +`gpt-5-6-thinking` run. Send-path endpoints → CHANGED (`f/conversation`). +User-message identity → CONFIRMED for this path by 1B: outgoing +client-message ID = backend mapping node ID = backend message ID; UUID +capture preceded DOM acknowledgment by ~300 ms. Next in queue: #51 deterministic +transport-level replay reproduction (offline), then the generation +lifecycle capture. diff --git a/docs/research/experiment-1a-mounting-race.md b/docs/research/experiment-1a-mounting-race.md new file mode 100644 index 00000000..4fba36e2 --- /dev/null +++ b/docs/research/experiment-1a-mounting-race.md @@ -0,0 +1,87 @@ +# Experiment 1A — Long-Chat Mounting Race (2026-09-21) + +Controlled read-only experiment, executed under the authorization recorded +in the Phase 0 report addendum. Question: does the composer become ready +before the mounted DOM of a sectioned long conversation is stable — the +ordering the bridge's pre-send DOM-count baseline implicitly assumes? + +## Verdict + +**The race was not observed: 0 of 5 runs.** In every run the mounted DOM was +completely empty through document-complete, then the composer and the entire +mounted conversation window appeared together in a single atomic commit. The +claim is scoped exactly as the acceptance criterion requires: no +post-readiness mounting change was observed in *this* environment, +conversation, Chrome build, account, and capture runs. This falsifies +nothing globally; it fails to confirm the hazard on the path tested. + +## Method + +- One positively owned tab, created via `PUT /json/new?about:blank` and + closed afterward. No existing tab was touched; no content was mutated. +- `Page.navigate` cold load — the same mechanism the bridge uses + (`cdp_driver.py:1301`) — to the natural 314-node specimen conversation + (`6a8149d3`, web2api project; read-only). +- Sampling every ~150 ms from navigation: URL path, `readyState`, composer + presence, user/assistant counts, mounted `conversation-turn` ids, mounted + `data-message-id`s. After 8 s of unchanged count+identity, a 10 s extended + watch at 250 ms looked for late flaps and identity churn at constant + count. Five cold-load runs. +- Harness: `scripts/phase0_experiment_1a.py`; raw data: + `captures/phase0/exp1a-20260921-092848.json`. + +## Results + +All five runs showed the same shape (times from navigate): + +| Milestone | Range across runs | +| --- | --- | +| `readyState` → loading | ~0.97–1.08 s | +| `readyState` → interactive | ~2.55–2.66 s | +| `readyState` → complete (DOM still empty: 0 composer, 0 messages) | ~2.84–2.94 s | +| Atomic appearance: composer + full window (3 user + 3 assistant, 6 message ids) | ~8.91–9.55 s | +| Post-appearance identity churn (10 s extended watch) | 0 events in 5/5 runs | +| Intermediate count values before the final window | none in 5/5 runs | +| Sample errors | 0 | + +Run-to-run timing was tight (composer within a 0.64 s band), and the +ordering was deterministic: composer-ready and mounted-identity-stable +coincide, because they are the same React commit. + +## Findings + +1. **Atomic mount, not progressive mounting, on this path.** The hydration + commit brings the composer and the conversation tail window up together. + Under this model, "composer ready" implies "DOM baseline readable" — + the bridge's existing sequencing (composer readiness poll, then + `_read_assistant_count_baseline`) lands after the commit in practice. +2. **Sectioned loading re-confirmed from a second angle.** The stable + mounted window is 6 turns (3+3) against 314 backend mapping nodes. The + DOM is a tail window, and it is small even when loading is fully + complete. +3. **A ~6 s hydration dead zone exists after `readyState=complete`.** + Between ~2.9 s and ~9 s the document is complete but React has rendered + nothing. Any logic gating on `readyState` alone would act ~6 s before + content exists. The bridge is not exposed today because its readiness + polls the composer, but this explains the historical + `send_baseline_unavailable` failure class if a baseline read is ever + reached from a readyState-only gate. +4. **The mounted window is not a fixed size across sessions.** The 2026-09-20 + passive snapshot saw 5 mounted turns for this conversation; today's runs + saw 6, with no intervening sends. Window size varies; only the backend + graph is stable. (Consistent with the provisional invariant that DOM + counts are diagnostics, not authority.) + +## Scope and exclusions + +- One account, one Chrome build (153), one network condition, one + conversation, five runs, cold-load path only. +- Not covered: memory-saver/discarded-tab revival, heavy CPU or network + stress during hydration, conversations whose mounted window is larger + than 6 turns (progressive rendering may appear with bigger windows), and + any non-`Page.navigate` navigation path. The bridge's + `navigate_conversation` uses `Page.navigate`, so the tested path is the + bridge's real path shape — but revival and stress remain open. +- Per the authorization: this experiment made no writes. Experiment 1B + (one normal controlled send in a disposable project) is the next + authorized step and had not been executed at the time of writing. diff --git a/docs/research/experiment-1b-send-timeline.md b/docs/research/experiment-1b-send-timeline.md new file mode 100644 index 00000000..ee7ffafa --- /dev/null +++ b/docs/research/experiment-1b-send-timeline.md @@ -0,0 +1,112 @@ +# Experiment 1B — Normal-Send Causal Timeline (2026-09-21) + +Controlled mutation experiment under the Phase 0 authorization: exactly one +sentinel turn (`PHASE0-1B-d69d36b8`) through the real bridge pipeline of an +isolated second instance, instrumented end-to-end. Outcome class: **A — +outgoing-UUID evidence is the earliest causal acknowledgment signal, ahead +of every DOM signal.** + +## Setup + +- Experiment bridge: second instance, REST 8081, config + `captures/phase0/config-1b.json` (owned tabs, parallel per-target locks, + pool disabled). Live bridge on 8080 untouched throughout. +- Browser: **shared Chrome/CDP 9222 with the live bridge** — recorded + environmental limitation (the 9223 listener is Edge/MSN, not an + authenticated ChatGPT Chrome; an independent profile cannot be logged in + without user credentials). Observers attached only to the experiment + bridge's owned tab (`34372C4A…`, diffed against a pre-start snapshot). +- Disposable project: `Web2API Phase0 Controlled Recon` + (`g-p-6ab0df421d288191b97e2781eb7e16a1`, memory_scope `project_v2`). +- Code exercised: working tree `feat/sse-pool-status-endpoint` (master + + `/health` telemetry) — not literally `master`. +- Instrumentation: two CDP observer sessions (Network recorder; DOM sampler + at ~100 ms), bridge log (ms-precision), post-hoc authed projection fetch. + Harness: `scripts/phase0_experiment_1b.py`; raw: + `captures/phase0/exp1b-20260921-104735.json`; bridge log: + `captures/phase0/exp1b-bridge.log`. +- No failure injected; model/effort not changed (observed: effort chip + "High", runtime slug `gpt-5-6-thinking`, reply "ACK", REST 200 in + 15.4 s). + +## Timeline (offsets from the send mutation: POST `/f/conversation`) + +| Offset | Event | +| ---: | --- | +| −8460 ms | REST request fired at bridge (includes navigation) | +| −2180 ms | speculative POST `/conversation/init` (project-page conversation shell, before typing) | +| −578 ms | POST `/f/conversation/prepare` | +| −530 ms | composer filled (first sample >50 chars) | +| −55 ms | typing completed (bridge log) | +| **0 ms** | **POST `/backend-api/f/conversation` — message dispatch** | +| +139 ms | IdentityListener captures outgoing user UUID | +| +141 ms | click `Runtime.evaluate` returns ("Message sent") | +| −39 ms | backend user-node `create_time` (server clock; skew ≈ −40 ms, not the ~+5.1 s the June `SKEW_TOLERANCE` note assumed) | +| +437 ms | composer cleared; stop-button appears | +| +437 ms | DOM user message + `data-message-id`; DOM assistant message (same 100 ms tick) | +| +758 ms | bridge detector: "Assistant message appeared" | +| +1879 ms | POST `/sentinel/req` | +| +3830 ms | `data-message-model-slug` = `gpt-5-6-thinking` visible | +| +3832 ms | conversation id resolved | +| +6910 ms | REST response returned | + +## Evidence table + +| Evidence | Observed? | Offset | What it proves | +| --- | --- | ---: | --- | +| JS click returned | yes | +141 ms | local JS execution returned | +| Composer cleared | yes | +437 ms | UI reacted | +| DOM user count +1 | yes | +437 ms | mounted DOM changed | +| Outgoing UUID captured | yes | **+139 ms** | submission carried causal identity | +| Backend contains UUID | yes | post-hoc | remote user node exists — **node id == captured UUID** | +| Assistant generation started | yes | +437/+758 ms | model turn began (stop-button; detector) | + +The four previously conflated claims separate cleanly here: CDP command +executed (click returned, +141), React accepted the submit (composer +cleared, +437), backend created the USER turn (UUID in projection), +assistant generation started (stop/detector, +437–758). + +## Key contract findings + +1. **The send path is now `/backend-api/f/conversation` (+ + `/f/conversation/prepare`), wrapped in a live sentinel flow** + (`chat-requirements/prepare` → `ping` → `finalize` → `sentinel/req`). + The plain conversation POST is not what carries the message. + `/conversation/init` fires after the send (+3.66 s), and a speculative + shell init precedes typing. The July-era IdentityListener still captured + the client-message UUID from this path unchanged. +2. **Identity chain is exact**: outgoing client-message-id == backend + mapping node id == message id (`b2de19c3-f641-47cd-a4f8-1d7b2ecc473b`). + UUID capture (+139 ms) precedes composer clearing and DOM-count change + (+437 ms) by ~300 ms — the A-outcome. IdentityListener is the strongest + *and* earliest acknowledgment signal; DOM count is strictly slower. +3. **Backend clock skew ≈ −40 ms** on this run — the June-era + `~+5.1 s lead` assumption baked into `turn_anchor.py`'s degraded-mode + freshness floor is stale and should be re-derived (it is env-overridable + but the default encodes the old measurement). +4. **Generation-state classes absent**: `.result-thinking` and + `.result-streaming` never appeared at 100 ms sampling during a + `gpt-5-6-thinking` generation. Generation was observable via stop-button, + assistant-node appearance, and `data-message-model-slug` (+3.8 s) instead. + One controlled run; the generation-lifecycle experiment remains queued, + now with a concrete negative data point. +5. **User and assistant DOM nodes appear in the same 100 ms tick** (+437 ms) + — the assistant placeholder mounts with the user turn, before any text. + +## Scope and exclusions + +- One run, one sentinel, shared Chrome (timing co-tenancy), single + account/plan, working tree ≠ master, 100 ms DOM sampling (sub-100 ms + class toggles could be missed — though stop-button and text were caught). +- No failure injection (by design — ambiguity tests come later, offline + first). Nothing here tests reconnect/rate-limit replay. +- The disposable project now contains one conversation (`6ab0dff4…`) with + the sentinel turn; retained for the remaining controlled experiments. + +## Consequence for #51 + +The A-outcome supports the recorded design direction directly: primary +acknowledgment = outgoing UUID captured → backend UUID reconciliation; +`SEND_CONFIRMED` on match; DOM count/composer-clearing demoted to +diagnostics. The deterministic transport replay fixtures remain the next +step before implementation. diff --git a/docs/research/issue51-no-replay-implementation.md b/docs/research/issue51-no-replay-implementation.md new file mode 100644 index 00000000..42589033 --- /dev/null +++ b/docs/research/issue51-no-replay-implementation.md @@ -0,0 +1,216 @@ +# Issue #51 — No-Replay Send Mutations: Red Tests, Implementation, Verification (2026-09-21) + +Branch: `fix/51-no-replay-send-mutations` (off `master` = `1711491`, which +contains `497527d`). + +## Defect evidence (red tests, preserved) + +- `captures/phase0/51-red-evidence-part1.txt` (local, gitignored) — the + contract test fails against current master: no mutation primitive exists. +- `captures/phase0/51-red-evidence-part2-replay-characterization.txt` + (local, gitignored) — the mechanical proof: through `click_send`'s real + path (plain `_js` → `_cdp`), an ambiguous socket death sends the mutating + `Runtime.evaluate` **twice** (frame ids [1, 2]) after one reconnect. The + duplicate-send hazard is real and mechanically reproducible offline. The + reproduction logic is preserved as the (now-green) contract tests in + `tests/test_no_replay_send.py`, so the defect stays demonstrable in-repo. + +Outer replay: `retry_on_rate_limit` re-invokes `factory()` fresh on +`RateLimitError` (default 3 attempts), and both the REST non-stream chat +path (`api_server.py`) and the MCP chat tools (`mcp_server.py`, two +dispatch sites) wrapped their full mutating operation in it. + +## Implementation (the narrow seam) + +1. `cdp_transport.CDPTransport._js_mutation(expr)` — mutating + `Runtime.evaluate` sent with `_retry=False`: never reconnected-and-replayed. + Reconnect-class send failures and post-dispatch response timeouts convert + to `SendOutcomeUnknownError`; other errors propagate unchanged. + `_js` / `_js_strict` remain replayable observation primitives. +2. `cdp_driver.CDPDriver._js_mutation` — explicit driver-level delegation + (observation `_js` vs effect `_js_mutation` reads correctly at call sites). +3. `SendOutcomeUnknownError` (cdp_driver) — carries `code`, + `stage`, `effect_state="unknown"`, `captured_user_id`, `retry_safe=False`, + `reconciliation_safe=True`, so surfaces can decide without parsing text. +4. `chatgpt_dom.click_send` — the click now dispatches through + `_js_mutation`; the readiness poll stays on `_js`. +5. Breaker boundary moved: `click_send` no longer records + `COMPOSER_SEND_READINESS` success (JS return proves only that synthetic + events ran — 1B: click returned +141 ms, React acceptance +437 ms). + `send_and_stream` records success at submission evidence: captured UUID + (primary), or the DOM acknowledgment fallback returning True. +6. `single_send` opt-in: `resilience.chat_retry_attempts(args)` (1 vs 3); + REST resolves it from body top-level or `metadata` (`_resolve_single_send`, + passed into `_full_response`); MCP reads it from tool arguments at both + chat dispatch sites. Declared in the `chat_completion` and `chat_with_gpt` + input schemas. **Scoped**: `_SINGLE_SEND_TOOLS` = + {chat_completion, chat_with_gpt} via `_chat_tool_attempts` — + `create_memory` (also a `_CHAT_TOOLS` member for rate-limit retry) does + NOT inherit send semantics; if memory mutations ever need at-most-once + behavior they get their own effect analysis. +7. UUID preservation on the ambiguous path: `send_and_stream` catches + `SendOutcomeUnknownError` from `click_send` and, before the `finally` + closes the capture scope and clears listener state, waits one bounded + window (`AMBIGUOUS_CAPTURE_WINDOW_S = 1.0`; 1B measured healthy capture + at +139 ms) for the in-flight IdentityListener capture and attaches it + as `exc.captured_user_id`. Evidence preservation only — the outcome + stays UNKNOWN; this is not reconciliation and never resends. +8. Surface mappings preserve the UNKNOWN distinction (pre-push review): + REST non-stream maps `SendOutcomeUnknownError` to **HTTP 409** with + `code=send_outcome_unknown`, `retry_safe=false`, and + `captured_user_id` when preserved — not a bare 500. REST streaming + (status locked at 200 once SSE starts) emits an inline + `[Error: send_outcome_unknown …]` marker chunk with the same evidence, + mirroring the established rate-limit marker precedent. MCP + `_map_tool_exception` returns a structured isError result carrying the + code, the UUID when preserved, and do-not-resend guidance. + **Independent-review fix (82a37d7)**: the 409 also sends + `x-should-retry: false` — the official OpenAI SDKs check that header + BEFORE their retryable-status rules (409 is otherwise auto-retried by + default), so without it the SDK layer re-enabled the replay this patch + removes. Pinned by test alongside `Retry-After` absence. Review also + tightened the transport test's no-reconnect assertion (explicit + counter, replacing a tautological socket-identity check) and fixed the + sync `_assert_owned_tab_required` seam mock. + + Independent-review fixes continued (0a5dd0d, f94f02e): the committed + recon harness now reproduces the report's authenticated projection + method (in-page Bearer token; the committed script had been + cookie-only, which the report itself records as masked-404), data + injection uses the IIFE-parameter pattern instead of top-level + `var __D` (page-global collision, a known bug class), exp1b sets its + completion event on the event-loop thread (asyncio.Event is not + thread-safe), and the lint gate went deterministic: + `known-first-party = ["chatgpt_web2api"]` (the local-0.15/CI-0.16 I001 + disagreement was classification drift, not rule drift) plus the + inherited baseline cleared so the lint job — and the build job it + gates — can pass. + +## Automated-review round 2 (2026-09-21, commits 54d5593 + 9f52fe6) + +New Codex/GitWire findings after the live certification; PR returned to +draft while they were addressed. + +- **Codex P1 (blocker) — cancellation during the mutating evaluate.** + CancelledError escaped unclassified although the frame may already have + reached Chrome. Deliberate design: CancelledError propagates UNCHANGED + (converting it would break the asyncio cancellation contract and turn + client-disconnect cancellations into 409 responses); the + UNKNOWN/do-not-resend classification lands on a driver side channel — + `note_send_outcome_unknown()` sets `last_send_outcome_unknown` (cleared + at each send start) and logs `code=send_outcome_unknown` at ERROR. No + evidence-window wait on this path (cancellation must not be delayed). + Two tests: cancellation-during-mutation (propagates + records + 1 frame + + 0 reconnects) and the scope check (cancellation during a read records + nothing). +- **Codex P2 — breaker recovery on inconclusive evidence.** With no UUID + and `acknowledged=None`, a completed turn never cleared + `COMPOSER_SEND_READINESS` failures. Downstream-success fallback added + after stream completion only. Three tests (fires exactly once on + completion; no double-record with UUID; failed stream records nothing). +- **GitWire streaming claim — not valid, but pinned.** `_stream_response` + retries only the read-only preflight and consumes `send_and_stream` + once. Two regression tests, including the disproving property (mid-stream + RateLimitError does not restart the generator). Evidence reply posted on + the PR (issuecomment-5767819420). +- **Harness defects (commit 9f52fe6)**: `Network.getResponseBody` now + rides the observer's own CDP session (the wrong-session bug is why the + live run could not resolve `conv_id` in-band); `reconnect_calls` now + counts `driver.reconnect` directly (socket-creation total kept as + explicitly informational); live-cert cleanup closes ONLY the positively + owned `_target_id`; exp1b derives its owned tab from the bridge log + instead of the non-committed snapshot file. Not acted on, per review + disposition: the ≤3.10 TimeoutError note (project requires ≥3.11), the + latch/nonce concerns (no such mechanisms exist here), and committed-ID + hygiene in research docs (acknowledged low-severity debt). +- CI on `9f52fe6`: build, lint, secret-scan, and all six test jobs PASS. + Local suite 720 passed / 1 pre-existing master failure; ruff clean. + The prior live certification remains valid for the certified path; the + new cancellation and breaker-fallback paths are deterministically tested + only. + +## Round-2 follow-ups (current-head pass; commits 546c345 + 78b8ed2) + +- **CDP pending-entry cancellation leak** (independently confirmed, red + proven before the fix): `_cdp` inserted `d._pending[mid] = fut`, but + `CancelledError` bypassed both exception handlers, so cancelling during + the mutating evaluate or any response wait left the entry behind + indefinitely on a dead/hanging socket. Not a replay hazard — a state + leak on the newly supported cancellation path. Both await points now + pop the entry and re-raise unchanged; both cancellation tests assert + `driver._pending == {}` afterward (verified failing against the previous + code first). +- **Experiment 1B early-stop** (GitWire current-head finding, valid): + `stop_at_holder["until"]` was evaluated and passed as a plain float, so + the controller's later update could not affect the running recorder + loops — they ran to the full deadline instead of stopping after the + post-REST grace (which also explains the original run's ~5-minute + duration). Now a shared `asyncio.Event` stops both recorders, with the + absolute deadline kept as a backstop. +- **Evidence provenance**: both harnesses derive working-tree provenance + from git at run time (`_git_describe()` → branch@head + dirty state). + Previously `phase0_live_cert_51.py` hard-coded `code_head: 162c80c …` + and exp1b a hard-coded tree description, so reruns at newer heads would + have produced evidence falsely labeled as an older tree. +- PR description refreshed (720-pass round, review-rounds section with + per-commit provenance). CI on `78b8ed2`: all gates PASS. Still pending: + fresh automated review of this head; single_send default decision, + ready-for-review, and #51 closure remain downstream. + +## Claim boundary (for the PR description) + +Claim: **the bridge now provides an opt-in at-most-one automatic USER-send +mutation attempt across both transport reconnect ambiguity and outer +rate-limit retry; ambiguous outcomes surface explicitly rather than being +replayed.** Do NOT claim exactly-once delivery or automatic reconciliation +of ambiguous sends — UNKNOWN→CONFIRMED backend lookup remains future work. + +Default-mode status, stated as legacy rather than safe: with +`single_send` absent (the default), the transport fix applies (the send +click is never replayed), but the outer wrapper retains its legacy +three-attempt budget — a `RateLimitError` surfacing after a dispatched +send may therefore re-run the whole turn, including the USER send. That +behavior is unchanged from master and is precisely the policy decision +(`single_send` opt-in vs default) that remains open after certification. + +## Verification + +- `tests/test_no_replay_send.py` — the four contract tests: mutation not + replayed (1 frame, 0 reconnects, `SendOutcomeUnknownError`), read retry + preserved (2 frames, 1 reconnect), single-shot wrapper semantics, default + backward compatibility. 4/4 green. +- `tests/test_single_send_propagation.py` — seam tests for the helper, REST + body/metadata resolution, MCP arguments mapping, and wrapper combinations + (one invocation on post-mutation 429 with opt-in; default still retries). + 8/8 green. +- `tests/test_send_outcome_unknown.py` — drives the real `send_and_stream` + orchestration with mocked collaborators: (A) an ambiguous click preserves + the in-flight captured UUID on the error (bounded window, scope closed + after) and records no breaker success; no-capture leaves the field None; + (B) breaker success records exactly once on captured UUID (DOM probe + skipped) and exactly once on the DOM-ack fallback; no success on UNKNOWN; + (C) `single_send` honored by chat_completion/chat_with_gpt only; + create_memory keeps the default budget even with the flag present. + 6/6 green. +- Updated to the new contract: `test_chatgpt_dom.py` + (click records NO breaker success; poll/click split across `_js` / + `_js_mutation`), `test_send_readiness.py`, `test_composer_selectors.py`. +- Full suite: **709 passed, 1 failed** — the failure + (`test_parallel_tabs_pr4.py::test_config_default_parallel_tabs_false`) + fails identically on clean master (verified via `git stash`); pre-existing, + not addressed here. +- `ruff check src tests`: 10 errors — one BELOW the 11-error master + baseline (an unsorted import block in a new test file was fixed). Zero + new. + +## Deliberately not done (gaps, stated) + +- The `EFFECT_UNKNOWN → CONFIRMED` reconciliation upgrade (backend lookup by + exact captured UUID, per 1B's UUID == backend node id) is designed, not + implemented. The error now carries `captured_user_id` from the preserved + in-flight capture, so the lookup has its key when capture succeeded. +- `type_message` still dispatches through replayable `_js` (a replayed type + re-sets composer text; narrower blast radius than a replayed click, and + #51's scope was the send mutation). Deliberate scope boundary. +- No live ambiguous-write certification yet — gated behind commit + PR + review, per the agreed sequence. diff --git a/docs/research/live-cert-51-ambiguous-send.md b/docs/research/live-cert-51-ambiguous-send.md new file mode 100644 index 00000000..0ea1820a --- /dev/null +++ b/docs/research/live-cert-51-ambiguous-send.md @@ -0,0 +1,73 @@ +# Live Ambiguous-Write Certification — PR #52 / Issue #51 (2026-09-21) + +One instrumented injection, executed under the authorized protocol after +delta re-review passed. This is the executed-but-response-lost branch, +live, against the real ChatGPT Web account. + +## Protocol + +- Live bridge (8080) untouched throughout; experiment bridge ran the PR + #52 tree (`162c80c` = `346411c` + comment-only fix) in-process on 8081. +- Chrome 9222 shared, strict owned-tab isolation (the experiment tab was + the driver's own owned tab, closed on shutdown: log-verified). +- Disposable project `Web2API Phase0 Controlled Recon`; non-stream REST; + `single_send=true`; sentinel `PHASE0-LC51-3ac2ae77`. +- Fault injection per protocol — the socket was NOT killed: the + experiment instance's websocket `send()` was wrapped so the ONE + send-click `Runtime.evaluate` frame was **forwarded to real Chrome** + (the mutation executed remotely) and the local caller then raised a + reconnect-class `ConnectionClosedError`. `recv()` always delegated + normally, so the IdentityListener kept working on the live socket. +- Harness: `scripts/phase0_live_cert_51.py`; raw counters: + `captures/phase0/live-cert-51-20260921-181705.json` (+ post-hoc + backend confirmation merged); run log: + `captures/phase0/live-cert-51-run3.log`. + +## Result — every counter at the strongest-success target + +| Counter | Target | Observed | +| --- | --- | --- | +| mutating `Runtime.evaluate` frames | 1 | **1** | +| driver reconnect calls | 0 | **0** (log-verified: no reconnect line; the 4 total `websockets.connect` calls were the driver's page socket, browser-level startup sockets, and the harness observer) | +| `/backend-api/f/conversation` POSTs | 1 | **1** | +| REST result | 409 `send_outcome_unknown` | **409 `send_outcome_unknown`** (8.5 s) | +| `x-should-retry` header | false | **false** | +| captured UUID | UUID X | **`8f471fb2-e658-4313-b8c7-26ee966a19f5`** (listener capture at +~200 ms after the fault; preserved onto the error before scope close; identical in the 409 body) | +| backend USER nodes containing sentinel | 1 | **1** (conversation "Live ambiguity certification", generation ran to completion ~14 s after the 409 — the remote effect was real) | +| backend node/message id vs `captured_user_id` | equal | **equal — both are `8f471fb2-e658-4313-b8c7-26ee966a19f5`** | +| automatic resend | 0 | **0** | + +## What this certifies + +The exact scenario #51 exists for, live: + +```text +remote effect occurred (turn created, generation completed) + + +local command outcome was ambiguous (fault injected after delivery) + ↓ +UNKNOWN surfaced (409, x-should-retry:false, retry_safe=false) + ↓ +evidence preserved (captured_user_id == backend node id == message id) + ↓ +NO replay (one mutation frame, zero reconnects, one submission POST) +``` + +Per the authorizing review: this is the strongest possible live +certification short of implementing automatic UNKNOWN→CONFIRMED +reconciliation — the backend lookup key on the error is proven exact +against the live backend. + +## Scope and notes + +- One run, one sentinel, single account, shared Chrome (recorded + limitation). Repetition belongs to later compatibility certification. +- Two launcher attempts were discarded before the successful one (a + buffered-output run killed before any request; a run blocked on + `service.start()`'s shutdown wait — launcher bugs, not bridge bugs). + The killed first attempt's registered owned tab was reclaimed by the + successful run and closed with it; no user tab was touched at any point. +- `backend_node_ids: []` in the frozen JSON's original counters is + superseded by the merged `posthoc_backend_confirmation` block (the + in-run lookup could not resolve the conversation id; the post-hoc + lookup did). diff --git a/pyproject.toml b/pyproject.toml index 542cb73a..5f19a3b5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,6 +77,13 @@ ignore = ["E501", "E402"] # conftest redefines event_loop for pytest-asyncio — intentional fixture override. "tests/conftest.py" = ["F811"] +[tool.ruff.lint.isort] +# Deterministic import classification for every environment. Without this, +# ruff infers first-party from whether the src-layout package happens to be +# installed, so local runs and the CI lint job disagreed on I001 (PR #52 +# review: local ruff 0.15.16 clean, CI ruff 0.16.8 failing). +known-first-party = ["chatgpt_web2api"] + [tool.pytest.ini_options] asyncio_mode = "auto" markers = [ diff --git a/scripts/phase0_experiment_1a.py b/scripts/phase0_experiment_1a.py new file mode 100644 index 00000000..5b29fa67 --- /dev/null +++ b/scripts/phase0_experiment_1a.py @@ -0,0 +1,303 @@ +"""Phase 0 Experiment 1A — long-chat mounting race (authorized controlled read). + +Measures, on a positively OWNED tab (created by this script, closed by this +script; the user's 26 tabs are never touched), whether ChatGPT's composer +becomes ready BEFORE the mounted DOM of a 314-node sectioned conversation is +count-stable and identity-stable — the ordering the bridge's send baseline +(`_read_assistant_count_baseline`) implicitly assumes. + +Safety contract: + - Creates ONE new tab via PUT /json/new?about:blank and operates only on + that target id. Never adopts, navigates, or closes any other tab. + - Read/navigation only. NO sends, NO clicks, NO typing, no DOM mutation. + - Navigates the owned tab to the natural 314-node specimen conversation + (web2api project). Read-only evidence; the conversation is not modified. + - Closes the owned tab in a finally block. + +Per run (5 cold-load runs): Page.navigate (same mechanism as +cdp_driver.py:1301), then sample every ~150 ms: url path, readyState, +composer presence, user/assistant counts, mounted conversation-turn ids, +mounted data-message-ids. After 8 s of unchanged count+identity, an +extended 10 s watch at 250 ms catches late flaps (the virtualization case: +identity churn with constant count). + +Output: captures/phase0/exp1a-run.json + stdout analysis. +""" +from __future__ import annotations + +import asyncio +import datetime as _dt +import json +import pathlib +import time +import urllib.request + +import websockets + +CDP = "http://127.0.0.1:9222" +CONV_URL = ("https://chatgpt.com/g/g-p-6a3a9ddf947481918dcd9f9bbc80017e-" + "chatgpt-web2api/c/6a8149d3-3f8c-83eb-89ec-27ed24f8b259") +RUNS = 5 +SAMPLE_MS = 150 +STABLE_MS = 8000 # count+identity unchanged this long → "stable" +EXTENDED_WATCH_MS = 10000 +EXTENDED_SAMPLE_MS = 250 +RUN_CAP_MS = 75000 +OUT_DIR = pathlib.Path("captures/phase0") + +SAMPLER_JS = """ +(function() { + var comp = document.querySelector('#prompt-textarea'); + var turns = document.querySelectorAll('[data-testid^="conversation-turn"]'); + var tids = []; + for (var i = 0; i < turns.length; i++) tids.push(turns[i].getAttribute('data-testid')); + var msgs = document.querySelectorAll('[data-message-author-role]'); + var mids = []; + for (var j = 0; j < msgs.length; j++) { + var m = msgs[j].getAttribute('data-message-id'); + mids.push(m ? m.slice(0, 8) : '-'); + } + return { + path: location.pathname.slice(-24), + rs: document.readyState, + comp: !!comp, + u: document.querySelectorAll('[data-message-author-role="user"]').length, + a: document.querySelectorAll('[data-message-author-role="assistant"]').length, + tn: tids.join(','), + m: mids.join(',') + }; +})() +""" + + +def http_json(path: str, method: str = "GET") -> dict: + req = urllib.request.Request(CDP + path, method=method) + return json.loads(urllib.request.urlopen(req, timeout=8).read()) + + +class OwnedTab: + """The one tab this script owns; every command goes to this target only.""" + + def __init__(self) -> None: + created = http_json("/json/new?about:blank", method="PUT") + self.target_id: str = created["id"] + self._ws_url: str = created["webSocketDebuggerUrl"] + self._ws = None + self._next_id = 1 + print(f"owned tab created: target={self.target_id[:12]}…") + + async def connect(self) -> None: + self._ws = await websockets.connect(self._ws_url, max_size=10**7, open_timeout=5) + + async def reconnect(self) -> None: + try: + if self._ws: + await self._ws.close() + except Exception: + pass + for t in http_json("/json/list"): + if t.get("id") == self.target_id and t.get("webSocketDebuggerUrl"): + self._ws_url = t["webSocketDebuggerUrl"] + break + await self.connect() + + async def call(self, method: str, params: dict | None = None, timeout: float = 5) -> dict: + mid = self._next_id + self._next_id += 1 + msg = {"id": mid, "method": method, "params": params or {}} + for attempt in (1, 2): + try: + await self._ws.send(json.dumps(msg)) + while True: + raw = await asyncio.wait_for(self._ws.recv(), timeout=timeout) + got = json.loads(raw) + if got.get("id") == mid: + return got.get("result", {}) + except Exception: + if attempt == 2: + raise + await asyncio.sleep(0.4) + await self.reconnect() + raise RuntimeError("unreachable") + + async def eval(self, expr: str, timeout: float = 4) -> dict: + res = await self.call("Runtime.evaluate", + {"expression": expr, "returnByValue": True, + "silent": True, "awaitPromise": False}, timeout) + r = res.get("result", {}) + if r.get("subtype") == "error": + return {"__err": (r.get("description") or "js error")[:120]} + return r.get("value") if r.get("type") != "undefined" else {"__err": "undefined"} + + async def close(self) -> None: + try: + if self._ws: + await self._ws.close() + except Exception: + pass + try: + req = urllib.request.Request(f"{CDP}/json/close/{self.target_id}") + body = urllib.request.urlopen(req, timeout=8).read().decode(errors="replace") + print(f"owned tab closed ({body.strip()[:40]})") + except Exception as e: + print(f"WARN: tab close failed ({e}) — close target {self.target_id} manually") + + +def signature(s: dict) -> str: + return f"{s.get('u')}|{s.get('a')}|{s.get('m')}|{s.get('tn')}" + + +async def run_once(tab: OwnedTab, run_no: int) -> dict: + samples: list[dict] = [] + t0 = time.monotonic() + + async def sample() -> None: + t = round((time.monotonic() - t0) * 1000) + try: + v = await tab.eval(SAMPLER_JS) + if "__err" in v: + samples.append({"t": t, "err": v["__err"]}) + else: + samples.append({"t": t, **v}) + except Exception as e: + samples.append({"t": t, "err": str(e)[:80]}) + + await tab.call("Page.bringToFront") + await tab.call("Page.navigate", {"url": CONV_URL}) + nav_t = round((time.monotonic() - t0) * 1000) + + stable_since = None + last_sig = None + extended_until = None + while True: + elapsed = (time.monotonic() - t0) * 1000 + if elapsed > RUN_CAP_MS: + break + await sample() + loop_start = time.monotonic() + valid = [s for s in samples if "err" not in s] + if valid: + sig = signature(valid[-1]) + if extended_until is None: + if sig == last_sig and sig != "": + if stable_since is None: + stable_since = valid[-1]["t"] + if valid[-1]["t"] - stable_since >= STABLE_MS and valid[-1].get("comp"): + extended_until = valid[-1]["t"] + EXTENDED_WATCH_MS + print(f" run {run_no}: stable at {stable_since} ms " + f"(count+identity); extended watch until {extended_until} ms") + else: + stable_since = None + last_sig = sig + elif valid[-1]["t"] >= extended_until: + break + # sleep to the next tick + tick = EXTENDED_SAMPLE_MS if extended_until is not None else SAMPLE_MS + await asyncio.sleep(max(0.005, tick / 1000 - (time.monotonic() - loop_start))) + + return analyze(samples, run_no, nav_t) + + +def analyze(samples: list[dict], run_no: int, nav_t: int) -> dict: + valid = [s for s in samples if "err" not in s] + errors = [s for s in samples if "err" in s] + + def first(pred) -> int | None: + for s in valid: + if pred(s): + return s["t"] + return None + + composer_ready = first(lambda s: s.get("comp")) + counts = [(s["t"], s.get("u", 0) + s.get("a", 0)) for s in valid] + sigs = [(s["t"], signature(s)) for s in valid] + + def stable_from(series) -> int | None: + if not series: + return None + final = series[-1][1] + for idx, (t, v) in enumerate(series): + if v == final and all(w == final for _, w in series[idx:]): + return t + return None + + count_stable = stable_from(counts) + ident_stable = stable_from(sigs) + + # churn: identity changed while count constant (after composer ready) + churn = 0 + for i in range(1, len(valid)): + if valid[i - 1].get("comp") and valid[i].get("comp"): + if (valid[i].get("u"), valid[i].get("a")) == (valid[i - 1].get("u"), valid[i - 1].get("a")) \ + and signature(valid[i]) != signature(valid[i - 1]): + churn += 1 + + rs_timeline = [] + for s in valid: + if not rs_timeline or rs_timeline[-1][0] != s.get("rs"): + rs_timeline.append([s.get("rs"), s["t"]]) + + if composer_ready is None: + verdict = "NO-COMPOSER" + elif (count_stable or 10**9) > composer_ready or (ident_stable or 10**9) > composer_ready: + verdict = "RACE" + else: + verdict = "CLEAN" + + return { + "run": run_no, "nav_sent_at_ms": nav_t, "verdict": verdict, + "composer_ready_ms": composer_ready, "count_stable_ms": count_stable, + "identity_stable_ms": ident_stable, + "final_u": valid[-1].get("u") if valid else None, + "final_a": valid[-1].get("a") if valid else None, + "final_mounted_ids": valid[-1].get("m") if valid else None, + "identity_churn_with_constant_count": churn, + "sample_errors": len(errors), + "readyState_timeline": rs_timeline, + "samples": samples, + } + + +async def main() -> None: + OUT_DIR.mkdir(parents=True, exist_ok=True) + tab = OwnedTab() + results = [] + try: + await tab.connect() + for i in range(1, RUNS + 1): + print(f"run {i}: cold navigate → {CONV_URL[:80]}…") + r = await run_once(tab, i) + results.append(r) + print(f" run {i}: verdict={r['verdict']} composer_ready={r['composer_ready_ms']} ms " + f"count_stable={r['count_stable_ms']} ms identity_stable={r['identity_stable_ms']} ms " + f"churn={r['identity_churn_with_constant_count']} " + f"final(u/a)={r['final_u']}/{r['final_a']} err={r['sample_errors']}") + if i < RUNS: + await tab.call("Page.navigate", {"url": "about:blank"}) + await asyncio.sleep(2.5) + finally: + await tab.close() + + stamp = _dt.datetime.now().strftime("%Y%m%d-%H%M%S") + (OUT_DIR / f"exp1a-{stamp}.json").write_text( + json.dumps({"captured_at": stamp, "url": CONV_URL, "runs": results}, + indent=1), encoding="utf-8") + + races = sum(1 for r in results if r["verdict"] == "RACE") + churns = sum(1 for r in results if r["identity_churn_with_constant_count"] > 0) + print(f"\n==== EXPERIMENT 1A SUMMARY ({stamp}) ====") + print(f"runs={len(results)} RACE={races} CLEAN={len(results) - races} " + f"runs_with_identity_churn_at_constant_count={churns}") + for r in results: + print(f" run {r['run']}: composer@{r['composer_ready_ms']} count_stable@{r['count_stable_ms']} " + f"identity_stable@{r['identity_stable_ms']} → {r['verdict']}") + if races == 0 and churns == 0: + print("verdict: no post-readiness mounting change observed in THIS environment/" + "conversation/build/account — DOM-count ack not falsified here (scoped claim)") + else: + print("verdict: composer-ready does NOT imply mounted-DOM stability → " + "baseline-race hazard CONFIRMED at contract level") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/scripts/phase0_experiment_1b.py b/scripts/phase0_experiment_1b.py new file mode 100644 index 00000000..9a65b3c0 --- /dev/null +++ b/scripts/phase0_experiment_1b.py @@ -0,0 +1,380 @@ +"""Phase 0 Experiment 1B — normal-send causal timeline (authorized controlled mutation). + +Fires exactly ONE sentinel turn through the real bridge pipeline of the +isolated experiment instance (REST 8081 → send_and_stream → type → baseline +→ click → IdentityListener → detector → TurnAnchor), while two CDP observer +sessions attached to the experiment's OWNED tab record: + + - Network: every /backend-api/ + /api/auth request (method, url, wallTime) + - DOM @100ms: composer text length, user/assistant counts, mounted user + message ids, stop-button/thinking/streaming classes, last assistant + data-message-model-slug, effort chip label + +Safety contract: + - Observers only attach to the experiment bridge's owned tab (diffed + against captures/phase0/pre-1b-tabids.txt). No other tab is touched. + - ONE send, no failures injected, no model/effort change (recorded only). + - Live bridge on 8080 and the user's tabs are untouched. + +Outputs: captures/phase0/exp1b-.json + stdout summary. +""" +from __future__ import annotations + +import asyncio +import datetime as _dt +import json +import pathlib +import re +import secrets +import time +import urllib.request + +import websockets + +CDP_HTTP = "http://127.0.0.1:9222" +BRIDGE = "http://127.0.0.1:8081" +PROJECT_ID = "g-p-6ab0df421d288191b97e2781eb7e16a1" +OUT_DIR = pathlib.Path("captures/phase0") +SAMPLE_MS = 100 +GRACE_AFTER_REST_S = 12 +REST_TIMEOUT_S = 240 + +SAMPLER_JS = """ +(function() { + var c = document.querySelector('#prompt-textarea'); + var users = document.querySelectorAll('[data-message-author-role="user"]'); + var assists = document.querySelectorAll('[data-message-author-role="assistant"]'); + var uids = []; + for (var i = 0; i < users.length; i++) { + var m = users[i].getAttribute('data-message-id'); + if (m) uids.push(m.slice(0, 8)); + } + var lastA = assists[assists.length - 1]; + var chip = document.querySelector('button.__composer-pill'); + return { + comp: !!c, + compLen: c ? (c.textContent || '').length : 0, + u: users.length, a: assists.length, + uids: uids.join(','), + stop: !!document.querySelector('[data-testid="stop-button"]'), + think: !!document.querySelector('.result-thinking'), + stream: !!document.querySelector('.result-streaming'), + slug: lastA ? lastA.getAttribute('data-message-model-slug') : null, + effort: chip ? (chip.textContent || '').replace(/\\s+/g, ' ').trim().slice(0, 20) : null + }; +})() +""" + +# Parameterized (NOT self-called): the caller appends the JSON argument — +# IIFE-parameter injection, never a top-level var __D (ChatGPT's page owns a +# global __D; collision is a known bug class). Token fetched transiently +# in-page; never returned. +PROJECTION_FIND_JS = """ +(async (__D) => { + var s = await fetch('/api/auth/session', {credentials: 'include'}); + var tok = (await s.json()).accessToken; + var r = await fetch('/backend-api/conversation/' + __D.conv_id + '?offset=0&limit=200', + {credentials: 'include', headers: {'Authorization': 'Bearer ' + tok}}); + var j = await r.json().catch(function() { return {}; }); + var mapping = j.mapping || {}; + var found = null; var assistants = []; + for (var k in mapping) { + var n = mapping[k]; + var msg = n.message || {}; + var role = msg.author ? msg.author.role : null; + var text = ''; + var parts = (msg.content && msg.content.parts) || []; + for (var p = 0; p < parts.length; p++) { + if (typeof parts[p] === 'string') text += parts[p]; + } + if (role === 'user' && text.indexOf(__D.sentinel) !== -1 && !found) { + found = {node_id: k, id: msg.id || null, create_time: msg.create_time || null, + end_turn: msg.end_turn || null}; + } + if (role === 'assistant') { + assistants.push({node_id: k, recipient: msg.recipient || null, + model: (msg.metadata && (msg.metadata.model_slug || msg.metadata.default_model_slug)) || null, + create_time: msg.create_time || null, end_turn: msg.end_turn || null, + text_len: text.length}); + } + } + assistants.sort(function(x, y) { return (x.create_time||0) - (y.create_time||0); }); + return {status: r.status, sentinel_user_node: found, + assistants_tail: assistants.slice(-4), current_node: j.current_node || null}; +}) +""" + + +def tab_ws_for(target_id: str) -> str | None: + d = json.loads(urllib.request.urlopen(f"{CDP_HTTP}/json/list", timeout=5).read()) + for t in d: + if t.get("id") == target_id: + return t.get("webSocketDebuggerUrl") + return None + + +def find_experiment_tab() -> str: + """Positively identify the experiment bridge's owned tab (review fix). + + The bridge's own log names its owned tab — 'Created owned tab: ' or + 'Reclaimed owned tab from registry: ' — which is authoritative and + committed with the run. The previous snapshot-file diff could select a + wrong tab if the snapshot was stale or missing. + """ + import re as _re + + log = pathlib.Path("captures/phase0/exp1b-bridge.log") + owned = None + for line in log.read_text(encoding="utf-8", errors="replace").splitlines(): + m = _re.search( + r"(?:Created owned tab|Reclaimed owned tab from registry): ([0-9A-F]{32})", + line) + if m: + owned = m.group(1) + if owned is None: + raise SystemExit("owned tab id not found in captures/phase0/exp1b-bridge.log") + targets = json.loads( + urllib.request.urlopen(f"{CDP_HTTP}/json/list", timeout=5).read()) + for t in targets: + if t.get("id") == owned and "chatgpt.com" in t.get("url", "") \ + and t.get("webSocketDebuggerUrl"): + return t["id"] + raise SystemExit(f"log-derived owned tab {owned} not present in /json/list") + + +async def network_recorder(ws_url: str, events: list, stop_at: float, + stop_evt: asyncio.Event | None = None) -> None: + async with websockets.connect(ws_url, max_size=10**7, open_timeout=5) as ws: + await ws.send(json.dumps({"id": 1, "method": "Network.enable"})) + while time.monotonic() < stop_at and not (stop_evt and stop_evt.is_set()): + try: + raw = await asyncio.wait_for(ws.recv(), timeout=0.5) + except TimeoutError: + continue + m = json.loads(raw) + if m.get("method") == "Network.requestWillBeSent": + p = m["params"] + url = p.get("request", {}).get("url", "") + if "/backend-api/" in url or "/api/auth" in url: + events.append({"kind": "req", "mono": time.monotonic(), + "wall": p.get("wallTime"), + "method": p.get("request", {}).get("method"), + "url": url[:120], "rid": p.get("requestId")}) + elif m.get("method") == "Network.responseReceived": + p = m["params"] + url = p.get("response", {}).get("url", "") + if "/backend-api/" in url or "/api/auth" in url: + events.append({"kind": "resp", "mono": time.monotonic(), + "status": p.get("response", {}).get("status"), + "url": url[:120], "rid": p.get("requestId")}) + + +async def dom_sampler(ws_url: str, samples: list, stop_at: float, + stop_evt: asyncio.Event | None = None) -> None: + async with websockets.connect(ws_url, max_size=10**7, open_timeout=5) as ws: + mid = 500 + while time.monotonic() < stop_at and not (stop_evt and stop_evt.is_set()): + t0 = time.monotonic() + mid += 1 + try: + await ws.send(json.dumps({"id": mid, "method": "Runtime.evaluate", + "params": {"expression": SAMPLER_JS, + "silent": True, + "returnByValue": True}})) + while True: + raw = await asyncio.wait_for(ws.recv(), timeout=3) + m = json.loads(raw) + if m.get("id") == mid: + v = m.get("result", {}).get("result", {}).get("value") + if v is not None: + samples.append({"mono": t0, "wall": time.time(), **v}) + break + except Exception as e: + samples.append({"mono": t0, "wall": time.time(), "err": str(e)[:60]}) + await asyncio.sleep(max(0.005, SAMPLE_MS / 1000 - (time.monotonic() - t0))) + + +def fire_rest(sentinel: str) -> tuple[float, float, dict]: + body = json.dumps({ + "model": "auto", + "stream": False, + "project_id": PROJECT_ID, + "messages": [{"role": "user", + "content": f"{sentinel} — instrumentation turn. " + f"Reply with exactly: ACK"}], + }).encode() + req = urllib.request.Request( + f"{BRIDGE}/v1/chat/completions", data=body, + headers={"Content-Type": "application/json"}, method="POST") + t0 = time.monotonic() + wall0 = time.time() + try: + with urllib.request.urlopen(req, timeout=REST_TIMEOUT_S) as r: + payload = json.loads(r.read().decode(errors="replace")) + return t0, wall0, {"status": r.status, "body": payload} + except urllib.error.HTTPError as e: + return t0, wall0, {"status": e.code, + "body": e.read().decode(errors="replace")[:500]} + except Exception as e: + return t0, wall0, {"error": str(e)[:300]} + + +async def post_projection(ws_url: str, conv_id: str, sentinel: str) -> dict: + async with websockets.connect(ws_url, max_size=10**7, open_timeout=5) as ws: + # IIFE-parameter injection (no top-level var __D — see note above) + expr = PROJECTION_FIND_JS + "(" + json.dumps( + {"conv_id": conv_id, "sentinel": sentinel}) + ")" + await ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate", + "params": {"expression": expr, "silent": True, + "awaitPromise": True, + "returnByValue": True}})) + while True: + m = json.loads(await asyncio.wait_for(ws.recv(), timeout=25)) + if m.get("id") == 1: + return m.get("result", {}).get("result", {}).get("value") or {} + + +def _git_describe() -> str: + """Working-tree provenance for the evidence record — derived at RUN + time, never hard-coded (round-2 review: a hard-coded tree description + mislabels reruns at newer heads). Git is anchored at THIS repository + (derived from the script's own location, never the process CWD — + invoking the harness from some other repository must not record that + repo's branch/SHA, and running outside any repository must not produce + a misleading "detached@ (clean)"), every command must succeed + (check=True; genuine failure returns "git-unavailable"), and an empty + head is rejected. The dirty flag reflects TRACKED working-tree state + only (--untracked-files=no).""" + import subprocess + from pathlib import Path + + repo_root = Path(__file__).resolve().parents[1] + + def _run(*args: str) -> str: + return subprocess.run( + ["git", "-C", str(repo_root), *args], + capture_output=True, text=True, timeout=5, check=True, + ).stdout.strip() + + try: + head = _run("rev-parse", "--short", "HEAD") + if not head: + return "git-unavailable" + branch = _run("branch", "--show-current") + dirty = _run("status", "--porcelain", "--untracked-files=no") + return (f"{branch or 'detached'}@{head}" + + (" (dirty)" if dirty else " (clean)")) + except Exception: + return "git-unavailable" + + +async def main() -> None: + sentinel = f"PHASE0-1B-{secrets.token_hex(4)}" + tab = find_experiment_tab() + ws_url = tab_ws_for(tab) + print(f"experiment tab: {tab[:12]} sentinel: {sentinel}") + + t_start = time.monotonic() + rest_done = asyncio.Event() + rest_result: dict = {} + + def _fire() -> None: + # Runs on a WORKER thread via asyncio.to_thread. Must not touch + # asyncio primitives here — asyncio.Event is not thread-safe. The + # caller (delayed_fire) sets rest_done back on the event-loop + # thread after this returns. + t0, wall0, res = fire_rest(sentinel) + rest_result.update(res) + rest_result["t0"] = t0 + rest_result["wall0"] = wall0 + rest_result["t_done"] = time.monotonic() + rest_result["wall_done"] = time.time() + + net_events: list = [] + samples: list = [] + + async def delayed_fire() -> None: + await asyncio.sleep(2.0) + await asyncio.to_thread(_fire) + rest_done.set() # on the event-loop thread — see note in _fire + + observers_done = asyncio.Event() + + async def run_observers() -> None: + await rest_done.wait() + await asyncio.sleep(GRACE_AFTER_REST_S) + observers_done.set() + + # Early-stop mechanism (round-2 review / GitWire finding): a shared + # asyncio.Event. The old code passed stop_at_holder["until"] as a plain + # FLOAT to the recorders, so later mutating the dict could not affect + # the running loops — they ran to the full REST_TIMEOUT+60 deadline + # instead of stopping after GRACE_AFTER_REST_S. The absolute deadline + # remains as a backstop. + stop_evt = asyncio.Event() + stop_backstop = t_start + REST_TIMEOUT_S + 60 + + async def net_task() -> None: + await network_recorder(ws_url, net_events, stop_backstop, stop_evt) + async def dom_task() -> None: + await dom_sampler(ws_url, samples, stop_backstop, stop_evt) + + async def controller() -> None: + await observers_done.wait() + stop_evt.set() # actually stops the recorder loops + + await asyncio.gather( + asyncio.create_task(net_task()), + asyncio.create_task(dom_task()), + asyncio.create_task(delayed_fire()), + asyncio.create_task(run_observers()), + asyncio.create_task(controller()), + ) + + # conversation id: from network POST url? conv id appears in later GETs + conv_id = None + for e in net_events: + mm = re.search(r"/backend-api/conversation/([0-9a-f-]{36})", e.get("url", "")) + if mm: + conv_id = mm.group(1) + break + proj = {} + if conv_id: + try: + proj = await post_projection(ws_url, conv_id, sentinel) + except Exception as e: + proj = {"error": str(e)[:150]} + + out = { + "captured_at": _dt.datetime.now().isoformat(timespec="seconds"), + "sentinel": sentinel, "project_id": PROJECT_ID, "conv_id": conv_id, + "env": {"chrome": "shared 9222 with live bridge (recorded limitation)", + "bridge": f"8081 isolated instance, working tree {_git_describe()}"}, + "rest": {k: v for k, v in rest_result.items() if k != "t0"}, + "network": net_events, "dom_samples": samples, "projection": proj, + } + ts = _dt.datetime.now().strftime("%Y%m%d-%H%M%S") + path = OUT_DIR / f"exp1b-{ts}.json" + path.write_text(json.dumps(out, indent=1), encoding="utf-8") + print(f"wrote {path}") + + # compact summary + r = rest_result + print(f"REST status={r.get('status')} err={r.get('error')} " + f"latency={round((r.get('t_done', 0) - r.get('t0', 0)) * 1000)} ms") + content = "" + try: + content = r.get("body", {}).get("choices", [{}])[0] \ + .get("message", {}).get("content", "")[:80] + except Exception: + pass + print(f"reply: {content!r}") + print(f"conv_id: {conv_id}") + print(f"network events: {len(net_events)}, dom samples: {len(samples)}") + if proj.get("sentinel_user_node"): + print("backend sentinel node:", proj["sentinel_user_node"]) + print("assistant tail:", json.dumps(proj.get("assistants_tail", []))[:300]) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/scripts/phase0_live_cert_51.py b/scripts/phase0_live_cert_51.py new file mode 100644 index 00000000..1f440dba --- /dev/null +++ b/scripts/phase0_live_cert_51.py @@ -0,0 +1,480 @@ +"""Live ambiguous-write certification for PR #52 / issue #51 (authorized). + +Protocol (as authorized 2026-09-21): ONE instrumented injection, same +isolation model as Experiment 1B — live bridge 8080 untouched; this script +runs the experiment bridge IN-PROCESS on 8081 from the PR #52 working tree; +Chrome 9222 shared with strict owned-tab isolation; disposable project; +non-stream REST with single_send=true. + +Fault injection (per protocol — the socket is NOT killed): + The experiment instance's websocket send() is wrapped. For the ONE + send-click Runtime.evaluate (expression carrying dispatchEvent + + MouseEvent), the frame is forwarded to REAL Chrome (the mutation may + execute remotely), then the local caller sees a reconnect-class + ConnectionClosedError — the exact "executed-but-ambiguous" branch. + recv() always delegates normally, so the IdentityListener keeps + capturing on the same live socket. + +Counters recorded (the certification table): + mutating_Runtime_evaluate_frames, websocket connections created + (= reconnect proxy), /backend-api/f/conversation POSTs, captured UUIDs + (bridge log + REST body), REST status, x-should-retry header, backend + USER nodes containing the sentinel, backend node id vs captured_user_id. + +Output: captures/phase0/live-cert-51-.json + stdout table. +""" +from __future__ import annotations + +import asyncio +import datetime as _dt +import json +import pathlib +import secrets +import time +import urllib.error +import urllib.request + +import websockets + +CDP_HTTP = "http://127.0.0.1:9222" +BRIDGE = "http://127.0.0.1:8081" +CONFIG = "captures/phase0/config-1b.json" +OUT_DIR = pathlib.Path("captures/phase0") + +SENTINEL = f"PHASE0-LC51-{secrets.token_hex(4)}" + +STATE = { + "mutating_frames": 0, # send-click Runtime.evaluate frames sent + "injected": False, + "ws_connections": 0, # every websockets.connect == connect + reconnects + "all_frames": 0, + "f_conversation_posts": 0, # from the tab observer +} + + +class ConnectionClosedError(Exception): + """Local stand-in; the transport's _should_reconnect classifier matches + this CLASS NAME, so the mutation path sees a reconnect-class failure — + while the real socket underneath stays healthy (the protocol's point).""" + + +class SendFaultWrapper: + """Wraps a real websockets connection. Passes everything through except + send(), which — exactly once, on the send-click mutation frame — + forwards the frame to Chrome and then raises ConnectionClosedError.""" + + def __init__(self, real): + self._real = real + + async def send(self, data): + is_mutation = False + try: + frame = json.loads(data) + expr = (frame.get("params") or {}).get("expression", "") + if (frame.get("method") == "Runtime.evaluate" + and "dispatchEvent" in expr and "MouseEvent" in expr): + is_mutation = True + except Exception: + pass + STATE["all_frames"] += 1 + if is_mutation: + STATE["mutating_frames"] += 1 + # ALWAYS forward to real Chrome first — the remote may execute it. + await self._real.send(data) + if is_mutation and not STATE["injected"]: + STATE["injected"] = True + raise ConnectionClosedError( + "connection closed (injected ambiguity: frame was delivered)") + + async def recv(self): + return await self._real.recv() + + async def close(self, *a, **kw): + return await self._real.close(*a, **kw) + + def __getattr__(self, name): + return getattr(self._real, name) + + +async def _patched_connect(*args, **kwargs): + real_ws = await _REAL_CONNECT(*args, **kwargs) + STATE["ws_connections"] += 1 + return SendFaultWrapper(real_ws) + + +class _PatchedConnectFactory: + """websockets.connect returns an object usable as BOTH `await connect()` + and `async with connect()`. This factory preserves that dual protocol: + awaiting or entering returns the fault-wrapped real connection.""" + + def __init__(self, args, kwargs): + self._args, self._kwargs = args, kwargs + self._ws = None + + async def _connect(self): + real_ws = await _REAL_CONNECT(*self._args, **self._kwargs) + STATE["ws_connections"] += 1 + self._ws = SendFaultWrapper(real_ws) + return self._ws + + def __await__(self): + return self._connect().__await__() + + async def __aenter__(self): + return await self._connect() + + async def __aexit__(self, *exc): + if self._ws is not None: + await self._ws.close() + + +_REAL_CONNECT = websockets.connect +websockets.connect = lambda *a, **kw: _PatchedConnectFactory(a, kw) +# covers cdp_driver connect() AND reconnect(); the browser-level +# `async with websockets.connect(...)` in _browser_cdp keeps working. + + +def tab_ids(): + d = json.loads(urllib.request.urlopen(f"{CDP_HTTP}/json/list", timeout=5).read()) + return {t["id"]: t for t in d if t.get("type") == "page"} + + +class TabObserver: + """Network observer bound to ONE CDP session on the experiment tab. + + Review fix: Network.getResponseBody must be issued by the SAME session + that enabled Network and tracked the request — a fresh connection has + neither. Commands therefore go through this observer's own live socket + (single-reader loop routes responses to caller futures). + """ + + def __init__(self, ws_url: str): + self._ws_url = ws_url + self._ws = None + self._cmds: dict[int, asyncio.Future] = {} + self._next_cmd = 1000 + self.f_conversation_posts = 0 + self.resp_events: list[dict] = [] + + async def run(self, stop_at: float) -> None: + async with websockets.connect(self._ws_url, max_size=10**7, + open_timeout=5) as ws: + self._ws = ws + await ws.send(json.dumps({"id": 1, "method": "Network.enable"})) + while time.monotonic() < stop_at: + try: + raw = await asyncio.wait_for(ws.recv(), timeout=0.5) + except TimeoutError: + continue + m = json.loads(raw) + mid = m.get("id") + if mid is not None and mid in self._cmds: + fut = self._cmds.pop(mid) + if not fut.done(): + fut.set_result(m) + continue + if m.get("method") == "Network.requestWillBeSent": + p = m["params"] + url = p.get("request", {}).get("url", "") + if url.endswith("/f/conversation") and \ + p["request"].get("method") == "POST": + self.f_conversation_posts += 1 + elif m.get("method") == "Network.responseReceived": + p = m["params"] + if p.get("response", {}).get("url", "").endswith("/f/conversation"): + self.resp_events.append( + {"rid": p.get("requestId"), + "status": p.get("response", {}).get("status")}) + + async def call(self, method: str, params: dict, timeout: float = 10) -> dict: + if self._ws is None: + raise RuntimeError("observer session not running") + self._next_cmd += 1 + cid = self._next_cmd + fut = asyncio.get_running_loop().create_future() + self._cmds[cid] = fut + await self._ws.send(json.dumps( + {"id": cid, "method": method, "params": params})) + return await asyncio.wait_for(fut, timeout) + + +async def get_response_body(observer: TabObserver, rid: str) -> dict: + """Fetch the f/conversation response body VIA THE OBSERVER SESSION.""" + m = await observer.call("Network.getResponseBody", {"requestId": rid}) + body = (m.get("result") or {}).get("body") or "" + try: + return json.loads(body) + except Exception: + return {"_raw": body[:200]} + + +async def projection_lookup(ws_url: str, conv_id: str) -> dict: + """Authed, structure-only: count USER nodes containing the sentinel.""" + js = """ +(async (__D) => { + var s = await fetch('/api/auth/session', {credentials: 'include'}); + var tok = (await s.json()).accessToken; + var r = await fetch('/backend-api/conversation/' + __D.conv_id + '?offset=0&limit=200', + {credentials: 'include', headers: {'Authorization': 'Bearer ' + tok}}); + var j = await r.json().catch(function() { return {}; }); + var matches = []; + var m = j.mapping || {}; + for (var k in m) { + var msg = (m[k].message || {}); + if (msg.author && msg.author.role === 'user') { + var text = ''; + var parts = (msg.content && msg.content.parts) || []; + for (var p = 0; p < parts.length; p++) if (typeof parts[p] === 'string') text += parts[p]; + if (text.indexOf(__D.sentinel) !== -1) { + matches.push({node_id: k, message_id: msg.id || null, + create_time: msg.create_time || null}); + } + } + } + return {status: r.status, conversation_id: j.conversation_id || null, + sentinel_user_nodes: matches}; +}) +""" + expr = js + "(" + json.dumps({"conv_id": conv_id, "sentinel": SENTINEL}) + ")" + async with websockets.connect(ws_url, max_size=10**7, open_timeout=5) as ws: + await ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate", + "params": {"expression": expr, "silent": True, + "awaitPromise": True, + "returnByValue": True}})) + while True: + m = json.loads(await asyncio.wait_for(ws.recv(), timeout=25)) + if m.get("id") == 1: + return m.get("result", {}).get("result", {}).get("value") or {} + + +def _git_describe() -> str: + """Run-time working-tree provenance — never hard-coded (round-2 review: + a hard-coded code_head mislabels evidence produced by reruns at newer + heads). Git is anchored at THIS repository (derived from the script's + own location, never the process CWD — invoking the harness from some + other repository must not record that repo's branch/SHA), every + command must succeed (check=True; genuine failure returns + "git-unavailable" rather than a misleading label), and an empty head + is rejected. The dirty flag reflects TRACKED working-tree state only + (--untracked-files=no).""" + import subprocess + from pathlib import Path + + repo_root = Path(__file__).resolve().parents[1] + + def _run(*args: str) -> str: + return subprocess.run( + ["git", "-C", str(repo_root), *args], + capture_output=True, text=True, timeout=5, check=True, + ).stdout.strip() + + try: + head = _run("rev-parse", "--short", "HEAD") + if not head: + return "git-unavailable" + branch = _run("branch", "--show-current") + dirty = _run("status", "--porcelain", "--untracked-files=no") + return (f"{branch or 'detached'}@{head}" + + (" (dirty)" if dirty else " (clean)")) + except Exception: + return "git-unavailable" + + +def fire_rest() -> dict: + body = json.dumps({ + "model": "auto", "stream": False, "single_send": True, + "messages": [{"role": "user", + "content": f"{SENTINEL} — live ambiguity certification. " + f"Reply with exactly: ACK"}], + }).encode() + req = urllib.request.Request(f"{BRIDGE}/v1/chat/completions", data=body, + headers={"Content-Type": "application/json"}, + method="POST") + t0 = time.time() + try: + with urllib.request.urlopen(req, timeout=180) as r: + return {"status": r.status, + "headers": dict(r.headers), + "x_should_retry": r.headers.get("x-should-retry"), + "body": json.loads(r.read().decode(errors="replace")), + "elapsed_s": round(time.time() - t0, 2)} + except urllib.error.HTTPError as e: + raw = e.read().decode(errors="replace") + try: + parsed = json.loads(raw) + except Exception: + parsed = {"_raw": raw[:400]} + return {"status": e.code, "headers": dict(e.headers), + "x_should_retry": e.headers.get("x-should-retry"), + "body": parsed, + "elapsed_s": round(time.time() - t0, 2)} + except Exception as e: + return {"error": str(e)[:300], "elapsed_s": round(time.time() - t0, 2)} + + +async def main() -> None: + import logging + + # In-process the service does not configure logging (the CLI entry + # does); enable INFO so the bridge's own causal lines (send_baseline, + # identity_capture_success, send_outcome_unknown) appear in the run log. + logging.basicConfig(level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(name)s: %(message)s") + from chatgpt_web2api.config import Config + from chatgpt_web2api.service import Service + + OUT_DIR.mkdir(parents=True, exist_ok=True) + pre = set(tab_ids()) + print(f"sentinel: {SENTINEL}") + print(f"pre-existing page targets: {len(pre)}") + + config = Config.load(CONFIG) + service = Service(config) + # start() ends with `await shutdown_event.wait()` — it is a run-forever + # call. Run it as a task; request_shutdown() at the end releases it. + svc_task = asyncio.create_task(service.start()) + for _ in range(60): + try: + urllib.request.urlopen(f"{BRIDGE}/health", timeout=3).read() + break + except Exception: + await asyncio.sleep(0.5) + print("experiment bridge started on 8081 (PR #52 tree, in-process)") + + # Honest reconnect counter (review fix): count the DRIVER's reconnect + # calls directly — total websockets.connect creations conflate the + # harness observer and browser-level startup sockets. + STATE["driver_reconnect_calls"] = 0 + _orig_reconnect = service._driver.reconnect + + async def _counting_reconnect(): + STATE["driver_reconnect_calls"] += 1 + return await _orig_reconnect() + + service._driver.reconnect = _counting_reconnect + + # The experiment tab: the driver's OWN tab (may be reclaimed from the + # registry by id — no NEW tab appears in that case, so diffing is wrong). + exp = None + for _ in range(20): + await asyncio.sleep(1) + tid = getattr(service._driver, "_target_id", None) + now = tab_ids() + if tid and tid in now: + exp = now[tid] + break + new = [t for t_, t in now.items() + if t_ not in pre and "chatgpt.com" in t.get("url", "")] + if new: + exp = new[0] + break + if exp is None: + raise SystemExit("experiment owned tab not found") + ws_url = exp["webSocketDebuggerUrl"] + print(f"experiment tab: {exp['id'][:12]} {exp['url'][:60]}") + + observer = TabObserver(ws_url) + stop_at = time.monotonic() + 200 + obs = asyncio.create_task(observer.run(stop_at)) + + await asyncio.sleep(2.0) + rest = await asyncio.to_thread(fire_rest) + print(f"REST: status={rest.get('status')} elapsed={rest.get('elapsed_s')}s " + f"x-should-retry={rest.get('x_should_retry')}") + await asyncio.sleep(4.0) # let any aftermath settle; observer keeps recording + STATE["f_conversation_posts"] = observer.f_conversation_posts + + # conversation id from the f/conversation response body — fetched VIA + # the observer session (same CDP session that tracked the request). + conv_id = None + if observer.resp_events: + try: + rb = await get_response_body(observer, observer.resp_events[0]["rid"]) + conv_id = rb.get("conversation_id") + if not conv_id: + c = rb.get("c") # {conversation_id: ...} keyed shape fallback + if isinstance(c, dict): + conv_id = list(c.keys())[0] + except Exception as e: + print(f"response body fetch failed: {e}") + print(f"conv_id from f/conversation response: {conv_id}") + + proj = {} + if conv_id: + try: + proj = await projection_lookup(ws_url, conv_id) + except Exception as e: + proj = {"error": str(e)[:150]} + + obs.cancel() + service.request_shutdown() + try: + await asyncio.wait_for(svc_task, timeout=20) + except TimeoutError: + print("service.start() did not return after shutdown request") + try: + await asyncio.wait_for(service.stop(), timeout=20) + except Exception as e: + print(f"service stop: {e}") + # Review fix (safety contract): close ONLY the positively owned + # experiment tab — never any new ChatGPT tab, which could be the + # user's. The owned target id comes from the experiment driver itself. + owned_tid = getattr(service._driver, "_target_id", None) + now = tab_ids() + if owned_tid and owned_tid in now and "chatgpt.com" in now[owned_tid].get("url", ""): + try: + urllib.request.urlopen(f"{CDP_HTTP}/json/close/{owned_tid}", timeout=5).read() + print(f"closed owned experiment tab {owned_tid[:12]}") + except Exception: + pass + + captured_user_id = None + try: + captured_user_id = (rest["body"].get("error", {}) + .get("captured_user_id")) + except Exception: + pass + nodes = proj.get("sentinel_user_nodes") or [] + result = { + "captured_at": _dt.datetime.now().isoformat(timespec="seconds"), + "sentinel": SENTINEL, "conv_id": conv_id, + "code_head": f"{_git_describe()} (derived at run time)", + "env": {"chrome": "shared 9222, owned-tab isolation", + "live_bridge": "8080 untouched", "injection": "ws.send wrapper: " + "forward to real Chrome, then raise ConnectionClosedError — once"}, + "counters": { + "mutating_Runtime_evaluate_frames": STATE["mutating_frames"], + "fault_injected": STATE["injected"], + # Honest reconnect accounting (review fix): the DRIVER's own + # reconnect calls, counted by wrapping driver.reconnect. + "driver_reconnect_calls": STATE.get("driver_reconnect_calls", 0), + # Informational only: total websockets.connect creations in this + # process — conflates the page socket, browser-level startup + # sockets, and the harness observer. NOT a reconnect measure. + "websockets_created_total_informational": STATE["ws_connections"], + "f_conversation_posts": STATE["f_conversation_posts"], + "rest_status": rest.get("status"), + "x_should_retry": rest.get("x_should_retry"), + "rest_error_code": (rest.get("body", {}).get("error", {}) or {}).get("code"), + "captured_user_id": captured_user_id, + "backend_sentinel_user_nodes": len(nodes), + "backend_node_ids": [n.get("node_id") for n in nodes], + "backend_ids_match_captured": bool( + captured_user_id and nodes + and all(n.get("node_id") == captured_user_id or + n.get("message_id") == captured_user_id for n in nodes)), + }, + "rest": {k: rest.get(k) for k in ("status", "elapsed_s")}, + "f_conversation_responses": observer.resp_events[:5], + "projection": proj, + } + ts = _dt.datetime.now().strftime("%Y%m%d-%H%M%S") + path = OUT_DIR / f"live-cert-51-{ts}.json" + path.write_text(json.dumps(result, indent=1), encoding="utf-8") + print(f"\nwrote {path}\n==== LIVE CERTIFICATION COUNTERS ====") + for k, v in result["counters"].items(): + print(f" {k}: {v}") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/scripts/phase0_passive_recon.py b/scripts/phase0_passive_recon.py new file mode 100644 index 00000000..45b002ae --- /dev/null +++ b/scripts/phase0_passive_recon.py @@ -0,0 +1,331 @@ +"""Phase 0 passive upstream recon — selector-agnostic structure capture. + +Part of the 2026-09 upstream contract audit (docs/research/). +Safety contract (hard rules, do not relax): + - Discovers EXISTING chatgpt.com tabs only. Never creates, closes, or + navigates a tab (tab displacement is a known production defect). + - Runs read-only JS probes (querySelector/count/attribute inventory). + - Fetches are same-origin GETs the page itself already makes + (/api/auth/session, /backend-api/models, /backend-api/conversation/{id}). + - Records STRUCTURE ONLY: no message text, no conversation titles beyond + what the tab list already exposes, no accessToken value (key names only). + +Output: captures/phase0/passive-.json plus a stdout summary. +The comparison this exists for: backend mapping node count vs mounted DOM +message count per open conversation — the sectioned-loading question. +""" +from __future__ import annotations + +import asyncio +import datetime as _dt +import json +import pathlib +import urllib.request + +import websockets + +CDP = 9222 +OUT_DIR = pathlib.Path("captures/phase0") +MAX_PROJECTION_CONVS = 4 # keep GET volume minimal +PROJECTION_LIMIT = 500 # ask for more than any open conversation has + +COLLECTOR_JS = r""" +(async function() { + function attrs(el) { + var out = []; + for (var i = 0; i < el.attributes.length; i++) out.push(el.attributes[i].name); + return out; + } + function slim(el) { + if (!el) return null; + return { + tag: el.tagName.toLowerCase(), + id: el.id || null, + classes: (el.className && el.className.toString + ? el.className.toString().split(/\s+/).slice(0, 4) : []), + role: el.getAttribute('role'), + testid: el.getAttribute('data-testid'), + attrNames: attrs(el).slice(0, 12), + }; + } + var res = { path: location.pathname, hrefIsConv: location.pathname.indexOf('/c/') !== -1 }; + + // Composer: July selectors individually + generic contenteditable inventory. + res.julyComposer = { + primary_div_prompt_textarea: !!document.querySelector('div[role="textbox"]#prompt-textarea'), + primary_div_prosemirror: !!document.querySelector('div[role="textbox"].ProseMirror'), + fallback_textarea: !!document.querySelector('textarea#prompt-textarea'), + any_prompt_textarea_id: !!document.querySelector('#prompt-textarea'), + }; + res.contentEditables = []; + var ces = document.querySelectorAll('[contenteditable="true"]'); + for (var i = 0; i < Math.min(ces.length, 5); i++) res.contentEditables.push(slim(ces[i])); + + // Buttons: data-testid inventory (values only) + Send/Stop/Model labels. + var testids = {}; + var btns = document.querySelectorAll('button[data-testid]'); + for (var j = 0; j < btns.length; j++) { + var t = btns[j].getAttribute('data-testid'); + testids[t] = (testids[t] || 0) + 1; + } + res.buttonTestids = testids; + res.julySendButton = { + aria_label_send_not_stop: !!document.querySelector('button[aria-label*="Send" i]:not([data-testid="stop-button"])'), + testid_send_button: !!document.querySelector('button[data-testid="send-button"]'), + testid_stop_button: !!document.querySelector('button[data-testid="stop-button"]'), + }; + res.notableLabels = []; + var lbls = document.querySelectorAll('button[aria-label]'); + for (var k = 0; k < lbls.length; k++) { + var L = lbls[k].getAttribute('aria-label') || ''; + if (/send|stop|model|attach|upload|plus/i.test(L)) res.notableLabels.push(L.slice(0, 80)); + } + res.notableLabels = res.notableLabels.slice(0, 15); + + // Message tree: counts + structural sample of first/last turn. + var roleSel = '[data-message-author-role]'; + res.messages = { + containers_by_author_role: { + user: document.querySelectorAll('[data-message-author-role="user"]').length, + assistant: document.querySelectorAll('[data-message-author-role="assistant"]').length, + system: document.querySelectorAll('[data-message-author-role="system"]').length, + }, + attr_message_id: document.querySelectorAll('[data-message-id]').length, + conversation_turn_testids: document.querySelectorAll('[data-testid^="conversation-turn"]').length, + articles: document.querySelectorAll('article').length, + }; + var all = document.querySelectorAll(roleSel); + function sample(el) { + if (!el) return null; + var s = slim(el); + s.hasMarkdown = !!el.querySelector('.markdown'); + s.hasResultThinking = !!el.querySelector('.result-thinking'); + s.hasResultStreaming = !!el.querySelector('.result-streaming'); + s.hasCopyTestid = !!el.querySelector('[data-testid*="copy"]'); + s.hasMessageIdAttr = el.hasAttribute('data-message-id'); + return s; + } + res.firstMessage = all.length ? sample(all[0]) : null; + res.lastMessage = all.length ? sample(all[all.length - 1]) : null; + + // Virtualization hints: sentinels, load-more, shrunk history containers. + res.virtualizationHints = { + load_more_buttons: document.querySelectorAll('button[aria-label*="load" i], [data-testid*="load-more"]').length, + virtualization_attr_nodes: document.querySelectorAll('[data-virtualization], [data-virt]').length, + main_children: document.querySelector('main') ? document.querySelector('main').children.length : null, + }; + return res; +})() +""" + +# Structure-only backend probes. The token is fetched transiently INSIDE the +# page JS and never returned or persisted; the projection fetch reproduces +# the report's authenticated method (Bearer header) — cookie-only requests +# to this endpoint return masked 404s, which the report records explicitly. +AUTH_SHAPE_JS = """ +(async function() { + try { + var r = await fetch('/api/auth/session', {credentials: 'include'}); + var j = await r.json().catch(function() { return {}; }); + return { status: r.status, keys: Object.keys(j).sort(), + expires: j.expires || null, hasAccessToken: 'accessToken' in j }; + } catch (e) { return { error: String(e) }; } +})() +""" + +MODELS_JS = """ +(async function() { + try { + var s = await fetch('/api/auth/session', {credentials: 'include'}); + var tok = (await s.json()).accessToken; + var r = await fetch('/backend-api/models?iim=false&is_gizmo=false', + {credentials: 'include', + headers: {'Authorization': 'Bearer ' + tok}}); + var j = await r.json().catch(function() { return {}; }); + var rows = (j && j.models) ? j.models : []; + return { status: r.status, topKeys: Object.keys(j).sort(), count: rows.length, + slugs: rows.map(function(m) { return m.slug + '|' + (m.title || ''); }).sort() }; + } catch (e) { return { error: String(e) }; } +})() +""" + +# Parameterized (NOT self-called): eval_on appends the JSON argument when +# injecting data — the IIFE-parameter pattern, never a top-level var __D. +PROJECTION_JS = """ +(async (__D) => { + try { + var s = await fetch('/api/auth/session', {credentials: 'include'}); + var tok = (await s.json()).accessToken; + var r = await fetch('/backend-api/conversation/' + __D.conv_id + '?offset=0&limit=' + __D.limit, + {credentials: 'include', + headers: {'Authorization': 'Bearer ' + tok}}); + var j = await r.json().catch(function() { return {}; }); + var mapping = j.mapping || {}; + var counts = { user: 0, assistant: 0, system: 0, tool: 0, other: 0 }; + var node0 = null; + for (var k in mapping) { + var node = mapping[k]; + var role = node && node.message ? node.message.author && node.message.author.role : null; + if (role && counts[role] !== undefined) counts[role]++; else counts.other++; + if (!node0) node0 = { keys: Object.keys(node).sort(), + messageKeys: node.message ? Object.keys(node.message).sort() : null }; + } + return { status: r.status, topKeys: Object.keys(j).sort(), + mappingNodes: Object.keys(mapping).length, roleCounts: counts, + currentNode: j.current_node || null, sampleNode: node0, + mappingNodeKeysHaveChildren: mapping[j.current_node] ? ('children' in mapping[j.current_node]) : null }; + } catch (e) { return { error: String(e) }; } +}) +""" + + +def list_targets() -> list[dict]: + data = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{CDP}/json/list", timeout=5).read()) + return [t for t in data if t.get("type") == "page" and "chatgpt.com" in t.get("url", "")] + + +def conv_id_from_url(url: str) -> str | None: + if "/c/" not in url: + return None + tail = url.split("/c/")[1] + return tail.split("/")[0].split("?")[0] or None + + +async def eval_on(ws, expr: str, data: dict | None = None, timeout: float = 8.0) -> dict: + """Runtime.evaluate with awaitPromise; returns result value or error info.""" + msg_id = eval_on._next # type: ignore[attr-defined] + eval_on._next += 1 # type: ignore[attr-defined] + if data is not None: + # Data injected as an IIFE CALL ARGUMENT, never a top-level + # `var __D` — ChatGPT's page defines its own global __D, and that + # collision is a bug class this project already diagnosed (the + # production code moved to this pattern). Data-bearing JS constants + # in this script are therefore parameterized, not self-called. + expression = f"{expr}({json.dumps(data)})" + payload = {"id": msg_id, "method": "Runtime.evaluate", + "params": {"expression": expression, + "awaitPromise": True, "returnByValue": True, "silent": True}} + else: + payload = {"id": msg_id, "method": "Runtime.evaluate", + "params": {"expression": expr, "awaitPromise": True, + "returnByValue": True, "silent": True}} + await ws.send(json.dumps(payload)) + while True: + raw = await asyncio.wait_for(ws.recv(), timeout=timeout) + msg = json.loads(raw) + if msg.get("id") != msg_id: + continue + if "error" in msg: + return {"cdpError": msg["error"].get("message")} + result = msg.get("result", {}).get("result", {}) + if result.get("subtype") == "error": + return {"jsError": result.get("description", "")[:300]} + return result.get("value", {"noValue": True}) + + +eval_on._next = 1 # type: ignore[attr-defined] + + +async def probe_tab(target: dict, do_backend: bool) -> dict: + out: dict = {"url": target["url"][:120], "title": target.get("title", "")[:60]} + if not target.get("webSocketDebuggerUrl"): + out["error"] = "no webSocketDebuggerUrl (prerendered/discarded)" + return out + try: + async with websockets.connect(target["webSocketDebuggerUrl"], max_size=10**7, + open_timeout=5) as ws: + try: + out["dom"] = await eval_on(ws, COLLECTOR_JS) + except Exception as e: # frozen/discarded tab — record and move on + out["error"] = f"dom probe failed: {e}" + return out + if do_backend: + out["authShape"] = await eval_on(ws, AUTH_SHAPE_JS) + out["models"] = await eval_on(ws, MODELS_JS) + except Exception as e: + out["error"] = f"connect failed: {e}" + return out + + +async def probe_projection(target: dict, conv_ids: list[str]) -> dict: + results = {} + try: + async with websockets.connect(target["webSocketDebuggerUrl"], max_size=10**7, + open_timeout=5) as ws: + for cid in conv_ids[:MAX_PROJECTION_CONVS]: + try: + results[cid] = await eval_on(ws, PROJECTION_JS, + {"conv_id": cid, "limit": PROJECTION_LIMIT}) + except Exception as e: + results[cid] = {"error": str(e)} + except Exception as e: + for cid in conv_ids[:MAX_PROJECTION_CONVS]: + results[cid] = {"error": f"connect failed: {e}"} + return results + + +async def main() -> None: + targets = list_targets() + print(f"chatgpt.com tabs found: {len(targets)} (passive only — no tab lifecycle, no navigation)") + OUT_DIR.mkdir(parents=True, exist_ok=True) + ts = _dt.datetime.now().strftime("%Y%m%d-%H%M%S") + + fresh = next((t for t in targets if t["url"].rstrip("/") == "https://chatgpt.com"), targets[0]) + tabs = await asyncio.gather(*(probe_tab(t, do_backend=(t is fresh)) for t in targets)) + + conv_ids = [cid for cid in (conv_id_from_url(t["url"]) for t in targets) if cid] + # Prefer diverse, live tabs: first plain /c/ conversation, then project-scoped. + plain = [t for t in targets if conv_id_from_url(t["url"]) + and t["url"].split("/c/")[0].endswith("chatgpt.com/")] + ordered = [] + for t in plain + [t for t in targets if t not in plain]: + cid = conv_id_from_url(t["url"]) + if cid and cid not in ordered: + ordered.append(cid) + print(f"conversation ids in open tabs: {len(conv_ids)}; probing projection for up to {MAX_PROJECTION_CONVS}") + projections = await probe_projection(fresh, ordered) if ordered else {} + + report = { + "captured_at": _dt.datetime.now().isoformat(timespec="seconds"), + "chrome": json.loads(urllib.request.urlopen(f"http://127.0.0.1:{CDP}/json/version", timeout=5).read())["Browser"], + "mode": "passive-only (no navigation, no clicks, GET fetches only)", + "tabs": tabs, + "projections": projections, + } + out_path = OUT_DIR / f"passive-{ts}.json" + out_path.write_text(json.dumps(report, indent=1), encoding="utf-8") + print(f"wrote {out_path}\n") + + # Stdout summary: the sectioned-loading comparison + selector verdicts. + for tab in tabs: + dom = tab.get("dom", {}) + msgs = dom.get("messages", {}) + mounted = msgs.get("containers_by_author_role", {}) + cid = conv_id_from_url(tab["url"]) + proj = projections.get(cid) if cid else None + print(f"— {tab['url'][:100]}") + print(f" composer july-hit={dom.get('julyComposer')} send={dom.get('julySendButton')}") + print(f" mounted: user={mounted.get('user')} assistant={mounted.get('assistant')} " + f"turn-testids={msgs.get('conversation_turn_testids')} articles={msgs.get('articles')}") + if proj and "mappingNodes" in proj: + rc = proj.get("roleCounts", {}) + print(f" backend mapping: total={proj['mappingNodes']} user={rc.get('user')} " + f"assistant={rc.get('assistant')} status={proj.get('status')} current_node={bool(proj.get('currentNode'))}") + dom_total = (mounted.get("user") or 0) + (mounted.get("assistant") or 0) + be_total = (rc.get("user") or 0) + (rc.get("assistant") or 0) + if be_total: + ratio = dom_total / be_total + verdict = "FULL-MOUNT" if ratio > 0.999 else ("PARTIAL-MOUNT" if ratio >= 0.3 else "HEAVILY-TRIMMED") + print(f" >>> sectioned-loading: mounted/backend = {dom_total}/{be_total} = {ratio:.2f} → {verdict}") + models = next((t.get("models") for t in tabs if t.get("models")), None) + if models: + print(f"\nmodels endpoint: status={models.get('status')} count={models.get('count')}") + for slug in (models.get("slugs") or [])[:25]: + print(f" {slug}") + auth = next((t.get("authShape") for t in tabs if t.get("authShape")), None) + if auth: + print(f"\nauth session: status={auth.get('status')} keys={auth.get('keys')} hasAccessToken={auth.get('hasAccessToken')}") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/src/chatgpt_web2api/api_server.py b/src/chatgpt_web2api/api_server.py index c0ce8894..2f81baa7 100644 --- a/src/chatgpt_web2api/api_server.py +++ b/src/chatgpt_web2api/api_server.py @@ -23,12 +23,20 @@ CDPDriver, GenerationStuckError, RateLimitError, + SendOutcomeUnknownError, is_rate_limited_text, ) from .config import Config from .cross_process_lock import LockAcquisitionError from .lock_resolver import MutationLock, OwnedTabRequiredError, resolve_mutation_lock -from .resilience import retry_on_rate_limit +from .resilience import chat_retry_attempts, retry_on_rate_limit + + +def _resolve_single_send(body: dict) -> bool: + """#51 opt-in: REST accepts ``single_send`` at the top level or under + ``metadata`` (mirroring ``project_id`` resolution above).""" + return bool(body.get("single_send") + or (body.get("metadata") or {}).get("single_send")) logger = logging.getLogger(__name__) @@ -390,7 +398,9 @@ async def _handle_chat(self, request: web.Request) -> web.Response: if stream: return await self._stream_response(request, model_slug, full_text, timeout) else: - return await self._full_response(request, model_slug, full_text, timeout) + return await self._full_response( + request, model_slug, full_text, timeout, + single_send=_resolve_single_send(body)) except Exception as e: logger.error("Chat error: %s", e, exc_info=True) @@ -451,6 +461,36 @@ def _error_response(self, exc: Exception) -> web.Response: status=429, headers={"Retry-After": retry_after}, ) + if isinstance(exc, SendOutcomeUnknownError): + # #51: an ambiguous send outcome is NOT a plain server error. + # 409 Conflict is the idempotency-aware "request may have been + # applied" status. The body carries the machine-readable + # distinction: retry_safe=False and the preserved causal + # evidence (captured_user_id) when the in-flight capture + # resolved. The x-should-retry:false header below is what + # prevents official OpenAI SDKs from auto-retrying the 409 — + # they check it before their status rules. Retry-After is + # deliberately absent: the instruction is "do not resend". + payload: dict = { + "message": f"{exc} (outcome unknown — do not blindly re-request; " + "reconcile via the conversation if captured_user_id is set)", + "type": "server_error", + "param": None, + "code": "send_outcome_unknown", + } + if exc.captured_user_id: + payload["captured_user_id"] = exc.captured_user_id + payload["retry_safe"] = exc.retry_safe + # x-should-retry is checked by the official OpenAI SDKs BEFORE + # their retryable-status rules (409 is otherwise auto-retried + # by default) — without this header the SDK layer re-enables + # the replay this whole patch removes. Deliberately no + # Retry-After: the instruction is "do not resend". + return web.json_response( + {"error": payload}, + status=409, + headers={"x-should-retry": "false"}, + ) if isinstance(exc, AuthExpiredError): return web.json_response( { @@ -521,7 +561,8 @@ def _error_response(self, exc: Exception) -> web.Response: # ── Response formatters ─────────────────────────────────── async def _full_response( - self, request: web.Request, model: str, text: str, timeout: float + self, request: web.Request, model: str, text: str, timeout: float, + single_send: bool = False, ) -> web.Response: """Non-streaming: collect all chunks, return one JSON. @@ -543,7 +584,12 @@ async def _send_and_collect() -> str: collected += chunk.delta return collected - full_text = await retry_on_rate_limit(self._driver, _send_and_collect) + # #51: single_send runs the (mutating) chat operation exactly once — + # a post-mutation rate limit surfaces instead of re-sending. + full_text = await retry_on_rate_limit( + self._driver, _send_and_collect, + max_attempts=chat_retry_attempts({"single_send": single_send}), + ) conv_id = self._driver._current_conv_id or "" self._last_conv_id = conv_id @@ -751,6 +797,35 @@ async def _preflight() -> None: ], }, ) + except SendOutcomeUnknownError as e: + # #51: ambiguous send outcome mid-stream (status locked at 200 — + # the inline marker is the only channel). Same precedent as the + # rate-limit marker above. The code + captured_user_id let a + # client reconcile instead of blindly re-requesting. + logger.warning("Mid-stream send outcome unknown: %s", e) + evidence = f", captured_user_id={e.captured_user_id}" if e.captured_user_id else "" + await self._send_sse( + resp, + { + "id": cid, + "object": "chat.completion.chunk", + "created": created, + "model": model, + "choices": [ + { + "index": 0, + "delta": { + "content": ( + f"\n\n[Error: send_outcome_unknown — the send may " + f"or may not have been applied; do not blindly " + f"re-request{evidence}]" + ) + }, + "finish_reason": "error", + } + ], + }, + ) except Exception as e: logger.error("Stream error: %s", e) await self._send_sse( diff --git a/src/chatgpt_web2api/cdp_driver.py b/src/chatgpt_web2api/cdp_driver.py index 3bc3eae4..d1172306 100644 --- a/src/chatgpt_web2api/cdp_driver.py +++ b/src/chatgpt_web2api/cdp_driver.py @@ -46,6 +46,12 @@ class StreamChunk: # a real cooldown clear but short enough that a transient blip recovers fast. RATE_LIMIT_DEFAULT_RETRY_AFTER = 60 +# #51: bounded window (seconds) to wait for an in-flight IdentityListener +# UUID capture after an ambiguous send mutation (SendOutcomeUnknownError), +# before the capture scope closes and clears that state. Healthy capture +# measured at +139 ms in the 2026-09-21 1B timeline; 1 s is the allowance. +AMBIGUOUS_CAPTURE_WINDOW_S = 1.0 + # Re-exported from backend_client (Phase 5 PR1 extraction) for back-compat. # Canonical home is now backend_client.py. from .backend_client import TOKEN_TTL_SECONDS # noqa: E402,F401 @@ -270,6 +276,41 @@ class SendReadinessError(RuntimeError): """ +class SendOutcomeUnknownError(RuntimeError): + """A chat mutation was dispatched but its external effect could not be + proven (issue #51 — ambiguous CDP send). + + Raised when the transport dies (or its response is lost) while a + MUTATING ``Runtime.evaluate`` (the send click) is in flight. The frame + may or may not have executed remotely, so the outcome is UNKNOWN, not + failed: + + - reads / reconciliation are permitted (``reconciliation_safe=True``): + if the IdentityListener captured a user UUID, a backend lookup by + exact UUID can upgrade UNKNOWN → CONFIRMED without another send. + - replaying the mutation is forbidden (``retry_safe=False``): a retry + could duplicate the user turn. Automatic retry must never happen. + + Carries structured evidence (not just text) so callers and the REST/MCP + surfaces can decide without parsing messages. + """ + + def __init__( + self, + message: str, + *, + stage: str = "send_mutation", + captured_user_id: str | None = None, + ) -> None: + super().__init__(message) + self.code = "send_outcome_unknown" + self.stage = stage + self.effect_state = "unknown" + self.captured_user_id = captured_user_id + self.retry_safe = False + self.reconciliation_safe = True + + class CDPReconnectError(RuntimeError): """Raised when CDP reconnect exhausts its 3-attempt backoff without re-establishing the websocket. @@ -383,6 +424,10 @@ def __init__( self._last_refresh_attempt_at: float = 0.0 self._current_conv_id: str | None = None self._current_model: str | None = None + # #51 cancellation side channel: outcome-unknown record for the most + # recent send (set by note_send_outcome_unknown; cleared at each + # send_and_stream start). See CDPTransport._js_mutation. + self.last_send_outcome_unknown: dict | None = None # CDP response routing (#7): id-keyed futures + background reader self._pending: dict[int, asyncio.Future] = {} self._reader_task: asyncio.Task | None = None @@ -1091,12 +1136,49 @@ def _should_reconnect(exc: Exception) -> bool: return CDPTransport._should_reconnect(exc) + def note_send_outcome_unknown(self, *, stage: str, reason: str) -> None: + """Record an outcome-unknown send mutation without raising (#51). + + Used by the cancellation path in ``CDPTransport._js_mutation``: + ``asyncio.CancelledError`` must propagate unchanged (cancellation + contract), so the UNKNOWN/do-not-resend classification cannot travel + on the exception — it lands here, observable for reconciliation: + + - ``self.last_send_outcome_unknown`` (dict; cleared at the start + of each send in ``send_and_stream``) — reflects the most recent + send only. + - a structured ERROR log line with code=send_outcome_unknown. + """ + self.last_send_outcome_unknown = { + "code": "send_outcome_unknown", + "stage": stage, + "reason": reason, + "at": time.time(), + "retry_safe": False, + "reconciliation_safe": True, + } + logger.error( + "send_outcome_unknown(cancelled): stage=%s reason=%s — do not " + "resend without reconciliation", + stage, reason, + ) + async def _js(self, expr: str, timeout: float = 15) -> str: """Soft ``Runtime.evaluate`` — returns "" on failure. Delegated to CDPTransport (Phase 5 PR2 extraction).""" return await self._transport._js(expr, timeout) + async def _js_mutation(self, expr: str, timeout: float = 15) -> str: + """MUTATING ``Runtime.evaluate`` (#51) — sent exactly once, never + replayed across a reconnect; ambiguous outcomes raise + SendOutcomeUnknownError (reconcile only, never resend). + + Delegated to CDPTransport like _js; kept as an explicit driver-level + primitive so call sites read as observation (_js) vs effect + (_js_mutation).""" + return await self._transport._js_mutation(expr, timeout) + async def _js_with_data(self, expr_template: str, data: dict, timeout: float = 15) -> str: """Evaluate JS with safely injected ``__D`` data variables (soft). @@ -1596,7 +1678,8 @@ async def _verify_send_acknowledged(self) -> bool | None: Polls briefly (3s at 0.5s intervals). Never raises. """ import time as _time - from .chatgpt_dom import COMPOSER_SELECTOR, COMPOSER_FALLBACK_SELECTOR + + from .chatgpt_dom import COMPOSER_FALLBACK_SELECTOR, COMPOSER_SELECTOR pre_send_count = getattr(self, "_pre_send_user_count", None) if pre_send_count is None: @@ -1744,6 +1827,9 @@ async def send_and_stream( from .identity_listener import hash_sent_text from .turn_anchor import TurnReconciliationError + # #51 cancellation side channel reflects the most recent send only. + self.last_send_outcome_unknown = None + # PR4 belt-and-suspenders: refuse to mutate the DOM in parallel mode. self._assert_owned_tab_required() # A1: count existing assistants BEFORE sending (fail-closed baseline). @@ -1768,7 +1854,32 @@ async def send_and_stream( try: # Type and send. await self.type_message(text) - await self.click_send() + try: + await self.click_send() + except SendOutcomeUnknownError as exc: + # #51: the click frame may have executed remotely. The + # IdentityListener may already hold — or be about to resolve — + # the outgoing UUID; preserve it on the error BEFORE the + # finally below closes the capture scope and clears that + # state. Evidence preservation only: the outcome stays + # UNKNOWN until a later reconciliation reads the backend. + # This path never resends. Window rationale: 1B measured + # healthy capture at +139 ms; 1 s is the bounded allowance. + if capture_scope is not None and self._identity_listener is not None: + try: + captured = await self._identity_listener.wait_for_captured_uuid( + timeout=AMBIGUOUS_CAPTURE_WINDOW_S + ) + except Exception: + captured = None + if captured: + exc.captured_user_id = captured + logger.info( + "send_outcome_unknown: captured_user_id=%s preserved " + "before scope close (stage=%s)", + captured, exc.stage, + ) + raise # A2 Step 6: wait for the IdentityListener to capture the UUID. captured_uuid = None @@ -1789,6 +1900,7 @@ async def send_and_stream( # environment, unusual DOM), DON'T block the send. The check is a # safety net for the overloaded-page case, not a hard gate that # could prevent sends in edge cases we haven't seen. + acknowledged: bool | None = None if not captured_uuid: try: acknowledged = await self._verify_send_acknowledged() @@ -1806,6 +1918,18 @@ async def send_and_stream( # the send — let completion detection proceed. Log so the # failure is traceable. logger.debug("Send acknowledgment probe failed (non-blocking): %s", ack_err) + else: + acknowledged = True # UUID captured — stronger than the DOM probe + + # #51 breaker boundary: COMPOSER_SEND_READINESS success now means + # SUBMISSION evidence, not "the click's JS returned". Primary: + # IdentityListener captured the outgoing UUID (the submission + # request was emitted, causally identified — 1B: +139 ms). + # Fallback: the DOM acknowledgment probe returned True. This is + # a send-readiness breaker, so remote persistence (backend node, + # TurnAnchor) stays with the turn/effect contract, not here. + if self._breakers and (captured_uuid or acknowledged is True): + self._breakers.record_success(BreakerKind.COMPOSER_SEND_READINESS) # A2 Step 7: build the final anchor (fallback + captured UUID). turn_anchor = fallback_anchor.with_captured_id(captured_uuid) @@ -1823,6 +1947,17 @@ async def send_and_stream( ): yield chunk + # Breaker downstream-success fallback (Codex P2 review): a turn + # that reached FULL generation completion is submission evidence + # even when the UUID capture missed AND the DOM acknowledgment + # probe was inconclusive (acknowledged is None). Without this, + # successful sends never clear COMPOSER_SEND_READINESS failure + # history in that path, and the breaker can trip despite + # interleaved successes. Reached only on stream COMPLETION — a + # raised/failed stream is not evidence. + if self._breakers and not (captured_uuid or acknowledged is True): + self._breakers.record_success(BreakerKind.COMPOSER_SEND_READINESS) + # Wait for URL to become /c/{id} conv_id = "" for _ in range(30): diff --git a/src/chatgpt_web2api/cdp_transport.py b/src/chatgpt_web2api/cdp_transport.py index cf6133ba..bceacd78 100644 --- a/src/chatgpt_web2api/cdp_transport.py +++ b/src/chatgpt_web2api/cdp_transport.py @@ -154,6 +154,13 @@ async def _cdp( d._pending[mid] = fut try: await d._ws.send(json.dumps({"id": mid, "method": method, "params": params or {}})) + except asyncio.CancelledError: + # Cancellation must not leak the pending entry (round-2 review): + # CancelledError bypasses `except Exception` below, and with a + # dead/hanging socket nothing would ever pop it. Re-raise + # unchanged — the cancellation contract is preserved. + d._pending.pop(mid, None) + raise except Exception as e: d._pending.pop(mid, None) if _retry and self._should_reconnect(e): @@ -163,6 +170,11 @@ async def _cdp( raise try: return await asyncio.wait_for(fut, timeout) + except asyncio.CancelledError: + # Same leak on the response wait: wait_for cancels the inner + # future, but the id-keyed entry stays. Pop it, re-raise. + d._pending.pop(mid, None) + raise except TimeoutError: d._pending.pop(mid, None) raise TimeoutError(f"CDP timeout: {method}") @@ -197,6 +209,72 @@ async def _js(self, expr: str, timeout: float = 15) -> str: ) return resp.get("result", {}).get("result", {}).get("value", "") + async def _js_mutation(self, expr: str, timeout: float = 15) -> str: + """MUTATING ``Runtime.evaluate`` — sent exactly once, never replayed. + + #51 no-replay seam. ``_js`` / ``_js_strict`` are observation + primitives: on a reconnect-class socket death they reconnect and + resend the same frame, which is the right resilience for reads. For + a mutation (the send click) that replay is a duplicate-send hazard: + the first frame may already have executed remotely. This primitive + disables the retry (``_retry=False``) and converts ambiguous + outcomes — reconnect-class send failure OR a response timeout after + the frame was handed to the socket — into + ``SendOutcomeUnknownError``: reconcile (read) only, never resend. + Non-reconnect errors raised before meaningful delivery propagate + unchanged. + """ + try: + resp = await self._driver._cdp( + "Runtime.evaluate", + { + "expression": expr, + "awaitPromise": True, + "returnByValue": True, + "timeout": int(timeout * 1000), + }, + timeout=timeout, + _retry=False, + ) + except TimeoutError as exc: + from .cdp_driver import SendOutcomeUnknownError + + raise SendOutcomeUnknownError( + "Send mutation outcome unknown: CDP response timed out after " + "the mutation frame was dispatched; it may have executed. " + "Reconcile (read) only — never resend." + ) from exc + except asyncio.CancelledError: + # Deliberate cancellation semantics (Codex P1 review): the frame + # may ALREADY have reached Chrome — cancellation can land after + # the transport write — so this is an outcome-unknown mutation. + # But swallowing or converting CancelledError would break the + # asyncio cancellation contract (a cancelled task must surface + # CancelledError; converting it would turn a client-disconnect + # cancellation into a 409 response). So: cancellation propagates + # UNCHANGED, and the UNKNOWN/do-not-resend classification is + # recorded as an observable side channel on the driver instead. + # No evidence-window wait here — cancellation must not be + # delayed; the scope still closes in send_and_stream's finally. + self._driver.note_send_outcome_unknown( + stage="send_mutation", + reason="cancelled during dispatch — the mutation frame may " + "have reached Chrome; do not resend without " + "reconciliation", + ) + raise + except Exception as exc: + if self._should_reconnect(exc): + from .cdp_driver import SendOutcomeUnknownError + + raise SendOutcomeUnknownError( + "Send mutation outcome unknown: CDP connection died while " + "dispatching the mutation; the frame may or may not have " + "executed. Reconcile (read) only — never resend." + ) from exc + raise + return resp.get("result", {}).get("result", {}).get("value", "") + async def _js_with_data(self, expr_template: str, data: dict, timeout: float = 15) -> str: """Evaluate JS with safely injected data variables. diff --git a/src/chatgpt_web2api/chatgpt_dom.py b/src/chatgpt_web2api/chatgpt_dom.py index fc76f168..966d6bca 100644 --- a/src/chatgpt_web2api/chatgpt_dom.py +++ b/src/chatgpt_web2api/chatgpt_dom.py @@ -450,7 +450,11 @@ async def click_send(self) -> None: break await asyncio.sleep(SEND_BUTTON_POLL_INTERVAL_S) - result = await d._js( + # #51: the click is a MUTATION — it goes through _js_mutation (sent + # exactly once, never replayed across a reconnect; an ambiguous + # outcome raises SendOutcomeUnknownError). The readiness POLL above + # stays on _js (a read — reconnect-retry is safe there). + result = await d._js_mutation( "(function() {" f" var btn = document.querySelector('{SEND_BUTTON_SELECTOR}')" f" || document.querySelector('{SEND_BUTTON_FALLBACK_SELECTOR}')" @@ -472,11 +476,13 @@ async def click_send(self) -> None: raise SendReadinessError(f"Send failed: {result}") logger.info("Message sent") - # Success: clear composer failure history and recover a half-open - # breaker. Only after the message is confirmed sent — not after - # type_message alone, since a successful type can still fail to send. - if d._breakers: - d._breakers.record_success(BreakerKind.COMPOSER_SEND_READINESS) + # #51: click_send does NOT record COMPOSER_SEND_READINESS success + # here. 'sent' only proves the synthetic mouse events ran — 1B + # measured the click's JS return at +141 ms while React acceptance + # (composer cleared, user node mounted) landed at +437 ms. The + # breaker's success point moved to send_and_stream, at submission + # evidence: captured UUID (primary) or the DOM acknowledgment + # fallback. # ── Rate-limit popup ────────────────────────────────────── diff --git a/src/chatgpt_web2api/mcp_server.py b/src/chatgpt_web2api/mcp_server.py index 8387f0c4..b9acf6bf 100644 --- a/src/chatgpt_web2api/mcp_server.py +++ b/src/chatgpt_web2api/mcp_server.py @@ -48,6 +48,7 @@ CDPDriver, GenerationStuckError, RateLimitError, + SendOutcomeUnknownError, ) from .config import Config from .cross_process_lock import LockAcquisitionError @@ -56,7 +57,7 @@ OwnedTabRequiredError, resolve_mutation_lock, ) -from .resilience import retry_on_rate_limit +from .resilience import CHAT_MAX_ATTEMPTS, chat_retry_attempts, retry_on_rate_limit from .tab_registry import TabRegistry logger = logging.getLogger(__name__) @@ -82,6 +83,14 @@ class ChatCompletionInput(BaseModel): """Input schema for chat_completion tool.""" message: str = Field(description="The user message to send to ChatGPT") + single_send: bool = Field( + default=False, + description=( + "#51 no-replay opt-in: run the send exactly once. A rate limit " + "surfacing after the message was dispatched is returned as an " + "error instead of re-sending (which could duplicate the turn)." + ), + ) system_prompt: str | None = Field( default=None, description=( @@ -282,9 +291,17 @@ class ChatWithGptInput(BaseModel): gpt_id: str = Field( description=( "Custom GPT gizmo ID (e.g. g-hkJGhxxx). Use list_gpts to discover available GPTs." - ), + ) ) message: str = Field(description="The message to send to the GPT") + single_send: bool = Field( + default=False, + description=( + "#51 no-replay opt-in: run the send exactly once. A rate limit " + "surfacing after the message was dispatched is returned as an " + "error instead of re-sending (which could duplicate the turn)." + ), + ) # ═══════════════════════════════════════════════════════════════ @@ -317,6 +334,30 @@ class ToolName(str, Enum): CHAT_WITH_GPT = "chat_with_gpt" +# #51: single_send controls USER-chat mutation replay ONLY. create_memory +# also sits in _CHAT_TOOLS (rate-limit retry applies to it) but must not +# inherit chat-send semantics by set membership — memory mutations would +# need their own effect analysis before getting at-most-once behavior. +_SINGLE_SEND_TOOLS = frozenset( + { + ToolName.CHAT_COMPLETION.value, + ToolName.CHAT_WITH_GPT.value, + } +) + + +def _chat_tool_attempts(name: str, arguments: dict) -> int: + """Retry budget for a chat-tool call (#51). + + Only the USER-send chat tools honor ``single_send`` (one attempt, no + re-invocation of a mutating send on 429). Every other chat tool keeps + the default budget regardless of any ``single_send`` argument. + """ + if name in _SINGLE_SEND_TOOLS: + return chat_retry_attempts(arguments) + return CHAT_MAX_ATTEMPTS + + # ═══════════════════════════════════════════════════════════════ # Output Schemas — structured output validation (Memory server pattern) # ═══════════════════════════════════════════════════════════════ @@ -1480,6 +1521,20 @@ def _map_tool_exception(exc: Exception) -> object: f"(rate_limit_exceeded, retry_after={exc.retry_after})"))], isError=True, ) + if isinstance(exc, SendOutcomeUnknownError): + # #51: ambiguous send outcome — surface the code, the preserved + # causal evidence, and the do-not-resend guidance. An agent must + # reconcile (read) rather than re-send. + evidence = (f", captured_user_id={exc.captured_user_id}" + if exc.captured_user_id else "") + return mcp_types.CallToolResult( + content=[mcp_types.TextContent(type="text", + text=(f"Send outcome unknown — the message may or may not have " + f"been sent. Do NOT send it again; inspect the " + f"conversation and reconcile instead" + f"{evidence}. (send_outcome_unknown, retry_safe=false)"))], + isError=True, + ) if isinstance(exc, CircuitOpenError): return mcp_types.CallToolResult( content=[mcp_types.TextContent(type="text", @@ -1649,7 +1704,11 @@ async def _call_tool_pooled( async def _run_pooled() -> dict: if name in _CHAT_TOOLS: - return await retry_on_rate_limit(driver, handler, on_progress=on_progress) + # #51: only chat_completion / chat_with_gpt honor + # single_send (_chat_tool_attempts scopes it). + return await retry_on_rate_limit( + driver, handler, on_progress=on_progress, + max_attempts=_chat_tool_attempts(name, arguments)) return await handler() if is_mutation and _lock_cdp_port is not None: @@ -1813,7 +1872,11 @@ async def _run() -> dict: # passes into the business function; here it's also used by # retry_on_rate_limit to signal the backoff pause. Same object # by design — two injection points, one notifier. - return await retry_on_rate_limit(_driver, handler, on_progress=on_progress) + # #51: only chat_completion / chat_with_gpt honor single_send + # (_chat_tool_attempts scopes it). + return await retry_on_rate_limit( + _driver, handler, on_progress=on_progress, + max_attempts=_chat_tool_attempts(name, arguments)) return await handler() # Serialize mutating tools through the cross-process lock diff --git a/src/chatgpt_web2api/resilience.py b/src/chatgpt_web2api/resilience.py index 63554888..887a67fd 100644 --- a/src/chatgpt_web2api/resilience.py +++ b/src/chatgpt_web2api/resilience.py @@ -37,6 +37,26 @@ _DEFAULT_CAP = 120 +CHAT_MAX_ATTEMPTS = 3 + + +def chat_retry_attempts(args) -> int: + """Single-send opt-in (#51): arguments carrying ``single_send`` truthy + run the chat operation exactly once (``max_attempts=1``). + + A rate limit surfacing AFTER a dispatched mutation must propagate, not + re-invoke the mutating operation. REST resolves ``single_send`` from the + request body (or its ``metadata``) before calling this; MCP passes the + tool arguments dict directly. Default (no opt-in): 3 attempts — + unchanged behavior for existing callers. + """ + try: + flag = args.get("single_send") if args is not None else None + except AttributeError: + flag = None + return 1 if flag else CHAT_MAX_ATTEMPTS + + async def retry_on_rate_limit( driver, factory: Callable[[], Awaitable[T]], diff --git a/tests/test_chatgpt_dom.py b/tests/test_chatgpt_dom.py index 7d60c12d..2cb21914 100644 --- a/tests/test_chatgpt_dom.py +++ b/tests/test_chatgpt_dom.py @@ -164,19 +164,32 @@ async def test_ensure_send_ready_routes_navigation_through_driver(): @pytest.mark.asyncio -async def test_click_send_records_success_through_driver_breaker(): - """A confirmed send must record_success via driver._breakers (half-open - recovery) — the registry stays on the driver, not the dom.""" +async def test_click_send_does_not_record_breaker_success(): + """#51 boundary change: click_send's JS return ("sent") only proves the + synthetic events ran — NOT React acceptance — so it must NOT record + COMPOSER_SEND_READINESS success. The success point moved to + send_and_stream, at submission evidence (captured UUID, or the DOM + acknowledgment fallback). A pre-existing failure count must therefore + survive a successful click untouched.""" from chatgpt_web2api.breakers import BreakerKind, BreakerRegistry dom, driver = _make_dom() reg = BreakerRegistry() driver._breakers = reg - driver._js = AsyncMock(return_value="sent") + driver._js = AsyncMock(return_value="yes") + driver._js_mutation = AsyncMock(return_value="sent") + + reg.record_failure(BreakerKind.COMPOSER_SEND_READINESS) + before = len(reg._states[BreakerKind.COMPOSER_SEND_READINESS].recent_failures) await dom.click_send() - # Not open after a success record (record_success clears failures). - assert not reg.is_open(BreakerKind.COMPOSER_SEND_READINESS) + + after = len(reg._states[BreakerKind.COMPOSER_SEND_READINESS].recent_failures) + assert after == before, ( + "click_send recorded success (cleared failure history) — the " + "breaker success boundary belongs at submission evidence in " + "send_and_stream, not at the click's JS return (#51)" + ) # ── 5. dismiss_rate_limit tri-state via driver._js_strict ──────────── @@ -283,14 +296,20 @@ async def test_click_send_waits_then_sends(monkeypatch): monkeypatch.setattr(dom_mod, "SEND_BUTTON_POLL_MAX_WAIT_S", 2.0) dom, driver = _make_dom() - # First 3 readiness checks → "no" (button not ready), then "yes", then "sent". - driver._js = AsyncMock(side_effect=["no", "no", "no", "yes", "sent"]) + # First 3 readiness checks → "no" (button not ready), then "yes"; the + # click itself now goes through _js_mutation (#51) → "sent". + driver._js = AsyncMock(side_effect=["no", "no", "no", "yes"]) + driver._js_mutation = AsyncMock(return_value="sent") await dom.click_send() # must not raise - # The readiness poll should have run 4 times (3×"no" + 1×"yes"), then the - # click once ("sent") = 5 total _js calls. - assert driver._js.await_count == 5, f"expected 5 _js calls, got {driver._js.await_count}" + # The readiness poll should have run 4 times (3×"no" + 1×"yes") on _js + # (a read — reconnect-retry is safe), then the click exactly once via + # _js_mutation (a mutation — never replayed). + assert driver._js.await_count == 4, f"expected 4 _js calls, got {driver._js.await_count}" + assert driver._js_mutation.await_count == 1 + click_expr = driver._js_mutation.await_args.args[0] + assert "dispatchEvent" in click_expr and "MouseEvent" in click_expr @pytest.mark.asyncio @@ -303,8 +322,10 @@ async def test_click_send_raises_on_budget_exhausted(monkeypatch): monkeypatch.setattr(dom_mod, "SEND_BUTTON_POLL_MAX_WAIT_S", 0.05) dom, driver = _make_dom() - # Every _js call returns "no" — button never appears. + # Every readiness poll returns "no" — button never appears; the final + # click (via _js_mutation) finds no button either. driver._js = AsyncMock(return_value="no") + driver._js_mutation = AsyncMock(return_value="no send button") from chatgpt_web2api.cdp_driver import SendReadinessError diff --git a/tests/test_composer_selectors.py b/tests/test_composer_selectors.py index bb47e763..d1155917 100644 --- a/tests/test_composer_selectors.py +++ b/tests/test_composer_selectors.py @@ -208,6 +208,10 @@ async def test_click_send_fails_when_no_send_button(monkeypatch): async def _fake_js(expr, timeout=15): return "no send button" d._js = _fake_js + # #51: the click itself is a mutation — goes through _js_mutation. + async def _fake_js_mutation(expr, timeout=15): + return "no send button" + d._js_mutation = _fake_js_mutation d._capture_selector_diagnostic = AsyncMock() # The wait-for-button loop also returns 'no', so it polls all 10 @@ -229,9 +233,14 @@ async def test_click_send_emits_new_selector_first(monkeypatch): async def _fake_js(expr, timeout=15): seen.append(expr) - # Wait-loop returns 'yes' immediately, then the click returns 'sent'. - return "yes" if "yes" in expr or "'no'" in expr else "sent" + # Wait-loop returns 'yes' immediately; the click (a #51 mutation, + # separately stubbed) returns 'sent'. + return "yes" d._js = _fake_js + async def _fake_js_mutation(expr, timeout=15): + seen.append(expr) + return "sent" + d._js_mutation = _fake_js_mutation monkeypatch.setattr("chatgpt_web2api.cdp_driver.asyncio.sleep", AsyncMock()) await d.click_send() @@ -248,7 +257,8 @@ async def _fake_js(expr, timeout=15): async def test_click_send_sent_on_success(monkeypatch): """Happy path: button present + click dispatched → 'sent' logged, no raise.""" d = _make_driver() - d._js = AsyncMock(return_value="sent") + d._js = AsyncMock(return_value="yes") + d._js_mutation = AsyncMock(return_value="sent") monkeypatch.setattr("chatgpt_web2api.cdp_driver.asyncio.sleep", AsyncMock()) # Should not raise. diff --git a/tests/test_composer_verification.py b/tests/test_composer_verification.py index c1423cc3..778ac7f6 100644 --- a/tests/test_composer_verification.py +++ b/tests/test_composer_verification.py @@ -12,13 +12,12 @@ realistic ProseMirror DOM shapes — not just mocked output strings. """ -import unicodedata from unittest.mock import AsyncMock import pytest -from chatgpt_web2api.chatgpt_dom import ChatGPTDom from chatgpt_web2api.cdp_driver import CDPDriver +from chatgpt_web2api.chatgpt_dom import ChatGPTDom def _make_dom(js_return_value=""): diff --git a/tests/test_no_replay_send.py b/tests/test_no_replay_send.py new file mode 100644 index 00000000..97d374f9 --- /dev/null +++ b/tests/test_no_replay_send.py @@ -0,0 +1,294 @@ +"""Issue #51 — deterministic replay reproduction (offline red tests + controls). + +These four tests pin the no-replay contract for chat mutations: + + 1. test_mutating_runtime_evaluate_ambiguous_send_is_not_replayed + A mutating Runtime.evaluate whose socket dies mid-dispatch must send + exactly ONE frame, must NOT reconnect-and-replay, and must raise + SendOutcomeUnknownError (reconcile-only, never resend). + + 2. test_readonly_runtime_evaluate_reconnect_retry_is_preserved + Control: a READ-only Runtime.evaluate keeps the existing resilience — + one reconnect, one retry, value returned. (#51 must not remove CDP + resilience globally.) + + 3. test_single_send_post_mutation_rate_limit_does_not_repeat_operation + With max_attempts=1 (the single-send mode), a RateLimitError raised + AFTER the factory mutated must propagate with exactly one invocation. + + 4. test_default_retry_mode_remains_backward_compatible + Default mode still retries a failing factory (3-attempt budget, + success on attempt 2). Behavior unchanged for non-single-send callers. + +Evidence protocol: test 1 fails against current master (no mutation +primitive / replay occurs); the failure output plus the mechanical +characterization (2 frames, 1 reconnect through plain _cdp) is preserved in +captures/phase0/ as the defect evidence for issue #51. +""" + +from __future__ import annotations + +import asyncio +import json +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from chatgpt_web2api.cdp_driver import RateLimitError +from chatgpt_web2api.cdp_transport import CDPTransport +from chatgpt_web2api.resilience import retry_on_rate_limit + + +class ConnectionClosedError(Exception): + """Local stand-in socket-death error. CDPTransport._should_reconnect + matches by exception CLASS NAME ('ConnectionClosedError' is in its name + set), so no real websockets exception is needed.""" + + +def _make_transport(): + """Mirror test_cdp_transport._make_transport: transport + mock driver. + ``driver._cdp`` is wired to the transport's real method so _js routes + through the genuine send/reconnect/future-table path.""" + driver = MagicMock() + driver._ws = MagicMock() + driver._ws.send = AsyncMock() + driver._ws.recv = AsyncMock() + driver._msg_id = 0 + driver._pending = {} + driver.reconnect = AsyncMock() + transport = CDPTransport(driver) + driver._cdp = transport._cdp + return transport, driver + + +class AmbiguousSendSocket: + """Socket that records every frame as delivered, then raises a + reconnect-class error on the FIRST send. The peer may already have + received/executed that frame — the ambiguity #51 is about. + + After driver.reconnect() swaps in a healthy socket, subsequent sends + resolve their own pending future (no reader loop needed). + """ + + def __init__(self, transport): + self.transport = transport + self.sent: list[dict] = [] + + async def send(self, raw): + self.sent.append(json.loads(raw)) + if len(self.sent) == 1: + raise ConnectionClosedError("connection closed") + driver = self.transport._driver + mid = self.sent[-1]["id"] + fut = driver._pending.get(mid) + if fut is not None and not fut.done(): + fut.set_result({ + "id": mid, + "result": {"result": {"type": "string", "value": "sent"}}, + }) + + async def recv(self): + await asyncio.sleep(3600) + + async def close(self): + pass + + +class HealthySocket: + """Answers every sent frame from its own pending table.""" + + def __init__(self, transport, value="ok"): + self.transport = transport + self.value = value + self.sent: list[dict] = [] + + async def send(self, raw): + self.sent.append(json.loads(raw)) + driver = self.transport._driver + mid = self.sent[-1]["id"] + fut = driver._pending.get(mid) + if fut is not None and not fut.done(): + fut.set_result({ + "id": mid, + "result": {"result": {"type": "string", "value": self.value}}, + }) + + async def recv(self): + await asyncio.sleep(3600) + + async def close(self): + pass + + +async def test_mutating_runtime_evaluate_ambiguous_send_is_not_replayed(): + """RED against current master. The mutation primitive (_js_mutation) + must send exactly one frame, never reconnect-and-replay it, and raise + SendOutcomeUnknownError so the caller reconciles instead of resending.""" + from chatgpt_web2api.cdp_driver import SendOutcomeUnknownError + + transport, driver = _make_transport() + ambiguous = AmbiguousSendSocket(transport) + healthy = HealthySocket(transport, value="sent") + driver._ws = ambiguous + + reconnect_calls = {"n": 0} + + async def reconnect(): + reconnect_calls["n"] += 1 + driver._ws = healthy + + driver.reconnect = reconnect + + with pytest.raises(SendOutcomeUnknownError) as exc_info: + await transport._js_mutation("(function(){ return 'sent'; })()") + + err = exc_info.value + assert getattr(err, "effect_state", None) == "unknown" + assert getattr(err, "retry_safe", None) is False + assert getattr(err, "reconciliation_safe", None) is True + + mutation_frames = [f for f in ambiguous.sent + healthy.sent + if f.get("method") == "Runtime.evaluate"] + assert len(mutation_frames) == 1, ( + f"mutation replayed: {len(mutation_frames)} Runtime.evaluate frames " + f"sent (ambiguous={len(ambiguous.sent)}, healthy={len(healthy.sent)})") + # Zero reconnects on the mutation path — the whole point of #51. Counted + # explicitly (an earlier version asserted socket identity, which was + # tautological in this fixture). + assert reconnect_calls["n"] == 0 + + +async def test_readonly_runtime_evaluate_reconnect_retry_is_preserved(): + """Control (green before and after): a read via _js keeps the existing + one-reconnect-one-retry resilience and returns the value.""" + transport, driver = _make_transport() + ambiguous = AmbiguousSendSocket(transport) + healthy = HealthySocket(transport, value="read-result") + driver._ws = ambiguous + + async def reconnect(): + driver._ws = healthy + + driver.reconnect = reconnect + + value = await transport._js("(function(){ return 'read-result'; })()") + + assert value == "read-result" + total_frames = [f for f in ambiguous.sent + healthy.sent + if f.get("method") == "Runtime.evaluate"] + assert len(total_frames) == 2 # first (failed) + retried (succeeded) + assert len(healthy.sent) == 1 # exactly one retry after reconnect + + +async def test_single_send_post_mutation_rate_limit_does_not_repeat_operation(): + """Single-send mode (max_attempts=1): a RateLimitError raised AFTER the + factory already mutated must propagate, with exactly one invocation — + the mutation is never repeated.""" + calls = {"attempts": 0, "mutations": 0} + driver = MagicMock() + driver.dismiss_rate_limit = AsyncMock(return_value=True) + + async def mutating_operation(): + calls["attempts"] += 1 + calls["mutations"] += 1 # the USER send happened + if calls["attempts"] == 1: + raise RateLimitError("rate limited", retry_after=0.01) + return "ok" + + with pytest.raises(RateLimitError): + await retry_on_rate_limit(driver, mutating_operation, max_attempts=1) + + assert calls["attempts"] == 1 + assert calls["mutations"] == 1 + + +# ── Cancellation semantics (Codex P1 review) ───────────────────────── + + +class HangingSendSocket: + """Socket whose send() records the frame and then hangs until cancelled + — the frame WAS handed toward the transport before cancellation.""" + + def __init__(self): + self.sent: list[dict] = [] + self.entered = asyncio.Event() + + async def send(self, raw): + self.sent.append(json.loads(raw)) + self.entered.set() + await asyncio.Event().wait() # never set — hang until cancelled + + async def recv(self): + await asyncio.sleep(3600) + + async def close(self): + pass + + +async def test_cancelled_mutation_propagates_cancellation_and_records_unknown(): + """Cancelling DURING the mutating evaluate: the frame may already have + reached Chrome, so this is outcome-unknown — but asyncio's cancellation + contract forbids converting CancelledError into another type. The + deliberate design: CancelledError propagates UNCHANGED and the + UNKNOWN/do-not-resend classification is recorded on the driver side + channel (note_send_outcome_unknown).""" + transport, driver = _make_transport() + sock = HangingSendSocket() + driver._ws = sock + + task = asyncio.get_running_loop().create_task( + transport._js_mutation("(function(){ return 'sent'; })()")) + await asyncio.wait_for(sock.entered.wait(), timeout=2) + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + driver.note_send_outcome_unknown.assert_called_once() + kwargs = driver.note_send_outcome_unknown.call_args.kwargs + assert kwargs.get("stage") == "send_mutation" + frames = [f for f in sock.sent if f.get("method") == "Runtime.evaluate"] + assert len(frames) == 1 # still exactly one mutation frame + driver.reconnect.assert_not_awaited() + # Cancellation cleanup (round-2 review): CancelledError must not leak + # the pending-future entry — with a hanging socket it would stay forever. + assert driver._pending == {}, ( + f"pending table leaked on cancellation: {list(driver._pending)}") + + +async def test_cancelled_read_does_not_record_unknown(): + """Scope check: cancellation during a READ (_js) also propagates + unchanged, but must NOT be classified as an outcome-unknown mutation.""" + transport, driver = _make_transport() + sock = HangingSendSocket() + driver._ws = sock + + task = asyncio.get_running_loop().create_task( + transport._js("(function(){ return 'r'; })()")) + await asyncio.wait_for(sock.entered.wait(), timeout=2) + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + driver.note_send_outcome_unknown.assert_not_called() + # Cancellation cleanup applies to the read path identically. + assert driver._pending == {}, ( + f"pending table leaked on cancellation: {list(driver._pending)}") + + +async def test_default_retry_mode_remains_backward_compatible(): + """Default mode: the wrapper still retries (success on attempt 2), for + callers that did not opt into single-send.""" + calls = {"attempts": 0} + driver = MagicMock() + driver.dismiss_rate_limit = AsyncMock(return_value=True) + + async def flaky_operation(): + calls["attempts"] += 1 + if calls["attempts"] == 1: + raise RateLimitError("rate limited", retry_after=0.01) + return "ok" + + result = await retry_on_rate_limit(driver, flaky_operation) + + assert result == "ok" + assert calls["attempts"] == 2 diff --git a/tests/test_send_acknowledgment.py b/tests/test_send_acknowledgment.py index 9fdf70e2..5cee63e9 100644 --- a/tests/test_send_acknowledgment.py +++ b/tests/test_send_acknowledgment.py @@ -18,14 +18,13 @@ Fix 2: include last_result.diagnostic in TurnReconciliationError. """ -import asyncio import json from unittest.mock import AsyncMock, MagicMock import pytest -from chatgpt_web2api.cdp_driver import CDPDriver, SendReadinessError -from chatgpt_web2api.turn_anchor import TurnReconciliationError, TurnTextResult +from chatgpt_web2api.cdp_driver import CDPDriver +from chatgpt_web2api.turn_anchor import TurnReconciliationError def _make_driver(): @@ -49,7 +48,6 @@ async def test_send_not_acknowledged_raises_when_no_signals(monkeypatch): """When click_send fires but no acknowledgment appears (no UUID, no DOM count increase, composer not cleared), the bridge must raise a typed error instead of silently entering completion detection.""" - from chatgpt_web2api.cdp_driver import CDPDriver driver = _make_driver() # Mock the send path @@ -106,9 +104,6 @@ async def test_send_acknowledged_when_user_count_increases(monkeypatch): anchor = TurnAnchor(sent_text="test", mode="fresh_chat") driver._capture_pre_send_fallback_anchor = AsyncMock(return_value=anchor) - # After send: user count goes from 0 to 1 (message landed) - poll_count = {"n": 0} - async def fake_js_strict(expr, timeout=15): # Send acknowledgment check: user count + composer present + empty if "userCount" in expr and "composerEmpty" in expr: @@ -126,7 +121,6 @@ async def fake_js_strict(expr, timeout=15): driver._js_strict = fake_js_strict # Mock the detector to return immediately - from chatgpt_web2api.completion_detector import CompletionDetector driver._completion = MagicMock() driver._completion.stream_until_complete = MagicMock() diff --git a/tests/test_send_outcome_unknown.py b/tests/test_send_outcome_unknown.py new file mode 100644 index 00000000..2558497a --- /dev/null +++ b/tests/test_send_outcome_unknown.py @@ -0,0 +1,328 @@ +"""#51 evidence-preservation and boundary tests. + +Three groups pinning the bridge between the transport fix and the future +UNKNOWN→CONFIRMED reconciliation path, plus the breaker's new success +boundary and the single_send scoping: + + A. An ambiguous click (SendOutcomeUnknownError) preserves an in-flight + captured UUID on the error before the capture scope closes. + B. COMPOSER_SEND_READINESS success records exactly at submission evidence + (captured UUID, or DOM ack True) and NOT on an unknown outcome. + C. single_send is honored by chat_completion / chat_with_gpt only; + create_memory keeps the default retry budget even if the argument + appears. + +These tests drive the REAL send_and_stream orchestration with mocked +collaborators (constructed like test_composer_selectors._make_driver). +The transport-level frame guarantees (one mutation frame, no replay) are +covered in test_no_replay_send.py and are not duplicated here. +""" + +from __future__ import annotations + +import asyncio +from unittest.mock import AsyncMock, MagicMock + +from chatgpt_web2api.breakers import BreakerKind, BreakerRegistry +from chatgpt_web2api.cdp_driver import ( + AMBIGUOUS_CAPTURE_WINDOW_S, + CDPDriver, + SendOutcomeUnknownError, +) + + +class _StopStream(Exception): + """Marker raised by the mocked completion stream to end send_and_stream + right after the breaker-success point — everything past it (URL wait, + reconciliation) is out of scope for these tests.""" + + +def _make_send_driver(*, captured_uuid, click_raises=None, dom_ack=None, + stream="raise"): + """A CDPDriver whose send_and_stream collaborators are mocked so the + orchestration up to (and including) the breaker-success point runs + genuinely. stream="raise" ends at the boundary point via _StopStream; + stream="complete" runs a full successful turn through reconciliation.""" + from types import SimpleNamespace + + d = CDPDriver(cdp_port=9222) + d._ws = MagicMock() + reg = BreakerRegistry() + d._breakers = reg + successes: list[BreakerKind] = [] + _orig = reg.record_success + reg.record_success = lambda kind: (successes.append(kind), _orig(kind))[1] # type: ignore[method-assign] + + d._assert_owned_tab_required = MagicMock() # sync seam — must not be AsyncMock + d._read_assistant_count_baseline = AsyncMock(return_value=0) + d._capture_pre_send_fallback_anchor = AsyncMock(return_value=MagicMock()) + + listener = MagicMock() + listener.reenable_if_stale = AsyncMock() + listener.is_alive.return_value = True + scope = MagicMock() + listener.arm_capture_scope.return_value = scope + listener.wait_for_captured_uuid = AsyncMock(return_value=captured_uuid) + d._identity_listener = listener + + d.type_message = AsyncMock() + if click_raises is not None: + d.click_send = AsyncMock(side_effect=click_raises) + else: + d.click_send = AsyncMock() + d._verify_send_acknowledged = AsyncMock(return_value=dom_ack) + + if stream == "raise": + async def _raising_stream(**kwargs): + raise _StopStream() + yield # pragma: no cover — makes this an async generator + + d._completion.stream_until_complete = _raising_stream + else: + async def _completing_stream(**kwargs): + yield MagicMock() + + d._completion.stream_until_complete = _completing_stream + d._completion.last_dom_text = "" + d._completion.had_non_text_content = False + d._js_strict = AsyncMock( + return_value="https://chatgpt.com/c/conv-p2") + d._fetch_text_for_turn = AsyncMock(return_value=SimpleNamespace( + status="matched", text="hello world", diagnostic=None)) + return d, listener, scope, successes + + +async def _driven_send(driver): + """Consume send_and_stream; return the exception it terminated with.""" + try: + async for _ in driver.send_and_stream("sentinel-text"): + pass + except Exception as e: # _StopStream or the error under test + return e + return None + + +# ── A. UUID preservation on the ambiguous path ────────────────────── + +async def test_ambiguous_click_preserves_captured_uuid_before_scope_close(): + """The click frame may have executed remotely; the listener may hold or + be resolving the outgoing UUID. send_and_stream must attach it to the + SendOutcomeUnknownError before the finally closes the capture scope.""" + err = SendOutcomeUnknownError("ambiguous dispatch") + d, listener, scope, successes = _make_send_driver( + captured_uuid="uuid-causal-1", click_raises=err) + + raised = await _driven_send(d) + + assert raised is err + assert err.captured_user_id == "uuid-causal-1" + # The preservation wait used the short bounded window, not the normal 5s + listener.wait_for_captured_uuid.assert_awaited_once() + assert listener.wait_for_captured_uuid.await_args.kwargs.get( + "timeout") == AMBIGUOUS_CAPTURE_WINDOW_S + # The capture scope DID close (finally) — evidence was preserved first. + scope.close.assert_called_once() + # Unknown outcome is never submission evidence → no breaker success. + assert successes == [] + + +async def test_ambiguous_click_without_capture_leaves_field_none(): + """No UUID resolved in the window → captured_user_id stays None and the + error still propagates (reconcile-only, evidence-less UNKNOWN).""" + err = SendOutcomeUnknownError("ambiguous dispatch") + d, listener, scope, successes = _make_send_driver( + captured_uuid=None, click_raises=err) + + raised = await _driven_send(d) + + assert raised is err + assert err.captured_user_id is None + assert successes == [] + + +# ── B. Breaker success at the new boundary ────────────────────────── + +async def test_breaker_success_on_captured_uuid(): + """Captured UUID (primary submission evidence) → exactly one + COMPOSER_SEND_READINESS record_success; the DOM ack probe is skipped.""" + d, listener, scope, successes = _make_send_driver(captured_uuid="uuid-1") + + raised = await _driven_send(d) + + assert isinstance(raised, _StopStream) # got past the boundary + assert successes == [BreakerKind.COMPOSER_SEND_READINESS] + d._verify_send_acknowledged.assert_not_awaited() + + +async def test_breaker_success_on_dom_ack_fallback(): + """No UUID, DOM acknowledgment True (fallback evidence) → one + record_success.""" + d, listener, scope, successes = _make_send_driver( + captured_uuid=None, dom_ack=True) + + raised = await _driven_send(d) + + assert isinstance(raised, _StopStream) + assert successes == [BreakerKind.COMPOSER_SEND_READINESS] + d._verify_send_acknowledged.assert_awaited_once() + + +async def test_breaker_no_success_on_unknown_outcome(): + """Covered by group A assertions; explicit here for the boundary trio: + an ambiguous send records no success (outcome unproven).""" + d, listener, scope, successes = _make_send_driver( + captured_uuid="uuid-2", + click_raises=SendOutcomeUnknownError("ambiguous dispatch")) + + raised = await _driven_send(d) + + assert isinstance(raised, SendOutcomeUnknownError) + assert successes == [] + + +# ── C2. Breaker downstream-success fallback (Codex P2 review) ──────── + +async def test_inconclusive_ack_completed_turn_clears_breaker_failures(): + """No UUID + an inconclusive DOM probe (acknowledged=None) + a turn that + reaches FULL completion → the fallback records COMPOSER_SEND_READINESS + success exactly once, so successful sends clear prior failure history.""" + from chatgpt_web2api.breakers import BreakerKind + + d, listener, scope, successes = _make_send_driver( + captured_uuid=None, dom_ack=None, stream="complete") + d.last_send_outcome_unknown = {"stale": True} # cleared at send start + + raised = await _driven_send(d) + + assert raised is None # full successful turn + assert successes == [BreakerKind.COMPOSER_SEND_READINESS] + assert d.last_send_outcome_unknown is None # side channel cleared at start + + +async def test_uuid_evidence_completed_turn_records_success_once(): + """UUID already recorded the success at submission evidence; the + downstream fallback must not double-record.""" + from chatgpt_web2api.breakers import BreakerKind + + d, listener, scope, successes = _make_send_driver( + captured_uuid="uuid-1", stream="complete") + + raised = await _driven_send(d) + + assert raised is None + assert successes == [BreakerKind.COMPOSER_SEND_READINESS] + + +async def test_inconclusive_ack_failed_stream_records_no_success(): + """An inconclusive probe plus a stream that RAISES is not evidence — + no breaker success (fallback unreachable on failure).""" + d, listener, scope, successes = _make_send_driver( + captured_uuid=None, dom_ack=None, stream="raise") + + raised = await _driven_send(d) + + assert isinstance(raised, _StopStream) + assert successes == [] + + +# ── C. single_send scoping ─────────────────────────────────────────── + +def test_single_send_scoped_to_chat_send_tools(): + """chat_completion / chat_with_gpt honor single_send; create_memory + keeps the default budget even when the argument is present.""" + from chatgpt_web2api.mcp_server import _chat_tool_attempts + + assert _chat_tool_attempts( + "chat_completion", {"single_send": True}) == 1 + assert _chat_tool_attempts( + "chat_with_gpt", {"single_send": True}) == 1 + # Non-send chat tool: scope isolation — the flag is ignored. + assert _chat_tool_attempts( + "create_memory", {"single_send": True}) == 3 + # Defaults unchanged when the flag is absent. + assert _chat_tool_attempts("chat_completion", {}) == 3 + assert _chat_tool_attempts("create_memory", {}) == 3 + + +# ── D. Surface mappings preserve the UNKNOWN distinction ──────────── + +def test_rest_error_response_maps_unknown_to_409_with_evidence(): + """#51: an ambiguous outcome must not surface as a bare 500 (the status + class SDKs blindly retry). It maps to 409 with code send_outcome_unknown, + retry_safe=false, and the preserved captured_user_id when present. The + x-should-retry:false header is load-bearing: the official OpenAI SDKs + check it BEFORE their status rules and auto-retry 409 otherwise.""" + import json as _json + + from chatgpt_web2api.api_server import APIServer + + server = object.__new__(APIServer) # _error_response is pure (no self use) + + exc = SendOutcomeUnknownError("ambiguous dispatch") + exc.captured_user_id = "uuid-causal-9" + resp = server._error_response(exc) + assert resp.status == 409 + assert resp.headers["x-should-retry"] == "false" + assert "Retry-After" not in resp.headers + body = _json.loads(resp.text) + assert body["error"]["code"] == "send_outcome_unknown" + assert body["error"]["retry_safe"] is False + assert body["error"]["captured_user_id"] == "uuid-causal-9" + + +def test_rest_error_response_unknown_without_evidence_omits_field(): + import json as _json + + from chatgpt_web2api.api_server import APIServer + + server = object.__new__(APIServer) + resp = server._error_response(SendOutcomeUnknownError("ambiguous dispatch")) + assert resp.status == 409 + assert resp.headers["x-should-retry"] == "false" + assert "Retry-After" not in resp.headers + body = _json.loads(resp.text) + assert body["error"]["code"] == "send_outcome_unknown" + assert "captured_user_id" not in body["error"] + + +def test_mcp_maps_unknown_to_structured_do_not_resend_error(): + from chatgpt_web2api.mcp_server import _map_tool_exception + + exc = SendOutcomeUnknownError("ambiguous dispatch") + exc.captured_user_id = "uuid-causal-7" + result = _map_tool_exception(exc) + assert result is not None and result.isError + text = result.content[0].text + assert "send_outcome_unknown" in text + assert "uuid-causal-7" in text + assert "NOT send it again" in text + + +# ── E. Cancellation during the ambiguity window ───────────────────── + +async def test_cancellation_during_ambiguity_window_still_closes_scope(): + """Cancelling while the preservation wait is in flight must propagate + CancelledError (never swallowed by the evidence wait) and still reach + the finally that closes the capture scope.""" + err = SendOutcomeUnknownError("ambiguous dispatch") + d, listener, scope, successes = _make_send_driver( + captured_uuid=None, click_raises=err) + + async def slow_capture(**kwargs): + await asyncio.sleep(5) + return "too-late-uuid" + + listener.wait_for_captured_uuid = slow_capture + + task = asyncio.get_running_loop().create_task(_driven_send(d)) + await asyncio.sleep(0.3) # inside the 1s window wait now + task.cancel() + try: + await task + raised = None + except asyncio.CancelledError: + raised = "cancelled" + + assert raised == "cancelled" # the wait did not convert cancellation + scope.close.assert_called_once() # finally still ran + assert successes == [] diff --git a/tests/test_send_readiness.py b/tests/test_send_readiness.py index ef6500e2..196a0b40 100644 --- a/tests/test_send_readiness.py +++ b/tests/test_send_readiness.py @@ -117,6 +117,8 @@ async def capturing_js_strict(expr, timeout=15): return _diag_json(send_candidates_count=3, enabled_send_candidates_count=1) driver._js = _make_scripted_js(poll_result="no", send_result="no send button") + # #51: the click is dispatched through _js_mutation (mutation — never replayed). + driver._js_mutation = AsyncMock(return_value="no send button") driver._js_strict = capturing_js_strict with pytest.raises(SendReadinessError): @@ -143,6 +145,7 @@ async def test_send_succeeds_with_submit_type_fallback(): dom, driver = _make_dom() driver._js = _make_scripted_js(poll_result="yes", send_result="sent") + driver._js_mutation = AsyncMock(return_value="sent") await dom.click_send() # should NOT raise @@ -158,6 +161,8 @@ async def test_error_distinguishes_empty_composer_from_missing_button(caplog): dom, driver = _make_dom() driver._js = _make_scripted_js(poll_result="no", send_result="no send button") + # #51: the click is dispatched through _js_mutation (mutation — never replayed). + driver._js_mutation = AsyncMock(return_value="no send button") driver._js_strict = AsyncMock(return_value=_diag_json( composer_text_length=0, # EMPTY — injection failed! )) @@ -187,6 +192,8 @@ async def test_diagnostic_captures_stop_button_when_generating(caplog): dom, driver = _make_dom() driver._js = _make_scripted_js(poll_result="no", send_result="no send button") + # #51: the click is dispatched through _js_mutation (mutation — never replayed). + driver._js_mutation = AsyncMock(return_value="no send button") driver._js_strict = AsyncMock(return_value=_diag_json( stop_button_present=True, generating_indicator_present=True, diff --git a/tests/test_single_send_propagation.py b/tests/test_single_send_propagation.py new file mode 100644 index 00000000..d30a50aa --- /dev/null +++ b/tests/test_single_send_propagation.py @@ -0,0 +1,184 @@ +"""#51 propagation tests — single_send reaches the retry seam on both servers. + +The wrapper-level behavior (max_attempts=1 → one invocation) is covered in +test_no_replay_send.py. These tests pin the SEAMS: + + - resilience.chat_retry_attempts maps arguments → attempts + - api_server._resolve_single_send maps the REST body (top level or + metadata) → the single_send flag + - the wrapper + helper combination keeps default callers at 3 attempts + and single-send callers at exactly one business invocation +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from chatgpt_web2api.api_server import _resolve_single_send +from chatgpt_web2api.cdp_driver import RateLimitError +from chatgpt_web2api.resilience import chat_retry_attempts, retry_on_rate_limit + +# ── resilience helper ──────────────────────────────────────────────── + +def test_chat_retry_attempts_default_is_three(): + assert chat_retry_attempts({}) == 3 + assert chat_retry_attempts({"single_send": False}) == 3 + assert chat_retry_attempts(None) == 3 + + +def test_chat_retry_attempts_single_send_is_one(): + assert chat_retry_attempts({"single_send": True}) == 1 + + +# ── REST seam ──────────────────────────────────────────────────────── + +def test_resolve_single_send_from_top_level(): + assert _resolve_single_send({"single_send": True, "messages": []}) is True + + +def test_resolve_single_send_from_metadata(): + assert _resolve_single_send( + {"metadata": {"single_send": True, "project_id": "g-p-x"}}) is True + + +def test_resolve_single_send_absent_is_false(): + assert _resolve_single_send({"messages": []}) is False + assert _resolve_single_send({"metadata": {}}) is False + + +# ── seam → wrapper combination (REST shape) ────────────────────────── + +async def test_rest_single_send_shape_runs_once_on_post_mutation_429(): + calls = {"mutations": 0} + driver = MagicMock() + driver.dismiss_rate_limit = AsyncMock(return_value=True) + + async def send_and_collect(): + calls["mutations"] += 1 # the USER send was dispatched + raise RateLimitError("rate limited", retry_after=0.01) + + body = {"messages": [{"role": "user", "content": "x"}], + "metadata": {"single_send": True}} + with pytest.raises(RateLimitError): + await retry_on_rate_limit( + driver, send_and_collect, + max_attempts=chat_retry_attempts( + {"single_send": _resolve_single_send(body)}), + ) + + assert calls["mutations"] == 1 + + +# ── seam → wrapper combination (MCP shape) ─────────────────────────── + +async def test_mcp_single_send_argument_runs_once_on_post_mutation_429(): + calls = {"mutations": 0} + driver = MagicMock() + driver.dismiss_rate_limit = AsyncMock(return_value=True) + + async def chat_handler(): + calls["mutations"] += 1 + raise RateLimitError("rate limited", retry_after=0.01) + + arguments = {"message": "x", "single_send": True} + with pytest.raises(RateLimitError): + await retry_on_rate_limit( + driver, chat_handler, + max_attempts=chat_retry_attempts(arguments), + ) + + assert calls["mutations"] == 1 + + +async def test_mcp_default_argument_still_retries(): + calls = {"attempts": 0} + driver = MagicMock() + driver.dismiss_rate_limit = AsyncMock(return_value=True) + + async def chat_handler(): + calls["attempts"] += 1 + if calls["attempts"] == 1: + raise RateLimitError("rate limited", retry_after=0.01) + return {"ok": True} + + arguments = {"message": "x"} # no opt-in → default behavior + result = await retry_on_rate_limit( + driver, chat_handler, max_attempts=chat_retry_attempts(arguments)) + + assert result == {"ok": True} + assert calls["attempts"] == 2 + + +# ── Streaming path: the mutating generator is consumed exactly once ── +# (GitWire asserted streaming might wrap the send in retry_on_rate_limit. +# It does not: _stream_response retries only the read-only rate-limit +# preflight, then consumes send_and_stream once. Pinned behaviorally.) + +def _stream_test_server(driver): + from chatgpt_web2api.api_server import APIServer + from chatgpt_web2api.breakers import BreakerRegistry + from chatgpt_web2api.config import Config + + server = APIServer.__new__(APIServer) + server._config = Config.load(None) + server._breakers = BreakerRegistry() + server._last_conv_id = None + server._last_project_id = None + server._last_successful_send_at = None + server._driver = driver + driver._js_strict = AsyncMock(return_value='{"text":"normal text"}') + return server + + +async def test_stream_response_consumes_send_and_stream_exactly_once(): + from aiohttp.test_utils import make_mocked_request + + from chatgpt_web2api.cdp_driver import StreamChunk + + driver = MagicMock() + driver._current_conv_id = "conv-stream-1" + sends = {"n": 0} + + async def send_and_stream(text, timeout=120, *, budgets=None, model=None): + sends["n"] += 1 + yield StreamChunk(delta="hello") + yield StreamChunk(delta="", finish_reason="stop") + + driver.send_and_stream = send_and_stream + server = _stream_test_server(driver) + + request = make_mocked_request("POST", "/v1/chat/completions") + resp = await server._stream_response(request, "auto", "hi", 30) + + assert sends["n"] == 1 + assert resp.status == 200 + + +async def test_stream_response_rate_limit_midstream_does_not_restart_send(): + """The GitWire-disproving property: a RateLimitError surfacing AFTER + the mutating generator started must not re-invoke it — streaming has + no outer retry around the send (the inline SSE marker is the channel).""" + from aiohttp.test_utils import make_mocked_request + + from chatgpt_web2api.cdp_driver import RateLimitError as RL + from chatgpt_web2api.cdp_driver import StreamChunk + + driver = MagicMock() + driver._current_conv_id = "conv-stream-2" + sends = {"n": 0} + + async def send_and_stream(text, timeout=120, *, budgets=None, model=None): + sends["n"] += 1 + yield StreamChunk(delta="partial") + raise RL("rate limited", retry_after=30) # mid-stream throttle + + driver.send_and_stream = send_and_stream + server = _stream_test_server(driver) + + request = make_mocked_request("POST", "/v1/chat/completions") + resp = await server._stream_response(request, "auto", "hi", 30) + + assert sends["n"] == 1 # NOT restarted — one business-send invocation + assert resp.status == 200 # SSE committed; the marker chunk is the channel