diff --git a/docs/specs/totalreclaw/client-consistency.md b/docs/specs/totalreclaw/client-consistency.md index 63759911..e8ac9edb 100644 --- a/docs/specs/totalreclaw/client-consistency.md +++ b/docs/specs/totalreclaw/client-consistency.md @@ -222,11 +222,11 @@ A UserOp submission (`eth_sendUserOperation`) that would take the account over i Every client must: 1. **Classify it.** Match `error_code` then `error` by string equality, on status 403 only. Anything else (another 403 code such as `call_target_not_allowed`, a non-JSON body, a 5xx) stays a plain HTTP error. -2. **Report per-fact outcomes on a batch.** Which facts landed before the refusal, so the caller neither drops stored facts nor re-stores them. +2. **Report per-fact outcomes on a batch.** Which facts landed before the refusal, so the caller neither drops stored facts nor re-stores them. The same applies to a batch that partly fails for any other reason, and to a batch where a group was skipped (#751, #753). 3. **Tell the user.** An explicit write tool returns a structured result (`error_code: "quota_exceeded"`, plus tier / usage / reset date from `/v1/billing/status` and `upgrade_url`), never the raw HTTP text. A background write (auto-extraction, mirrors) injects a notice into the next turn, at most once per session. 4. **Not lose automatic writes.** Keep refused auto-extracted facts and the session summary (Crystal) on the device (encrypted) and store them once a write succeeds again (reset or upgrade). Facts are deduplicated against the vault first. Each keeps the time it was refused as `created_at`, so it groups with its conversation. A queued Crystal is stored as it would have been at session end, minus any part the user has forgotten since. -Python/Hermes is the reference implementation (#747, #749): `relay.RelayWriteQuotaExceeded` / `raise_for_bundler_status`, `TotalReclaw.remember_batch` (`batch_results`), `agent/pending_store.py`, `agent/write_quota.py`, `agent/lifecycle.py::drain_pending_stores`. MCP returns a structured tool error and has no auto-extraction to queue. The OpenClaw plugin misreports it (#550). ZeroClaw maps it to `QuotaExceeded`. +Python/Hermes is the reference implementation (#747, #749, #753): `relay.RelayWriteQuotaExceeded` / `raise_for_bundler_status`, `TotalReclaw.remember_batch` (`batch_results` on `BatchIds`, `RelayWriteQuotaExceeded` and `BatchStoreError`; `client.per_fact_results`), `agent/pending_store.py`, `agent/write_quota.py`, `agent/lifecycle.py::drain_pending_stores`. MCP returns a structured tool error and has no auto-extraction to queue. The OpenClaw plugin misreports it (#550). ZeroClaw maps it to `QuotaExceeded`. ### Session Debrief diff --git a/python/CHANGELOG.md b/python/CHANGELOG.md index 0c37491f..d877e468 100644 --- a/python/CHANGELOG.md +++ b/python/CHANGELOG.md @@ -10,6 +10,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **A batch that partly fails no longer loses the facts it did store (#753).** When one group of a `remember_batch` call was stored and another failed for a reason other than the write quota, the call raised a plain `RuntimeError` that did not say which facts landed. Callers treated the whole batch as failed: + - Auto-extraction logged stored facts as "not stored" and did not record them. An `UPDATE` whose replacement was stored kept its old fact, so both versions stayed in the vault. + - The write-quota retry queue (#747) put stored facts back in the queue, where the next drain could write them a second time. + - Recrystallize lost the ids of a session's facts that had landed, so a resumed run wrote them again. + + Now: + - The error is a `BatchStoreError`, still a `RuntimeError`. It carries `batch_results`, with one entry per input fact holding the id or `None`, the same shape as `BatchIds.batch_results` and `RelayWriteQuotaExceeded.batch_results`. The new `totalreclaw.client.per_fact_results(outcome, n)` reads that per-fact outcome from a return value or an error. + - Auto-extraction records the stored facts, tombstones the old fact of a stored `UPDATE`, and reports only the facts that failed. A partial success counts as a successful write, so queued facts get their retry. + - The retry queue keeps only the facts that failed. + - Recrystallize saves the facts that landed in its checkpoint after every batch, including a batch that then failed. A resumed run writes only the facts still missing, even after a hard kill between batches. + - **Batch fact ids now stay with their own facts when part of a batch is skipped (#751).** `remember_batch` returns only the ids of the facts it stored. When a group was skipped as a duplicate, the list got shorter, and callers that matched ids to facts by position moved every later id onto the wrong fact. In auto-extraction, an `UPDATE` whose replacement was skipped could still delete (tombstone) the old fact, which lost data. In the write-quota retry queue (#747), skipped facts were dropped silently. - `remember_batch` still returns a `list[str]` of stored ids, now a `BatchIds` with a `batch_results` attribute. It has one entry per input fact, holding the id or `None` when that fact was skipped, the same shape as `RelayWriteQuotaExceeded.batch_results`. Auto-extraction and the retry queue match ids to facts through it. - A queued fact the batch writer skips goes back to the queue with one more attempt counted, and is dropped at the existing attempt limit. diff --git a/python/src/totalreclaw/agent/lifecycle.py b/python/src/totalreclaw/agent/lifecycle.py index a63f453f..0179ad8c 100644 --- a/python/src/totalreclaw/agent/lifecycle.py +++ b/python/src/totalreclaw/agent/lifecycle.py @@ -46,6 +46,7 @@ ) from ..forget_guard import ForgetLedger, is_forget_request from ..pii import auto_storable, drop_sensitive_facts +from ..client import per_fact_results from ..relay import RelayReadBlocked, RelayWriteQuotaExceeded from ..write_gate import ( decide_write, @@ -755,8 +756,7 @@ def _auto_extract_inner( # #747: the relay refused the batch for quota. Keep what landed # before the refusal, queue the rest of this chunk AND every later # chunk (they would be refused too) for a retry, and tell the user. - results = list(e.batch_results or []) - results += [None] * (len(chunk) - len(results)) + results = per_fact_results(e, len(chunk)) _process_batch_results(client, chunk, results, stored_texts, log_missing=False) refused = [fd for (_, _, fd), fid in zip(chunk, results) if not fid] refused += [ @@ -778,17 +778,23 @@ def _auto_extract_inner( state.invalidate_billing_cache() except Exception: pass - # On a total batch failure, none of the facts in this chunk landed. - # Log each one individually so the caller can diagnose. - for fact, _, _ in chunk: - logger.warning( - "Fact not stored (batch failure): %s", fact.text[:80] - ) + # Other groups of the chunk may have landed (#753): keep those + # (record them, tombstone the old fact of a stored UPDATE) and + # report only the facts that were not stored. + results = per_fact_results(e, len(chunk)) + _process_batch_results(client, chunk, results, stored_texts, log_missing=False) + for (fact, _, _), fact_id in zip(chunk, results): + if not fact_id: + logger.warning( + "Fact not stored (batch failure): %s", fact.text[:80] + ) + if any(results): + wrote_ok = True continue wrote_ok = True _process_batch_results( - client, chunk, _per_fact_ids(fact_ids, len(chunk)), stored_texts, log_missing=True + client, chunk, per_fact_results(fact_ids, len(chunk)), stored_texts, log_missing=True ) state.mark_messages_processed() @@ -846,21 +852,6 @@ def _process_batch_results( logger.warning("Failed to process batch result for fact: %s", e) -def _per_fact_ids(ids, n: int) -> list: - """One entry per submitted fact: its id, or ``None`` when it was not - stored (#751). - - ``remember_batch`` returns the stored ids only, so after a skipped group - the list is shorter than the input and position no longer says which - fact an id belongs to. Its ``batch_results`` does. A plain list (a stub - client) is taken as already lined up. - """ - results = getattr(ids, "batch_results", None) - if not isinstance(results, list): - results = list(ids or []) - return list(results[:n]) + [None] * (n - len(results)) - - # --------------------------------------------------------------------------- # Write-quota refusal: keep, tell, retry (#747) # @@ -1032,11 +1023,12 @@ def _drain_entries(state: "AgentState", client, queue: PendingStoreQueue, entrie if not to_store: return 0 + failure: Optional[Exception] = None try: ids = run_sync(client.remember_batch(to_store, source="hermes-auto")) + results = per_fact_results(ids, len(to_store)) except RelayWriteQuotaExceeded as e: - results = list(e.batch_results or []) - results += [None] * (len(to_store) - len(results)) + results = per_fact_results(e, len(to_store)) stored = 0 back = [] for fact, entry, fact_id in zip(to_store, to_store_entries, results): @@ -1057,31 +1049,31 @@ def _drain_entries(state: "AgentState", client, queue: PendingStoreQueue, entrie if isinstance(e, InterpreterShutdownError) or is_interpreter_shutdown_error(e): queue.put_back(to_store_entries) return 0 - kept, dropped = _retry_later(queue, to_store_entries) - logger.warning( - "pending store: queued facts not stored yet (%s); %d kept, %d dropped after %d attempts", - e, kept, dropped, _DRAIN_MAX_ATTEMPTS, - ) - return 0 + # Other groups may have landed (#753): only the rest is retried. + results = per_fact_results(e, len(to_store)) + failure = e stored = 0 not_stored = [] - for fact, entry, fact_id in zip(to_store, to_store_entries, _per_fact_ids(ids, len(to_store))): + for fact, entry, fact_id in zip(to_store, to_store_entries, results): if fact_id: record_write(client, fact_id, fact["text"], fact.get("embedding")) stored += 1 else: not_stored.append(entry) if not_stored: - # Skipped by the batch writer (#751). The relay never reports a - # duplicate, so this is not proof the fact is in the vault: retry it - # (the write gate drops it if it is), up to the attempt limit. + # Failed, or skipped by the batch writer (#751). The relay never + # reports a duplicate, so a skip is not proof the fact is in the + # vault: retry it (the write gate drops it if it is), up to the + # attempt limit. kept, dropped = _retry_later(queue, not_stored) logger.warning( - "pending store: %d queued fact(s) not stored; %d kept, %d dropped after %d attempts", - len(not_stored), kept, dropped, _DRAIN_MAX_ATTEMPTS, + "pending store: %d queued fact(s) not stored%s; %d kept, %d dropped after %d attempts", + len(not_stored), f" ({failure})" if failure is not None else "", + kept, dropped, _DRAIN_MAX_ATTEMPTS, ) - logger.info("pending store: saved %d queued fact(s)", stored) + if stored or failure is None: + logger.info("pending store: saved %d queued fact(s)", stored) return stored diff --git a/python/src/totalreclaw/client.py b/python/src/totalreclaw/client.py index 6e92312e..ff4951ae 100644 --- a/python/src/totalreclaw/client.py +++ b/python/src/totalreclaw/client.py @@ -277,6 +277,38 @@ def __init__(self, ids=(), batch_results=None): ) +class BatchStoreError(RuntimeError): + """:meth:`TotalReclaw.remember_batch` could not store one or more groups, + for a reason other than the write quota (#753). + + Other groups may have been stored before or after the failure. + ``batch_results`` says which, in the same shape as + :attr:`BatchIds.batch_results`: one entry per input fact, its id when it + was stored and ``None`` when it was not. It is a ``RuntimeError``, so + existing ``except RuntimeError`` handlers keep working. + """ + + def __init__(self, message: str, batch_results: list[Optional[str]]): + super().__init__(message) + self.batch_results: list[Optional[str]] = list(batch_results) + + +def per_fact_results(outcome: object, n: int) -> list[Optional[str]]: + """One entry per fact submitted to :meth:`TotalReclaw.remember_batch`: + its id when it was stored, else ``None``. Always *n* entries. + + *outcome* is what the call returned, or the error it raised. A + :class:`BatchIds`, a :class:`BatchStoreError` (#753) and a + ``RelayWriteQuotaExceeded`` (#747) carry ``batch_results``. A plain list + (a stub client's return value) is taken as already lined up with the + input. Any other error counts as nothing stored. + """ + results = getattr(outcome, "batch_results", None) + if not isinstance(results, list): + results = list(outcome) if isinstance(outcome, (list, tuple)) else [] + return list(results[:n]) + [None] * (n - len(results)) + + class TotalReclaw: """High-level TotalReclaw client with remember/recall/forget/export/status. @@ -1087,11 +1119,15 @@ async def remember_batch( used up (#747). Its ``batch_results`` lists, per input fact in input order, the fact id when that fact was stored before the refusal and ``None`` when it was not. - RuntimeError - If one or more groups fail to store for a non-duplicate reason - (the bundler / paymaster error is aggregated). AA25/AA10 are - retried up to 3 times inside the userop layer before surfacing; - a -32500 simulation-size revert triggers adaptive halving first. + BatchStoreError + A ``RuntimeError``. Raised if one or more groups fail to store + for a non-duplicate reason (the bundler / paymaster error is + aggregated). AA25/AA10 are retried up to 3 times inside the + userop layer before surfacing; a -32500 simulation-size revert + triggers adaptive halving first. Other groups may have been + stored: its ``batch_results`` lists, per input fact in input + order, the fact id when that fact was stored and ``None`` when + it was not (#753). Examples -------- @@ -1149,16 +1185,18 @@ async def _store_group(group: list[dict]) -> list[str]: ids, errors = await group_and_store_adaptive( _store_group, facts, MAX_BATCH_GROUP_COUNT, MAX_BATCH_BYTES ) + results = [stored_at.get(i) for i in range(len(facts))] if errors and quota_refusals: err = quota_refusals[0] - err.batch_results = [stored_at.get(i) for i in range(len(facts))] + err.batch_results = results raise err if errors: - raise RuntimeError( + raise BatchStoreError( f"remember_batch: {len(errors)} store group(s) failed: " - + " | ".join(errors) + + " | ".join(errors), + results, ) - return BatchIds(ids, [stored_at.get(i) for i in range(len(facts))]) + return BatchIds(ids, results) async def recall( self, diff --git a/python/src/totalreclaw/crystals/recrystallize.py b/python/src/totalreclaw/crystals/recrystallize.py index fb421ed2..b270ebd3 100644 --- a/python/src/totalreclaw/crystals/recrystallize.py +++ b/python/src/totalreclaw/crystals/recrystallize.py @@ -48,6 +48,8 @@ from pathlib import Path from typing import Any, Callable, Optional +from totalreclaw.client import per_fact_results + # Single source of truth for the relay URL (repo forbids duplicate URL literals). from totalreclaw.relay import _HARDCODED_DEFAULT_URL as _CANONICAL_PROD_URL @@ -695,6 +697,8 @@ class QuotaPaused(Exception): async def _rewrite_session_facts( client: Any, session: "CorrectedSession", + written: Optional[dict[str, str]] = None, + save: Optional[Callable[[], None]] = None, ) -> list[str]: """Write the re-keyed facts for one session; return their new ids. @@ -703,31 +707,70 @@ async def _rewrite_session_facts( ``session_id`` stamped into ``extra_metadata`` and ``import_source`` preserved. Batched ``remember_batch`` up to :data:`MAX_BATCH_SIZE` per ``executeBatch`` UserOp (design ยง4.1). + + ``written`` (old fact id โ†’ new fact id) holds the facts an interrupted + run already wrote; they are not written again. It is filled in place as + each batch lands, including the facts a batch that then raised did store + (#753), and ``save`` runs after every batch that stored something, so the + progress survives a failure or a hard kill. """ - new_ids: list[str] = [] - fact_dicts: list[dict[str, Any]] = [] + if written is None: + written = {} + fact_dicts: list[tuple[str, dict[str, Any]]] = [] for f in session.facts: + if f.fact_id in written: + continue extra_metadata: dict[str, Any] = {"session_id": session.fresh_session_id} import_source = f.metadata.get("import_source") if isinstance(import_source, str) and import_source: extra_metadata["import_source"] = import_source fact_dicts.append( - { - "text": f.text, - "importance": f.importance, - "embedding": f.embedding, - "fact_type": f.fact_type, - "entities": f.entities, - "provenance": f.provenance, - "extra_metadata": extra_metadata, - } + ( + f.fact_id, + { + "text": f.text, + "importance": f.importance, + "embedding": f.embedding, + "fact_type": f.fact_type, + "entities": f.entities, + "provenance": f.provenance, + "extra_metadata": extra_metadata, + }, + ) ) for start in range(0, len(fact_dicts), MAX_BATCH_SIZE): batch = fact_dicts[start : start + MAX_BATCH_SIZE] - ids = await client.remember_batch(batch, source="recrystallize") - new_ids.extend(ids) - return new_ids + try: + ids = await client.remember_batch( + [d for _, d in batch], source="recrystallize" + ) + except Exception as exc: + _keep_landed(batch, exc, written, save) + raise + _keep_landed(batch, ids, written, save) + return [written[f.fact_id] for f in session.facts if f.fact_id in written] + + +def _keep_landed( + batch: list[tuple[str, dict[str, Any]]], + outcome: object, + written: dict[str, str], + save: Optional[Callable[[], None]], +) -> None: + """Add the facts of *batch* that were stored to *written* and persist. + + *outcome* is what ``remember_batch`` returned or the error it raised; a + partly failed batch still says which facts landed (#753). + """ + landed = { + old_id: new_id + for (old_id, _), new_id in zip(batch, per_fact_results(outcome, len(batch))) + if new_id + } + written.update(landed) + if landed and save is not None: + save() async def _write_crystal( @@ -880,7 +923,9 @@ def _emit(msg: str) -> None: f"[{i}/{total}] writing {len(session.facts)} facts โ†’ " f"session {session.fresh_session_id}" ) - sc.new_fact_ids = await _rewrite_session_facts(client, session) + sc.new_fact_ids = await _rewrite_session_facts( + client, session, sc.new_fact_ids_by_old_id, checkpoint.save, + ) sc.facts_written = True checkpoint.save() # persist BEFORE the Crystal write if session.needs_crystal and not sc.crystal_written: @@ -981,6 +1026,11 @@ class SessionCheckpoint: #: quota-billed). ``crystal_written`` guards the Crystal the same way. facts_written: bool = False crystal_written: bool = False + #: The rewritten facts that have landed so far, as old fact id โ†’ new fact + #: id. Saved after each batch, including the facts a partly failed batch + #: did store, so a resume before ``facts_written`` writes only the facts + #: still missing (#753). + new_fact_ids_by_old_id: dict[str, str] = field(default_factory=dict) old_fact_ids_tombstoned: list[str] = field(default_factory=list) old_crystal_ids_tombstoned: list[str] = field(default_factory=list) diff --git a/python/tests/test_batch_id_alignment_751.py b/python/tests/test_batch_id_alignment_751.py index d6967656..7bac0714 100644 --- a/python/tests/test_batch_id_alignment_751.py +++ b/python/tests/test_batch_id_alignment_751.py @@ -9,6 +9,9 @@ On the managed relay the skip only fires by accident: the duplicate check matched ``409`` anywhere in an error message, e.g. inside a hex hash. + +#753: the same per-fact outcome now rides a batch that partly failed for a +reason other than the quota, so callers keep the facts that did land. """ from __future__ import annotations @@ -243,16 +246,18 @@ async def recall(self, *a, **kw): return [] -class TestDrainAlignment: - def _drain(self, client, store): - from totalreclaw.agent.lifecycle import drain_pending_stores +def _drain(client, store): + from totalreclaw.agent.lifecycle import drain_pending_stores + + state = MagicMock() + with patch("totalreclaw.client.store_fact_batch", new=store), \ + patch("totalreclaw.embedding.get_embedding", return_value=None), \ + patch("totalreclaw.agent.lifecycle.record_write") as rec: + saved = drain_pending_stores(state, client) + return saved, rec - state = MagicMock() - with patch("totalreclaw.client.store_fact_batch", new=store), \ - patch("totalreclaw.embedding.get_embedding", return_value=None), \ - patch("totalreclaw.agent.lifecycle.record_write") as rec: - saved = drain_pending_stores(state, client) - return saved, rec + +class TestDrainAlignment: def test_skipped_fact_is_kept_and_ids_stay_with_their_facts(self, state_dir): client = _DrainClient() @@ -261,7 +266,7 @@ def test_skipped_fact_is_kept_and_ids_stay_with_their_facts(self, state_dir): reason="write_quota", ) store, _ = _store_first_half_skipped() - saved, rec = self._drain(client, store) + saved, rec = _drain(client, store) assert saved == 1 assert [(c.args[1], c.args[2]) for c in rec.call_args_list] == [ @@ -283,6 +288,110 @@ def test_skipped_fact_is_dropped_after_the_attempt_limit(self, state_dir): async def store(facts, *a, **kw): raise RuntimeError(_DUPLICATE) - saved, _ = self._drain(client, store) + saved, _ = _drain(client, store) assert saved == 0 assert client.pending_store.count() == 0 + + +# --------------------------------------------------------------------------- +# 5. A batch that partly failed still says which facts landed (#753) +# --------------------------------------------------------------------------- + +_TIMEOUT = "bundler request timed out" + + +def _store_second_half_fails(): + """``store_fact_batch`` stand-in: a group of more than one fact reverts + for size (so it is halved), a group holding the "first" fact is stored, + and any other group fails with an error that is neither a quota refusal + nor a duplicate.""" + + async def _store(facts, *a, **kw): + texts = [f["text"] for f in facts] + if len(facts) > 1: + raise RuntimeError(_SIM_REVERT) + if texts[0].startswith("first"): + return [f"id:{t}" for t in texts] + raise RuntimeError(_TIMEOUT) + + return _store + + +class TestPartialFailureResults: + @pytest.mark.asyncio + async def test_partial_failure_raises_with_per_fact_results(self): + from totalreclaw.client import BatchStoreError + + client = _client_shell() + with patch("totalreclaw.client.store_fact_batch", new=_store_second_half_fails()): + with pytest.raises(RuntimeError, match=_TIMEOUT) as exc: + await client.remember_batch( + [{"text": "first fact"}, {"text": "second fact"}], source="t" + ) + assert isinstance(exc.value, BatchStoreError) + assert exc.value.batch_results == ["id:first fact", None] + + @pytest.mark.asyncio + async def test_total_failure_reports_nothing_stored(self): + from totalreclaw.client import BatchStoreError + + async def store(facts, *a, **kw): + raise RuntimeError(_TIMEOUT) + + client = _client_shell() + with patch("totalreclaw.client.store_fact_batch", new=store): + with pytest.raises(BatchStoreError) as exc: + await client.remember_batch([{"text": "a"}, {"text": "b"}], source="t") + assert exc.value.batch_results == [None, None] + + +class TestAutoExtractPartialFailure: + def test_stored_update_tombstones_and_only_the_failed_fact_is_reported(self, caplog): + shell = _client_shell() + client = MagicMock() + client.recall = AsyncMock(return_value=[]) + client.forget = AsyncMock(return_value=True) + client.get_fact_pin_status = AsyncMock(return_value=False) + client.remember_batch = shell.remember_batch # the real method + state = _plugin_state(client) + facts = [ + ExtractedFact(text="first: home city is Porto", type="claim", importance=8, + action="UPDATE", existing_fact_id=OLD_ID, source="user"), + ExtractedFact(text="second: likes espresso", type="preference", importance=7, + action="ADD", source="user"), + ] + with patch("totalreclaw.client.store_fact_batch", new=_store_second_half_fails()), \ + patch("totalreclaw.agent.lifecycle.record_write") as rec, \ + patch("totalreclaw.agent.lifecycle._after_successful_write") as after, \ + caplog.at_level("WARNING", logger="totalreclaw.agent.lifecycle"): + stored = _run_extract(state, facts) + + # The UPDATE's replacement landed, so its old fact is tombstoned. + client.forget.assert_awaited_once_with(OLD_ID) + assert stored == ["first: home city is Porto"] + assert [(c.args[1], c.args[2]) for c in rec.call_args_list] == [ + ("id:first: home city is Porto", "first: home city is Porto") + ] + not_stored = [r.getMessage() for r in caplog.records if "Fact not stored" in r.getMessage()] + assert len(not_stored) == 1 and "second: likes espresso" in not_stored[0] + # A write went through, so facts an earlier quota refusal queued get + # their retry. + after.assert_called_once() + + +class TestDrainPartialFailure: + def test_only_the_failed_fact_goes_back_to_the_queue(self, state_dir): + client = _DrainClient() + client.pending_store.add( + [{"text": "first queued fact about trains"}, {"text": "second queued fact about boats"}], + reason="write_quota", + ) + saved, rec = _drain(client, _store_second_half_fails()) + + assert saved == 1 + assert [(c.args[1], c.args[2]) for c in rec.call_args_list] == [ + ("id:first queued fact about trains", "first queued fact about trains") + ] + left = client.pending_store.take(10) + assert [e["fact"]["text"] for e in left] == ["second queued fact about boats"] + assert left[0]["attempts"] == 1 diff --git a/python/tests/test_recrystallize.py b/python/tests/test_recrystallize.py index 2d8d3dc7..9beefc78 100644 --- a/python/tests/test_recrystallize.py +++ b/python/tests/test_recrystallize.py @@ -580,6 +580,88 @@ async def test_execute_resume_never_double_mints(monkeypatch, tmp_path, _no_conf assert not wrote_multi_run2 # never re-minted +@pytest.mark.asyncio +async def test_execute_resume_after_partial_batch_writes_only_unwritten_facts( + monkeypatch, tmp_path, _no_confirm +): + """#753: a session write that partly failed keeps the ids that landed, so + the resume does not write those facts a second time.""" + import totalreclaw.recrystallize as rec + from totalreclaw.client import BatchStoreError + + monkeypatch.setattr(rec, "RECRYSTALLIZE_STATE_DIR", tmp_path) + + class _FirstFactOnly(_FakeClient): + async def remember_batch(self, facts, source="python-client"): + self.calls.append(("remember_batch", [f["text"] for f in facts])) + self._new_counter += 1 + raise BatchStoreError( + "remember_batch: 1 store group(s) failed: timed out", + [f"new-{self._new_counter}"] + [None] * (len(facts) - 1), + ) + + client1 = _FirstFactOnly() + with pytest.raises(RuntimeError): + await execute_recrystallize( + client1, _sample_plan(), write_side_fix_confirmed=True, confirm=True, + ) + [first_call] = [c[1] for c in client1.calls if c[0] == "remember_batch"] + assert len(first_call) == 2 + landed, failed = first_call + + client2 = _FakeClient() + cp2 = await execute_recrystallize( + client2, _sample_plan(), write_side_fix_confirmed=True, confirm=True, + checkpoint=RecrystallizeCheckpoint.load("0xabc"), + ) + assert cp2.status == "completed" + run2_texts = [t for c in client2.calls if c[0] == "remember_batch" for t in c[1]] + assert landed not in run2_texts + assert failed in run2_texts + # The session's new facts include the one written by the first run. + assert any("new-1" in sc.new_fact_ids for sc in cp2.sessions.values()) + + +@pytest.mark.asyncio +async def test_each_landed_batch_is_checkpointed_before_the_next( + monkeypatch, tmp_path, _no_confirm +): + """#753: a session split over several batches persists each batch's ids + as it lands, so even a hard kill between batches (no ``except`` runs) + never re-writes the batches that landed.""" + import totalreclaw.recrystallize as rec + + monkeypatch.setattr(rec, "RECRYSTALLIZE_STATE_DIR", tmp_path) + monkeypatch.setattr(rec, "MAX_BATCH_SIZE", 1) + on_disk_before_second: list[list[str]] = [] + + class _KilledOnSecondBatch(_FakeClient): + async def remember_batch(self, facts, source="python-client"): + if any(c[0] == "remember_batch" for c in self.calls): + cp = RecrystallizeCheckpoint.load("0xabc") + on_disk_before_second.append( + [i for sc in cp.sessions.values() for i in sc.new_fact_ids_by_old_id.values()] + ) + raise RuntimeError("simulated interrupt") + return await super().remember_batch(facts, source=source) + + client1 = _KilledOnSecondBatch() + with pytest.raises(RuntimeError): + await execute_recrystallize( + client1, _sample_plan(), write_side_fix_confirmed=True, confirm=True, + ) + assert on_disk_before_second == [["new-1"]] + [(_, [landed])] = [c for c in client1.calls if c[0] == "remember_batch"] + + client2 = _FakeClient() + await execute_recrystallize( + client2, _sample_plan(), write_side_fix_confirmed=True, confirm=True, + checkpoint=RecrystallizeCheckpoint.load("0xabc"), + ) + run2_texts = [t for c in client2.calls if c[0] == "remember_batch" for t in c[1]] + assert landed not in run2_texts + + @pytest.mark.asyncio async def test_execute_quota_pause_is_clean(monkeypatch, tmp_path, _no_confirm): import totalreclaw.recrystallize as rec