Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/specs/totalreclaw/client-consistency.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,11 +222,11 @@ A UserOp submission (`eth_sendUserOperation`) that would take the account over i
Every client must:

1. **Classify it.** Match `error_code` then `error` by string equality, on status 403 only. Anything else (another 403 code such as `call_target_not_allowed`, a non-JSON body, a 5xx) stays a plain HTTP error.
2. **Report per-fact outcomes on a batch.** Which facts landed before the refusal, so the caller neither drops stored facts nor re-stores them.
2. **Report per-fact outcomes on a batch.** Which facts landed before the refusal, so the caller neither drops stored facts nor re-stores them. The same applies to a batch that partly fails for any other reason, and to a batch where a group was skipped (#751, #753).
3. **Tell the user.** An explicit write tool returns a structured result (`error_code: "quota_exceeded"`, plus tier / usage / reset date from `/v1/billing/status` and `upgrade_url`), never the raw HTTP text. A background write (auto-extraction, mirrors) injects a notice into the next turn, at most once per session.
4. **Not lose automatic writes.** Keep refused auto-extracted facts and the session summary (Crystal) on the device (encrypted) and store them once a write succeeds again (reset or upgrade). Facts are deduplicated against the vault first. Each keeps the time it was refused as `created_at`, so it groups with its conversation. A queued Crystal is stored as it would have been at session end, minus any part the user has forgotten since.

Python/Hermes is the reference implementation (#747, #749): `relay.RelayWriteQuotaExceeded` / `raise_for_bundler_status`, `TotalReclaw.remember_batch` (`batch_results`), `agent/pending_store.py`, `agent/write_quota.py`, `agent/lifecycle.py::drain_pending_stores`. MCP returns a structured tool error and has no auto-extraction to queue. The OpenClaw plugin misreports it (#550). ZeroClaw maps it to `QuotaExceeded`.
Python/Hermes is the reference implementation (#747, #749, #753): `relay.RelayWriteQuotaExceeded` / `raise_for_bundler_status`, `TotalReclaw.remember_batch` (`batch_results` on `BatchIds`, `RelayWriteQuotaExceeded` and `BatchStoreError`; `client.per_fact_results`), `agent/pending_store.py`, `agent/write_quota.py`, `agent/lifecycle.py::drain_pending_stores`. MCP returns a structured tool error and has no auto-extraction to queue. The OpenClaw plugin misreports it (#550). ZeroClaw maps it to `QuotaExceeded`.

### Session Debrief

Expand Down
11 changes: 11 additions & 0 deletions python/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **A batch that partly fails no longer loses the facts it did store (#753).** When one group of a `remember_batch` call was stored and another failed for a reason other than the write quota, the call raised a plain `RuntimeError` that did not say which facts landed. Callers treated the whole batch as failed:
- Auto-extraction logged stored facts as "not stored" and did not record them. An `UPDATE` whose replacement was stored kept its old fact, so both versions stayed in the vault.
- The write-quota retry queue (#747) put stored facts back in the queue, where the next drain could write them a second time.
- Recrystallize lost the ids of a session's facts that had landed, so a resumed run wrote them again.

Now:
- The error is a `BatchStoreError`, still a `RuntimeError`. It carries `batch_results`, with one entry per input fact holding the id or `None`, the same shape as `BatchIds.batch_results` and `RelayWriteQuotaExceeded.batch_results`. The new `totalreclaw.client.per_fact_results(outcome, n)` reads that per-fact outcome from a return value or an error.
- Auto-extraction records the stored facts, tombstones the old fact of a stored `UPDATE`, and reports only the facts that failed. A partial success counts as a successful write, so queued facts get their retry.
- The retry queue keeps only the facts that failed.
- Recrystallize saves the facts that landed in its checkpoint after every batch, including a batch that then failed. A resumed run writes only the facts still missing, even after a hard kill between batches.

- **Batch fact ids now stay with their own facts when part of a batch is skipped (#751).** `remember_batch` returns only the ids of the facts it stored. When a group was skipped as a duplicate, the list got shorter, and callers that matched ids to facts by position moved every later id onto the wrong fact. In auto-extraction, an `UPDATE` whose replacement was skipped could still delete (tombstone) the old fact, which lost data. In the write-quota retry queue (#747), skipped facts were dropped silently.
- `remember_batch` still returns a `list[str]` of stored ids, now a `BatchIds` with a `batch_results` attribute. It has one entry per input fact, holding the id or `None` when that fact was skipped, the same shape as `RelayWriteQuotaExceeded.batch_results`. Auto-extraction and the retry queue match ids to facts through it.
- A queued fact the batch writer skips goes back to the queue with one more attempt counted, and is dropped at the existing attempt limit.
Expand Down
70 changes: 31 additions & 39 deletions python/src/totalreclaw/agent/lifecycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
)
from ..forget_guard import ForgetLedger, is_forget_request
from ..pii import auto_storable, drop_sensitive_facts
from ..client import per_fact_results
from ..relay import RelayReadBlocked, RelayWriteQuotaExceeded
from ..write_gate import (
decide_write,
Expand Down Expand Up @@ -755,8 +756,7 @@ def _auto_extract_inner(
# #747: the relay refused the batch for quota. Keep what landed
# before the refusal, queue the rest of this chunk AND every later
# chunk (they would be refused too) for a retry, and tell the user.
results = list(e.batch_results or [])
results += [None] * (len(chunk) - len(results))
results = per_fact_results(e, len(chunk))
_process_batch_results(client, chunk, results, stored_texts, log_missing=False)
refused = [fd for (_, _, fd), fid in zip(chunk, results) if not fid]
refused += [
Expand All @@ -778,17 +778,23 @@ def _auto_extract_inner(
state.invalidate_billing_cache()
except Exception:
pass
# On a total batch failure, none of the facts in this chunk landed.
# Log each one individually so the caller can diagnose.
for fact, _, _ in chunk:
logger.warning(
"Fact not stored (batch failure): %s", fact.text[:80]
)
# Other groups of the chunk may have landed (#753): keep those
# (record them, tombstone the old fact of a stored UPDATE) and
# report only the facts that were not stored.
results = per_fact_results(e, len(chunk))
_process_batch_results(client, chunk, results, stored_texts, log_missing=False)
for (fact, _, _), fact_id in zip(chunk, results):
if not fact_id:
logger.warning(
"Fact not stored (batch failure): %s", fact.text[:80]
)
if any(results):
wrote_ok = True
continue

wrote_ok = True
_process_batch_results(
client, chunk, _per_fact_ids(fact_ids, len(chunk)), stored_texts, log_missing=True
client, chunk, per_fact_results(fact_ids, len(chunk)), stored_texts, log_missing=True
)

state.mark_messages_processed()
Expand Down Expand Up @@ -846,21 +852,6 @@ def _process_batch_results(
logger.warning("Failed to process batch result for fact: %s", e)


def _per_fact_ids(ids, n: int) -> list:
"""One entry per submitted fact: its id, or ``None`` when it was not
stored (#751).

``remember_batch`` returns the stored ids only, so after a skipped group
the list is shorter than the input and position no longer says which
fact an id belongs to. Its ``batch_results`` does. A plain list (a stub
client) is taken as already lined up.
"""
results = getattr(ids, "batch_results", None)
if not isinstance(results, list):
results = list(ids or [])
return list(results[:n]) + [None] * (n - len(results))


# ---------------------------------------------------------------------------
# Write-quota refusal: keep, tell, retry (#747)
#
Expand Down Expand Up @@ -1032,11 +1023,12 @@ def _drain_entries(state: "AgentState", client, queue: PendingStoreQueue, entrie
if not to_store:
return 0

failure: Optional[Exception] = None
try:
ids = run_sync(client.remember_batch(to_store, source="hermes-auto"))
results = per_fact_results(ids, len(to_store))
except RelayWriteQuotaExceeded as e:
results = list(e.batch_results or [])
results += [None] * (len(to_store) - len(results))
results = per_fact_results(e, len(to_store))
stored = 0
back = []
for fact, entry, fact_id in zip(to_store, to_store_entries, results):
Expand All @@ -1057,31 +1049,31 @@ def _drain_entries(state: "AgentState", client, queue: PendingStoreQueue, entrie
if isinstance(e, InterpreterShutdownError) or is_interpreter_shutdown_error(e):
queue.put_back(to_store_entries)
return 0
kept, dropped = _retry_later(queue, to_store_entries)
logger.warning(
"pending store: queued facts not stored yet (%s); %d kept, %d dropped after %d attempts",
e, kept, dropped, _DRAIN_MAX_ATTEMPTS,
)
return 0
# Other groups may have landed (#753): only the rest is retried.
results = per_fact_results(e, len(to_store))
failure = e

stored = 0
not_stored = []
for fact, entry, fact_id in zip(to_store, to_store_entries, _per_fact_ids(ids, len(to_store))):
for fact, entry, fact_id in zip(to_store, to_store_entries, results):
if fact_id:
record_write(client, fact_id, fact["text"], fact.get("embedding"))
stored += 1
else:
not_stored.append(entry)
if not_stored:
# Skipped by the batch writer (#751). The relay never reports a
# duplicate, so this is not proof the fact is in the vault: retry it
# (the write gate drops it if it is), up to the attempt limit.
# Failed, or skipped by the batch writer (#751). The relay never
# reports a duplicate, so a skip is not proof the fact is in the
# vault: retry it (the write gate drops it if it is), up to the
# attempt limit.
kept, dropped = _retry_later(queue, not_stored)
logger.warning(
"pending store: %d queued fact(s) not stored; %d kept, %d dropped after %d attempts",
len(not_stored), kept, dropped, _DRAIN_MAX_ATTEMPTS,
"pending store: %d queued fact(s) not stored%s; %d kept, %d dropped after %d attempts",
len(not_stored), f" ({failure})" if failure is not None else "",
kept, dropped, _DRAIN_MAX_ATTEMPTS,
)
logger.info("pending store: saved %d queued fact(s)", stored)
if stored or failure is None:
logger.info("pending store: saved %d queued fact(s)", stored)
return stored


Expand Down
56 changes: 47 additions & 9 deletions python/src/totalreclaw/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,38 @@ def __init__(self, ids=(), batch_results=None):
)


class BatchStoreError(RuntimeError):
""":meth:`TotalReclaw.remember_batch` could not store one or more groups,
for a reason other than the write quota (#753).

Other groups may have been stored before or after the failure.
``batch_results`` says which, in the same shape as
:attr:`BatchIds.batch_results`: one entry per input fact, its id when it
was stored and ``None`` when it was not. It is a ``RuntimeError``, so
existing ``except RuntimeError`` handlers keep working.
"""

def __init__(self, message: str, batch_results: list[Optional[str]]):
super().__init__(message)
self.batch_results: list[Optional[str]] = list(batch_results)


def per_fact_results(outcome: object, n: int) -> list[Optional[str]]:
"""One entry per fact submitted to :meth:`TotalReclaw.remember_batch`:
its id when it was stored, else ``None``. Always *n* entries.

*outcome* is what the call returned, or the error it raised. A
:class:`BatchIds`, a :class:`BatchStoreError` (#753) and a
``RelayWriteQuotaExceeded`` (#747) carry ``batch_results``. A plain list
(a stub client's return value) is taken as already lined up with the
input. Any other error counts as nothing stored.
"""
results = getattr(outcome, "batch_results", None)
if not isinstance(results, list):
results = list(outcome) if isinstance(outcome, (list, tuple)) else []
return list(results[:n]) + [None] * (n - len(results))


class TotalReclaw:
"""High-level TotalReclaw client with remember/recall/forget/export/status.

Expand Down Expand Up @@ -1087,11 +1119,15 @@ async def remember_batch(
used up (#747). Its ``batch_results`` lists, per input fact in
input order, the fact id when that fact was stored before the
refusal and ``None`` when it was not.
RuntimeError
If one or more groups fail to store for a non-duplicate reason
(the bundler / paymaster error is aggregated). AA25/AA10 are
retried up to 3 times inside the userop layer before surfacing;
a -32500 simulation-size revert triggers adaptive halving first.
BatchStoreError
A ``RuntimeError``. Raised if one or more groups fail to store
for a non-duplicate reason (the bundler / paymaster error is
aggregated). AA25/AA10 are retried up to 3 times inside the
userop layer before surfacing; a -32500 simulation-size revert
triggers adaptive halving first. Other groups may have been
stored: its ``batch_results`` lists, per input fact in input
order, the fact id when that fact was stored and ``None`` when
it was not (#753).

Examples
--------
Expand Down Expand Up @@ -1149,16 +1185,18 @@ async def _store_group(group: list[dict]) -> list[str]:
ids, errors = await group_and_store_adaptive(
_store_group, facts, MAX_BATCH_GROUP_COUNT, MAX_BATCH_BYTES
)
results = [stored_at.get(i) for i in range(len(facts))]
if errors and quota_refusals:
err = quota_refusals[0]
err.batch_results = [stored_at.get(i) for i in range(len(facts))]
err.batch_results = results
raise err
if errors:
raise RuntimeError(
raise BatchStoreError(
f"remember_batch: {len(errors)} store group(s) failed: "
+ " | ".join(errors)
+ " | ".join(errors),
results,
)
return BatchIds(ids, [stored_at.get(i) for i in range(len(facts))])
return BatchIds(ids, results)

async def recall(
self,
Expand Down
80 changes: 65 additions & 15 deletions python/src/totalreclaw/crystals/recrystallize.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@
from pathlib import Path
from typing import Any, Callable, Optional

from totalreclaw.client import per_fact_results

# Single source of truth for the relay URL (repo forbids duplicate URL literals).
from totalreclaw.relay import _HARDCODED_DEFAULT_URL as _CANONICAL_PROD_URL

Expand Down Expand Up @@ -695,6 +697,8 @@ class QuotaPaused(Exception):
async def _rewrite_session_facts(
client: Any,
session: "CorrectedSession",
written: Optional[dict[str, str]] = None,
save: Optional[Callable[[], None]] = None,
) -> list[str]:
"""Write the re-keyed facts for one session; return their new ids.

Expand All @@ -703,31 +707,70 @@ async def _rewrite_session_facts(
``session_id`` stamped into ``extra_metadata`` and ``import_source``
preserved. Batched ``remember_batch`` up to :data:`MAX_BATCH_SIZE` per
``executeBatch`` UserOp (design §4.1).

``written`` (old fact id → new fact id) holds the facts an interrupted
run already wrote; they are not written again. It is filled in place as
each batch lands, including the facts a batch that then raised did store
(#753), and ``save`` runs after every batch that stored something, so the
progress survives a failure or a hard kill.
"""
new_ids: list[str] = []
fact_dicts: list[dict[str, Any]] = []
if written is None:
written = {}
fact_dicts: list[tuple[str, dict[str, Any]]] = []
for f in session.facts:
if f.fact_id in written:
continue
extra_metadata: dict[str, Any] = {"session_id": session.fresh_session_id}
import_source = f.metadata.get("import_source")
if isinstance(import_source, str) and import_source:
extra_metadata["import_source"] = import_source
fact_dicts.append(
{
"text": f.text,
"importance": f.importance,
"embedding": f.embedding,
"fact_type": f.fact_type,
"entities": f.entities,
"provenance": f.provenance,
"extra_metadata": extra_metadata,
}
(
f.fact_id,
{
"text": f.text,
"importance": f.importance,
"embedding": f.embedding,
"fact_type": f.fact_type,
"entities": f.entities,
"provenance": f.provenance,
"extra_metadata": extra_metadata,
},
)
)

for start in range(0, len(fact_dicts), MAX_BATCH_SIZE):
batch = fact_dicts[start : start + MAX_BATCH_SIZE]
ids = await client.remember_batch(batch, source="recrystallize")
new_ids.extend(ids)
return new_ids
try:
ids = await client.remember_batch(
[d for _, d in batch], source="recrystallize"
)
except Exception as exc:
_keep_landed(batch, exc, written, save)
raise
_keep_landed(batch, ids, written, save)
return [written[f.fact_id] for f in session.facts if f.fact_id in written]


def _keep_landed(
batch: list[tuple[str, dict[str, Any]]],
outcome: object,
written: dict[str, str],
save: Optional[Callable[[], None]],
) -> None:
"""Add the facts of *batch* that were stored to *written* and persist.

*outcome* is what ``remember_batch`` returned or the error it raised; a
partly failed batch still says which facts landed (#753).
"""
landed = {
old_id: new_id
for (old_id, _), new_id in zip(batch, per_fact_results(outcome, len(batch)))
if new_id
}
written.update(landed)
if landed and save is not None:
save()


async def _write_crystal(
Expand Down Expand Up @@ -880,7 +923,9 @@ def _emit(msg: str) -> None:
f"[{i}/{total}] writing {len(session.facts)} facts → "
f"session {session.fresh_session_id}"
)
sc.new_fact_ids = await _rewrite_session_facts(client, session)
sc.new_fact_ids = await _rewrite_session_facts(
client, session, sc.new_fact_ids_by_old_id, checkpoint.save,
)
sc.facts_written = True
checkpoint.save() # persist BEFORE the Crystal write
if session.needs_crystal and not sc.crystal_written:
Expand Down Expand Up @@ -981,6 +1026,11 @@ class SessionCheckpoint:
#: quota-billed). ``crystal_written`` guards the Crystal the same way.
facts_written: bool = False
crystal_written: bool = False
#: The rewritten facts that have landed so far, as old fact id → new fact
#: id. Saved after each batch, including the facts a partly failed batch
#: did store, so a resume before ``facts_written`` writes only the facts
#: still missing (#753).
new_fact_ids_by_old_id: dict[str, str] = field(default_factory=dict)
old_fact_ids_tombstoned: list[str] = field(default_factory=list)
old_crystal_ids_tombstoned: list[str] = field(default_factory=list)

Expand Down
Loading
Loading