Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions python/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **Batch fact ids now stay with their own facts when part of a batch is skipped (#751).** `remember_batch` returns only the ids of the facts it stored. When a group was skipped as a duplicate, the list got shorter, and callers that matched ids to facts by position moved every later id onto the wrong fact. In auto-extraction, an `UPDATE` whose replacement was skipped could still delete (tombstone) the old fact, which lost data. In the write-quota retry queue (#747), skipped facts were dropped silently.
- `remember_batch` still returns a `list[str]` of stored ids, now a `BatchIds` with a `batch_results` attribute. It has one entry per input fact, holding the id or `None` when that fact was skipped, the same shape as `RelayWriteQuotaExceeded.batch_results`. Auto-extraction and the retry queue match ids to facts through it.
- A queued fact the batch writer skips goes back to the queue with one more attempt counted, and is dropped at the existing attempt limit.
- A store error counts as a duplicate only when `409` appears as a whole word. Before, `409` anywhere matched, including the digits inside a hex hash or address in an unrelated bundler error, and that error was then dropped silently. The managed relay never returns 409 for writes. The import engine's per-fact path uses the same check.

- **The session summary is no longer lost when the monthly write allowance is used up (#749).** A session Crystal the relay refused for quota was only logged. It now goes into the same encrypted retry queue as refused auto-extracted facts (#747) and is stored once saving works again, as the Crystal that would have been written at session end: same metadata and `session_id`, no embedding, and the parts the user has forgotten since it was queued are removed (#605 / #640). The refusal reaches the user through the once-per-session notice. Queued facts now keep the time they were refused as `created_at`, instead of taking the time the queue drains, so the vault groups them with their conversation. The notice's count reads correctly for one memory ("1 new memory is kept on this device"). Tests: `tests/test_write_quota_747.py` (section 7).

- **A used-up monthly write allowance is no longer silent, and auto-extracted facts are no longer lost (#747).** The relay refuses a write over the cap with `403 {"error": "quota_exceeded"}` on `/v1/bundler`. The client never read that body, and `remember_batch` turned the error into a `RuntimeError` with no status and no "quota" in its text, so even the billing-cache invalidation added for #663 never ran. Auto-extraction logged "Fact not stored (batch failure)", marked the messages processed and moved on: 44 facts were dropped on one install over three days, and the user only found out by asking. Now:
Expand Down
56 changes: 45 additions & 11 deletions python/src/totalreclaw/agent/lifecycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -787,8 +787,9 @@ def _auto_extract_inner(
continue

wrote_ok = True
# fact_ids is a list[str] in the same order as chunk_dicts.
_process_batch_results(client, chunk, fact_ids, stored_texts, log_missing=True)
_process_batch_results(
client, chunk, _per_fact_ids(fact_ids, len(chunk)), stored_texts, log_missing=True
)

state.mark_messages_processed()
if wrote_ok:
Expand Down Expand Up @@ -845,6 +846,21 @@ def _process_batch_results(
logger.warning("Failed to process batch result for fact: %s", e)


def _per_fact_ids(ids, n: int) -> list:
"""One entry per submitted fact: its id, or ``None`` when it was not
stored (#751).

``remember_batch`` returns the stored ids only, so after a skipped group
the list is shorter than the input and position no longer says which
fact an id belongs to. Its ``batch_results`` does. A plain list (a stub
client) is taken as already lined up.
"""
results = getattr(ids, "batch_results", None)
if not isinstance(results, list):
results = list(ids or [])
return list(results[:n]) + [None] * (n - len(results))


# ---------------------------------------------------------------------------
# Write-quota refusal: keep, tell, retry (#747)
#
Expand Down Expand Up @@ -1041,28 +1057,46 @@ def _drain_entries(state: "AgentState", client, queue: PendingStoreQueue, entrie
if isinstance(e, InterpreterShutdownError) or is_interpreter_shutdown_error(e):
queue.put_back(to_store_entries)
return 0
retry = []
for entry in to_store_entries:
entry["attempts"] = int(entry.get("attempts") or 0) + 1
if entry["attempts"] < _DRAIN_MAX_ATTEMPTS:
retry.append(entry)
dropped = len(to_store_entries) - len(retry)
queue.put_back(retry)
kept, dropped = _retry_later(queue, to_store_entries)
logger.warning(
"pending store: queued facts not stored yet (%s); %d kept, %d dropped after %d attempts",
e, len(retry), dropped, _DRAIN_MAX_ATTEMPTS,
e, kept, dropped, _DRAIN_MAX_ATTEMPTS,
)
return 0

stored = 0
for fact, fact_id in zip(to_store, ids):
not_stored = []
for fact, entry, fact_id in zip(to_store, to_store_entries, _per_fact_ids(ids, len(to_store))):
if fact_id:
record_write(client, fact_id, fact["text"], fact.get("embedding"))
stored += 1
else:
not_stored.append(entry)
if not_stored:
# Skipped by the batch writer (#751). The relay never reports a
# duplicate, so this is not proof the fact is in the vault: retry it
# (the write gate drops it if it is), up to the attempt limit.
kept, dropped = _retry_later(queue, not_stored)
logger.warning(
"pending store: %d queued fact(s) not stored; %d kept, %d dropped after %d attempts",
len(not_stored), kept, dropped, _DRAIN_MAX_ATTEMPTS,
)
logger.info("pending store: saved %d queued fact(s)", stored)
return stored


def _retry_later(queue: PendingStoreQueue, entries: list[dict]) -> tuple[int, int]:
"""Put *entries* back with one more attempt counted, dropping those
that reach :data:`_DRAIN_MAX_ATTEMPTS`. Returns ``(kept, dropped)``."""
retry = []
for entry in entries:
entry["attempts"] = int(entry.get("attempts") or 0) + 1
if entry["attempts"] < _DRAIN_MAX_ATTEMPTS:
retry.append(entry)
queue.put_back(retry)
return len(retry), len(entries) - len(retry)


def crystal_gate_passes(n_messages: int, n_stored_facts: int) -> bool:
"""Content-aware length gate for a session Crystal.

Expand Down
40 changes: 31 additions & 9 deletions python/src/totalreclaw/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,25 @@ async def _derive_smart_account_address(mnemonic: str, rpc_url: str = "https://s
return address.lower()


class BatchIds(list):
"""What :meth:`TotalReclaw.remember_batch` returns: the ids of the stored
facts, in input order (a plain ``list[str]``), plus ``batch_results``.

``batch_results`` has one entry per input fact, in input order: the fact
id, or ``None`` when that fact was not stored (its group was skipped as a
duplicate). The list itself is shorter than the input whenever a fact was
skipped, so pair ids with input facts through ``batch_results``, never by
position in the list (#751). ``RelayWriteQuotaExceeded.batch_results``
has the same shape for a batch the quota interrupted.
"""

def __init__(self, ids=(), batch_results=None):
super().__init__(ids)
self.batch_results: list[Optional[str]] = (
list(batch_results) if batch_results is not None else list(self)
)


class TotalReclaw:
"""High-level TotalReclaw client with remember/recall/forget/export/status.

Expand Down Expand Up @@ -1001,7 +1020,7 @@ async def remember_batch(
self,
facts: list[dict],
source: str = "python-client",
) -> list[str]:
) -> "BatchIds":
"""Store N facts via byte-capped, adaptive batched UserOperations.

The batch analogue of :meth:`remember`. Accepts ANY number of facts
Expand Down Expand Up @@ -1049,12 +1068,15 @@ async def remember_batch(

Returns
-------
list[str]
Pre-assigned UUID fact IDs of the STORED facts, in input order.
Subsequent ``forget`` / ``pin_fact`` / ``unpin_fact`` calls
work against these IDs immediately. (A group rejected as a
duplicate contributes no ids and no error — it is silently
skipped, matching the import path's contract.)
BatchIds
A ``list[str]`` of the pre-assigned UUID fact IDs of the STORED
facts, in input order. Subsequent ``forget`` / ``pin_fact`` /
``unpin_fact`` calls work against these IDs immediately. (A group
rejected as a duplicate contributes no ids and no error — it is
silently skipped, matching the import path's contract.) Its
``batch_results`` attribute has one entry per input fact: the id,
or ``None`` when that fact was skipped. Use it to match ids to
facts (#751).

Raises
------
Expand Down Expand Up @@ -1094,7 +1116,7 @@ async def remember_batch(
lsh = self._get_lsh_hasher() if needs_lsh else None
wallet = self._wallet_context(chain_id)

# #747: which input facts landed, and the first write-quota refusal.
# #747/#751: which input facts landed, and the first write-quota refusal.
# The adaptive helper reduces failures to strings, and a group can be
# refused after an earlier group already stored (the relay refuses a
# batch whole when it does not fit), so record outcomes per fact here.
Expand Down Expand Up @@ -1136,7 +1158,7 @@ async def _store_group(group: list[dict]) -> list[str]:
f"remember_batch: {len(errors)} store group(s) failed: "
+ " | ".join(errors)
)
return ids
return BatchIds(ids, [stored_at.get(i) for i in range(len(facts))])

async def recall(
self,
Expand Down
3 changes: 2 additions & 1 deletion python/src/totalreclaw/imports/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ def _import_concurrency() -> int:
MAX_BATCH_BYTES as _MAX_BATCH_BYTES,
MAX_BATCH_GROUP_COUNT as IMPORT_MAX_BATCH_SIZE,
group_and_store_adaptive,
is_duplicate_rejection,
store_fact_batch,
)

Expand Down Expand Up @@ -1881,7 +1882,7 @@ async def _store_group(group: list) -> list[str]:
_record_stored_conv(stored_conv_ids, fact)
except Exception as e:
msg = str(e)
if '409' in msg or 'duplicate' in msg or 'fingerprint' in msg:
if is_duplicate_rejection(msg):
logger.debug("Skipped duplicate: %s", fact.get('text', '')[:60])
# Counted as on-chain duplicate → conversation durably stored.
_record_stored_conv(stored_conv_ids, fact)
Expand Down
18 changes: 17 additions & 1 deletion python/src/totalreclaw/operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import asyncio
import base64
import logging
import re
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
Expand Down Expand Up @@ -505,6 +506,21 @@ def group_payloads_by_size(payloads: list, max_count: int, max_bytes: int):
yield group


_HTTP_409 = re.compile(r"\b409\b")


def is_duplicate_rejection(msg: str) -> bool:
"""Whether a store error message is a duplicate rejection (HTTP 409 or a
duplicate content fingerprint).

``409`` must be a whole word (#751): matching it anywhere also matched
the digits inside a hex hash or address in an unrelated bundler error,
and that error was then silently dropped as a duplicate. The managed
relay's ``/v1/bundler`` never returns 409.
"""
return bool(_HTTP_409.search(msg)) or "duplicate" in msg or "fingerprint" in msg


async def _store_group_adaptive(store_fn, group: list) -> tuple[list[str], list[str]]:
"""Store one payload group via ``store_fn``; halve-and-retry on a
simulation-size revert.
Expand Down Expand Up @@ -532,7 +548,7 @@ async def _store_group_adaptive(store_fn, group: list) -> tuple[list[str], list[
return list(ids), []
except Exception as e:
msg = str(e)
if '409' in msg or 'duplicate' in msg or 'fingerprint' in msg:
if is_duplicate_rejection(msg):
logger.debug("Batch of %d facts rejected as duplicate", len(group))
return [], []
msg_l = msg.lower()
Expand Down
Loading
Loading