Skip to content

fix(python): keep the facts a partly failed batch did store (#753) - #754

Merged
p-diogo merged 1 commit into
mainfrom
fix/python-batch-partial-failure
Nov 13, 2026
Merged

p-diogo merged 1 commit into
mainfrom
fix/python-batch-partial-failure

Conversation

@p-diogo

@p-diogo p-diogo commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes #753.

Problem

In a remember_batch call, one group can be stored while another fails for a reason other than the write quota. The call then raised a plain RuntimeError that did not say which facts landed. #747 had already fixed this for quota refusals, but not for any other error. Each caller treated the whole batch as failed:

The import engine is not affected. It calls store_fact_batch directly.

Changes

  • client.py
  • agent/lifecycle.py
    • Auto-extraction, generic failure branch:
      • stored facts are recorded and the old fact of a stored UPDATE is tombstoned;
      • only the facts that failed are logged as not stored;
      • a partial success sets wrote_ok, so facts queued by an earlier refusal get their retry.
    • Drain: the success and failure paths share one bookkeeping loop, and only the facts that failed are re-queued, with the attempt limit unchanged.
    • The quota branches use the same helper, with no behaviour change.
  • crystals/recrystallize.py: new SessionCheckpoint.new_fact_ids_by_old_id (old fact id → new fact id).
    • _rewrite_session_facts skips facts already in it.
    • It records landed facts after each batch, including a batch that then raised, and saves the checkpoint after every batch that stored something. A resume therefore writes only the missing facts, even after a hard kill between batches.
    • Old checkpoints load with an empty map, since load already ignores unknown keys and fills defaults.
  • Docs:
    • CHANGELOG entry.
    • client-consistency.md: the "per-fact outcomes on a batch" rule now also covers non-quota partial failures and skipped groups.

Testing

  • New tests: 6.
    • 4 in tests/test_batch_id_alignment_751.py, run against the real remember_batch over a patched store_fact_batch: the per-fact results on a partial and on a total failure, auto-extraction, and the drain.
    • 2 in tests/test_recrystallize.py:
      • a partial batch, then a resume;
      • a session split into batches, which also checks that the checkpoint holds the first batch's id on disk before the second batch is sent.
    • All 6 failed before the fix for the expected reasons: no BatchStoreError; forget never awaited; drain saved == 0; nothing on disk before the second batch.
  • Mutation checks: turning off each piece fails exactly its own test. The four pieces are the per-batch save, keeping a raising batch's ids, wrote_ok on a partial success, and the drain's use of the per-fact results.
  • Full suite (serial): 3516 passed, 16 skipped (all opt-in env gates), 1 xfailed.
  • Staging E2E: 13/13 checks passed on a fresh throwaway account. A local stub forwards to api-staging.totalreclaw.xyz. It answers a 2-fact op with -32500, which the UserOp layer retries 3 times before the batch writer halves the group. The first single op goes to staging; the second gets a -32603 error, which is not retried.
    • 0: an old fact is stored and indexed.
    • A: auto-extraction with UPDATE old → new plus an ADD:
      • the replacement is stored and indexed;
      • the old fact is tombstoned on staging;
      • the failed ADD is not on staging and is the only fact logged as not stored.
    • B: the drain stores 1 of 2 queued facts. Only the failed one stays queued, with attempts == 1, and the stored one is indexed.
    • C: the next drain stores it, and each queued fact is on staging exactly once.
    • All test facts were tombstoned afterwards.
    • The first E2E run failed only the tombstone check, because the stub rejected every single-fact op after the first, including the tombstone UserOp. Narrowing the stub to fail only the batch's second op fixed it. All facts from that run were tombstoned too.
  • Recrystallize was not run on staging. It has not been run on a real vault yet, and its new logic is local checkpoint bookkeeping, which the unit tests cover.

🤖 Generated with Claude Code

When one group of a remember_batch call was stored and another failed
for a reason other than the write quota, the call raised a plain
RuntimeError that did not say which facts landed. Auto-extraction then
logged stored facts as not stored and skipped the tombstone of a stored
UPDATE (both versions stayed in the vault); the #747 retry-queue drain
re-queued stored facts, so the next drain could write them twice; and
recrystallize lost the ids of a session's landed facts, so a resume
re-wrote them.

- client: remember_batch raises BatchStoreError (a RuntimeError) that
  carries batch_results, one id or None per input fact, like BatchIds and
  RelayWriteQuotaExceeded. New per_fact_results(outcome, n) reads that
  outcome from a return value or an error; lifecycle's #751 helper folds
  into it.
- lifecycle: auto-extraction records stored facts, tombstones a stored
  UPDATE's old fact, reports only the failed facts, and counts a partial
  success as a write (so queued facts get their retry). The drain re-queues
  only the facts that failed.
- recrystallize: SessionCheckpoint.new_fact_ids_by_old_id records landed
  facts after every batch, including a batch that then raised, so a resume
  writes only the missing facts, even after a hard kill between batches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@p-diogo
p-diogo merged commit 57b8a52 into main Nov 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: a partly failed remember_batch loses the ids of the facts it did store

1 participant