Skip to content

fix(python): keep batch fact ids with their own facts when a group is skipped (#751) - #752

Merged
p-diogo merged 1 commit into
mainfrom
fix/python-batch-id-alignment
Nov 13, 2026
Merged

p-diogo merged 1 commit into
mainfrom
fix/python-batch-id-alignment

Conversation

@p-diogo

@p-diogo p-diogo commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes #751.

Problem

TotalReclaw.remember_batch returned only the ids of the facts it stored. A group skipped as a "duplicate" contributes no ids, so after a skip the list is shorter than the input. Two callers paired the list with the input by position:

On the managed service the skip only fires by accident. _store_group_adaptive treated '409' anywhere in an error message as a duplicate, including the digits inside a hex hash or address in an unrelated bundler error. The relay's /v1/bundler route never returns 409.

Changes

  • client.py: remember_batch returns a BatchIds. It is still a list[str] of the stored ids, so existing callers and == comparisons are unchanged. It also carries batch_results, with one entry per input fact holding the id or None. This is the same shape RelayWriteQuotaExceeded.batch_results already has for a quota refusal. The data comes from the per-fact stored_at map that Hermes: write-quota exhaustion (403 on /v1/bundler) never surfaces to user or agent — auto-extraction facts silently dropped (44+ since Nov 9) #747 already built.
  • agent/lifecycle.py:
    • New helper _per_fact_ids lines ids up with facts. A plain list from a stub client is taken as already aligned.
    • Auto-extraction and the drain both use it. A skipped fact is not recorded and its UPDATE tombstones nothing.
    • In the drain, a skipped fact goes back to the queue with attempts + 1, through a new _retry_later helper shared with the existing failure path. It is dropped at the existing limit of 5 attempts. If the fact really is already in the vault, the next drain's write gate drops it.
  • operations.py: new is_duplicate_rejection(msg) counts 409 only as a whole word (\b409\b); duplicate and fingerprint still match as before. It is used in _store_group_adaptive and in the import engine's per-fact path. Existing duplicate tests ("HTTP 409: duplicate fingerprint", "409 fingerprint duplicate") still pass.

Not changed: the MCP server's import-from.ts has the same includes('409') match. It is the self-hosted per-fact path, where the server really returns 409, and has no alignment problem, so it is left for a separate change.

Testing

  • New tests: 13 in python/tests/test_batch_id_alignment_751.py, run against the real remember_batch over a patched store_fact_batch.
    • Before the fix, 12 failed for the expected reasons. In the auto-extraction test, forget was awaited for the skipped UPDATE. In the drain test, the id was recorded against the wrong text. The hex-409 error was swallowed.
    • With only the lifecycle change disabled, the auto-extraction and drain tests fail.
  • Full suite (serial): 3510 passed, 16 skipped (all opt-in env gates), 1 xfailed.
  • Staging E2E: 9/9 checks passed on a fresh throwaway account. A local stub forwards to api-staging.totalreclaw.xyz.
    • A: a real 2-fact remember_batch returns batch_results that match the ids 1:1, and each id decrypts to its own fact text from the staging subgraph.
    • B: with the stub shaping sends, the 2-fact op gets -32500, which the UserOp layer retries 3 times and the batch writer then halves. The first single op gets HTTP 409 and the second goes to staging. The drain stores 1 fact. The skipped fact is still queued with attempts == 1, and the stored fact is indexed on staging.
    • C: the next drain stores the skipped fact and the queue is empty.
    • All 4 facts were tombstoned afterwards.

🤖 Generated with Claude Code

… skipped (#751)

remember_batch returns the ids of the stored facts only. A group skipped
as a duplicate contributes none, so callers that paired the list with the
input by position moved every later id onto the wrong fact: auto-extraction
could tombstone the old fact of an UPDATE whose replacement was never
written, and the #747 drain silently lost skipped queued facts.

- remember_batch returns BatchIds: still a list of the stored ids, plus
  batch_results (one id or None per input fact, the same shape as
  RelayWriteQuotaExceeded.batch_results).
- Auto-extraction and the drain pair ids with facts through it. A skipped
  queued fact goes back to the queue with one more attempt counted.
- A store error is a duplicate only when 409 is a whole word, not digits
  inside a hex hash. The managed relay never returns 409 for writes, so
  the old substring match only ever fired by accident.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@p-diogo
p-diogo merged commit 5d16923 into main Nov 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: a skipped batch group shifts fact ids onto the wrong facts (auto-extraction can tombstone a fact whose replacement was not stored)

1 participant