Skip to content

Python: a skipped batch group shifts fact ids onto the wrong facts (auto-extraction can tombstone a fact whose replacement was not stored) #751

Closed

Description

@p-diogo

Problem

TotalReclaw.remember_batch returns only the ids of the facts it stored. When a batch group is skipped as a "duplicate", that group's facts get no id, so the list is shorter than the input. Two callers in agent/lifecycle.py still pair the list with the input by position:

So after a skipped group, every later id is matched to the wrong fact.

What goes wrong

Example: a 15-fact auto-extraction chunk with embeddings goes over the 32 KB byte cap and is split into two groups. Group 1 (facts 0–7) is skipped and group 2 (facts 8–14) is stored, so 7 ids come back. Then:

  • Facts 0–6 are treated as stored, using the ids of facts 8–14.
    • Their text goes into the recent-write ledger and stored_texts under the wrong ids.
    • If one of them is an UPDATE, its old fact is deleted (tombstoned) although the replacement was never written. This loses data.
  • Facts 8–14 are logged as "not stored" although they were stored. Old facts they were meant to replace through an UPDATE are not tombstoned, so the vault keeps both.
  • The drain loses skipped queued facts silently. They were already taken off the queue, are not stored, and are not put back.

Why a group gets skipped on the managed service

operations._store_group_adaptive treats a store error as a duplicate, and drops it with no ids and no error, if its message contains '409', 'duplicate' or 'fingerprint' anywhere. The relay's /v1/bundler route never returns 409 or a duplicate error; only the self-hosted server did, and the Python client does not support self-hosted. So on the managed service the skip only fires by accident. For example, '409' appears inside a hex hash or address in a bundler error message. That accidental match drops a group silently and triggers the misalignment above.

Found by the review of #750. This bug comes from the #747 drain and the older auto-extraction path, not from #750.

Fix

  1. remember_batch keeps returning the stored ids, as before, but the returned list also carries batch_results: one entry per input fact, in input order, holding the fact id or None when that fact was not stored. This matches what RelayWriteQuotaExceeded.batch_results already provides on a quota refusal.
  2. Auto-extraction and the drain read batch_results, so each id stays with its own fact. A fact that was not stored is not recorded and its UPDATE does not tombstone anything. In the drain, it goes back to the queue with attempts + 1, capped by the existing attempt limit.
  3. _store_group_adaptive treats 409 as a duplicate only when it appears as a whole word, as in HTTP 409, not as three digits inside a hex string.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions