proposal: batched inference for TransformerDetector (#23)

[YassinNouh21]

Summary

Design proposal for #23, no code in this PR. One file, docs/proposals/0023-batched-inference.md: what is wrong today with line references, the batching design, and a test plan. Implementation can follow as a separate PR once the approach is agreed.

The short version: batch only the forward pass. Tokenize pairs together with padding, take each row's real length from attention_mask (the current answer_start formula uses the padded width, hallucination_dataset.py:178), run one forward per micro-batch, then decode every row with one shared _decode_row that the single path also uses.

flowchart LR
    A["validate"] --> B["sort by length,<br/>micro-batch"] --> C["tokenize together,<br/>answer_start per row"] --> D["one forward"] --> E["_decode_row<br/>per row"] --> F["restore order"]
    S["predict_prompt"] --> E

Loading

Also fixes the silent zip truncation at transformer.py:452 and llm.py:612,614 with a shared length check in BaseDetector.

Worked example from the doc, three pairs padded to [3, 13] with the test-suite tokenizer:

row	real_len	answer_len	today L - answer_len - 1	proposed mask.sum() - answer_len - 1
0	9	1	11, points at [PAD]	7, points at paris
1	7	2	10, points at [PAD]	4, points at paris
2	13	3	9, short	9, short

Only the longest row is right today.

Related issue

Refs #23

Type of change

• Documentation

Testing

• Other: mkdocs build

Checklist

• I kept the PR focused on one change.
• I added or updated tests/docs when needed.
• I checked that no secrets, API keys, or credentials are included.

Rights & sign-off (required)

• I certify that I have the right to submit this code and that it may be
  distributed under the repository's MIT license
  (see CONTRIBUTING).

[adaamko]

Thanks a lot for writing this up, @YassinNouh21. It's a clear design and it names exactly the traps an implementation has to avoid. #113 has now landed and covers your main points: it takes each row's answer start from the tokenizer's sequence_ids and its real length from attention_mask instead of the padded width, decodes single and batched inputs through one shared helper, and rejects mismatched prompts/answers lengths in the transformer and LLM detectors. Against the real ModernBERT model its batched spans match one-by-one exactly.

Two notes on the analysis. On main before #113, the L - answer_len - 1 formula at hallucination_dataset.py:178 was correct, because prepare_tokenized_input only ever tokenized one unpadded pair; it is a trap for a batched implementation rather than a live bug, but a very good one to call out. The silent zip truncation was real.

We'd rather not add a docs/proposals/ directory, so I'm closing this with thanks. Two ideas from your proposal are not in #113 yet, and either would be a welcome small follow-up PR:

1. A shared length check in BaseDetector that every detector calls; RAGFactCheckerDetector.predict_prompt_batch (rag_fact_checker.py:119) still zips silently.
2. Sort-by-length micro-batching that restores the original order afterwards, to cut padding waste on mixed-length inputs.

Thanks again, this made the review much easier.