Add doctor sft-mask and fix SFT loss-mask labels

[ProfSynapse]

Summary

PR 4 of 5. Stacked on #164.

This changes training labels. SFT runs before this PR aren't directly comparable with runs after it. Lineage now records preprocessing contract_version: 2.

python tuner.py doctor sft-mask

Loads only the tokenizer, no model and no GPU. It runs sampled rows through the trainer's real per-row path (materialize_sft_row → shared.sft_preprocessing.materialize_sft_example) and the dataset-contract checks.

Hard failures:

• rows with zero trained tokens;
• rows where prompt masking stopped before the end of the prompt render;
• a trained span that doesn't end on the template's end-of-turn token;
• a doubled BOS at the start of a sequence;
• dataset-contract violations;
• a dropped-row fraction above the threshold.

Warnings and info: truncation rate, p50/p95/max token lengths, and multi-turn rows whose earlier assistant turns are untrained.

It also prints a token-by-token preview of which tokens are masked and which are trained. Output is available as --json. Defaults are in Trainers/sft/configs/mask_doctor.yaml, which is loaded through the strict loader. End-of-turn tokens are found by rendering a probe conversation through the template; no model families are hardcoded.

Label fixes

1. prompt_completion closes with the template's end-of-turn token instead of eos_token. The raw completion encoding is unchanged. The output is byte-identical when the tokenizer's EOS already is its end-of-turn token, which is the usual case for Qwen instruct models. A template with no detectable end-of-turn token falls back to eos_token, with a log line.
2. prompt_completion honours completion_only_loss: false and trains the full sequence.
3. Default render masks tokens after the final end-of-turn, such as the template's trailing newline.
4. Rows left with zero trained tokens after truncation are dropped on every render path, with a logged count per reason. The run fails if the dropped fraction exceeds the new training.max_dropped_row_fraction (default 0.01).
5. Rows where prompt masking stopped early are never trained on prompt tokens. They follow the same drop-and-threshold policy, so a template-level mismatch, which hits every row, fails the run.

The trainer and the doctor share one end-of-turn detection function, derive_end_of_turn_tokens.

Locks

SFT closure members changed. Hashes were refreshed with the checked-in scripts, and all four checks report CURRENT.

Reviewer check needed

tests/training/test_chat_template_training_transport.py::test_verified_qwen_template_prompt_completion_boundary_offline was skipped here. It needs SYNAPTIC_TEST_QWEN35_TOKENIZER_ARTIFACT, which wasn't available. It should still pass, because of the byte-identical guarantee above. Please run it once with the artifact before merging.

Test plan

• test_mask_doctor.py (26) and test_sft_label_fixes.py (17) pass. The label tests use an offline tokenizer whose EOS differs from its end-of-turn token.
• Smoke run on a checked-in dataset (300 rows) with locally built ChatML tokenizers passes for both the default and prompt_completion renders.
• Pinned Qwen3.5 transport boundary test, run with the tokenizer artifact.
• Full CI run at the top of the stack passes (see Refuse unknown config keys and trainer flags the target does not accept #162).

🤖 Generated with Claude Code

https://claude.ai/code/session_013etFNjLuo22CCoQmUGYqCT

---

Generated by Claude Code