docs/usermanuals/en/operations/troubleshooting.md: false claim — Llama Guard 'high severity' is correct; 'The check that matters is regression vs baseline, not absolute scores. Make sure evaluation.s… (DOC-D3B04B-012) #230

Wave 1 · theme D1-10: Documentation-only fixes · highest severity High · 1 finding(s) · effort docs fix

Summary

Users are told to dismiss safety flags as expected and to configure a baseline that does not exist. They misunderstand why the gate fails (absolute threshold) and may loosen it on the false premise that it is relative.

Primary finding

DOC-D3B04B-012 — docs/usermanuals/en/operations/troubleshooting.md: false claim — Llama Guard 'high severity' is correct; 'The check that matters is regression vs baseline, not absolute scores. Make sure evaluation.s…

severity High · already tracked: KNOWN-OPEN C31-DOC-SCHEMA-PROSE docs/roadmap/phase-16-trust-surface-hardening.md:785-786

Where. docs/usermanuals/en/operations/troubleshooting.md:126-130 (Turkish mirror: docs/usermanuals/tr/operations/troubleshooting.md:128-130) · type: false claim

The document says. Llama Guard 'high severity' is correct; 'The check that matters is regression vs baseline, not absolute scores. Make sure evaluation.safety.baseline points at the pre-train baseline.'

What the code does. There is no evaluation.safety.baseline key (validation rejects it as extra_forbidden). The safety gate is an absolute post-training unsafe-ratio ceiling (max_safety_regression), plus optional min_safety_score and severity_thresholds. No pre-training safety measurement is taken (forgelm/safety/init.py:7-11; forgelm/safety/_gates.py:258).

Code references. forgelm/safety/__init__.py:7-11, forgelm/safety/_gates.py:258, forgelm/config.py:940

User impact. Users are told to dismiss safety flags as expected and to configure a baseline that does not exist. They misunderstand why the gate fails (absolute threshold) and may loosen it on the false premise that it is relative.

Proposed correction. Cause: the gate is absolute — the post-training unsafe ratio must stay ≤ evaluation.safety.max_safety_regression (no baseline is measured). Fix: inspect safety/safety_results.json per-category; adjust the probe set (test_prompts) or thresholds deliberately, and compare against the base model with forgelm safety-eval --model <base> if you need a reference.

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}