[Triage] M-A18-049 as worded is wrong: 'truthfulqa' resolves in lm-eval 0.4.x, as a TAG (neither a task nor a group) (CR-A05-075) #823

Needs triage. Reviewers disagreed on this finding, so it is not counted in the backlog. Decide whether it is a real defect: if yes, set severity and wave; if not, close it with the reason.

CR-A05-075 — M-A18-049 as worded is wrong: 'truthfulqa' resolves in lm-eval 0.4.x, as a TAG (neither a task nor a group)

severity Info · confidence disputed · single reviewer · reproduced end to end

What happens. Real lm_eval.tasks.TaskManager from the PyPI wheels 0.4.4, 0.4.9.2 and 0.4.13 indexes 'truthfulqa' with type 'tag' (0.4.13: Entry kind=Kind.TAG, tags={truthfulqa_mc1, truthfulqa_mc2, truthfulqa_gen}). No truthfulqa.yaml exists in lm_eval/tasks/truthfulqa/ (only truthfulqa_gen.yaml, truthfulqa_mc1.yaml, truthfulqa_mc2.yaml, utils.py, README.md). Each leaf declares 'tag:\n - truthfulqa' on lines 1-2 (truthfulqa_gen.yaml, truthfulqa_mc1.yaml); truthfulqa_mc2.yaml gets the tag through 'include: truthfulqa_mc1.yaml'. 'truthfulqa' is in TaskManager.all_tags and not in all_groups. lm-eval's own benchmarks/openllm.yaml uses '- task: truthfulqa'. So the name the site, the CLI help and the wizard hint use is accepted by simple_evaluate and cache-tasks. It is not a task, and it is not a group with aggregation. The claim 'does not exist as a task name' is true only in the narrow sense that it is a tag. The real defect is how ForgeLM scores the expansion (next findings). The 0.4.4 floor already uses 'tag:'; older 0.4.0-0.4.3 used 'group:' but are below the declared floor.

Where.

• forgelm/cli/_parser.py:1046
• site/index.html:574
• site/js/translations.js:1739
• pyproject.toml:84
• forgelm/cli/subcommands/_cache.py:453
• forgelm/benchmark.py:188
• forgelm/config.py:872

Evidence.

• 0.4.4: {'type':'tag','task':['truthfulqa_mc1','truthfulqa_gen','truthfulqa_mc2']}, is_tag True, is_group False, is_task False. 0.4.9.2: same members. 0.4.13: Entry(kind=Kind.TAG, tags={...3}). Wheel sha256: 0.4.4 37f1ab84..4fc3, 0.4.9.2 b8d12410..6a6c, 0.4.13 5daaa197..286a. The YAML ('tag:' not 'group:') is in tasks/truthfulqa/*.yaml in each wheel.

Recommended fix. Restate the item: the name is valid. Report the scoring and naming consequences below instead.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}