Skip to content

[Wave 3] Medium — Module sweep: Dev tooling: documentation, site & CLI-help CI guards #550

Open

Description

@cemililik

Wave 3 · 3 finding(s) in 3 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

  • CR-A17a-004 (code, Medium) — troubleshooting.md (EN+TR) advises non-existent training.gradient_checkpointing; YAML/code-block guards see only column-0 ```yaml fences and do not schema-check fragments outside docs/usermanuals
    severity Medium · reachability default · intent `contract-violation` · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN C31-DOC-SCHEMA-PROSE — docs/roadmap/phase-16-trust-surface-hardening.md:785-786; docs/roadmap/risks-and-decisions.md:184 (C-9)
    Failure scenario. An operator adds 'gradient_checkpointing: true' under training:. TrainingConfig(**{'gradient_checkpointing': True}) raises extra_forbidden on ('gradient_checkpointing',).
    Where. `docs/guides/troubleshooting.md:163`, `docs/guides/troubleshooting-tr.md:170`, `tools/check_bilingual_code_blocks.py:71-72` (confirmed), `tools/check_yaml_snippets.py:78-85`
    Recommended fix. Remove or replace the snippet in both mirrors and the fit_check recommendation (or add a real field). Extend fragment key-path validation (as in check_usermanual_schema_drift) to docs/guides and docs/reference. Make the code-block extractor indentation- and tilde-aware.

  • CR-A17a-005 (code, Medium) — check_cli_help_consistency skips every top-level 'forgelm --flag' invocation; top_level_flags is dead data and a shipped --output-dir doc error passes
    severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
    Failure scenario. An operator follows the GPU-cost calibration recipe and the first command exits with an argparse error. Any future renamed or removed main-CLI flag (--fit-check, --merge, --benchmark-only, ...) cited in docs also passes CI unnoticed.
    Where. tools/check_cli_help_consistency.py:475-479, tools/check_cli_help_consistency.py:210 is a local variable feeding :229, not a separate reader, tools/check_cli_help_consistency.py:229 (confirmed), tools/check_cli_help_consistency.py:694
    Recommended fix. Validate top-level invocations against the top-level usage block, not the whole help text with its epilog. Treat a leading '--' as a top-level form, and fix gpu-cost.md:84 (EN and TR) to use output settings that exist.

  • CR-A17a-006 (code, Medium) — check_cli_help_consistency builds its parser surface lexically from help text: 16 prose words become subcommands and prose-mentioned flags become valid
    severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
    Failure scenario. A doc example forgelm audit data.jsonl --pii-mask (a flag that exists only on ingest) passes CI because audit's help prose mentions --pii-mask, and the user gets an argparse error.
    Where. tools/check_cli_help_consistency.py:164-199 (confirmed), tools/check_cli_help_consistency.py:153, tools/check_cli_help_consistency.py:123-141 (confirmed), tools/check_cli_help_consistency.py:202-231 (confirmed)
    Recommended fix. Take subcommand names from the usage line's {a,b,...} choice set or from argparse introspection, and harvest flags only from the options section's leading option strings.

Done when

  • Every finding above is fixed or closed with a reason (tick it here).
  • Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.

Activity

  1. added
    bugSomething isn't working
    backlog: knownAlready tracked in docs/roadmap
    severity: mediumReal defect with limited reach or a workaround
    source: review-2026-10Found by the October 2026 code and documentation review
    wave: 3Roadmap wave 3: medium severity, module sweeps
    on Oct 9, 2026
  2. cemililik commented on Oct 22, 2026

    @cemililik
    CollaboratorAuthor

    Re-verified on development (735a02f) — CR-A17a-006. _parse_subcommand_list still returns 35 names: the 19 real subcommands plus 16 word fragments from wrapped description lines (runtime., optionally, Markdown), or, command, final_model/, staging, single, of, a, more, `pip, completeness, training., optional, model_integrity.json).

    Besides the prose-flag case in the finding, this also hides a non-existent subcommand. forgelm <fragment> --help exits with argparse's "invalid choice" error, whose usage text is the root parser's, so each fragment gets the root flag set as its "surface". Reproduced with --scope on a scratch page:

    $ forgelm staging --config run.yaml     -> not reported
    $ forgelm or --dry-run                  -> not reported
    $ forgelm stagingx --config run.yaml    -> "subcommand 'stagingx' not in parser surface"
    

    The recommended fix (take the names from the usage line's {a,b,...} set or from argparse introspection) closes both cases. Note for whoever takes it: #829 (now a good first issue) changes _run_help in the same file (a scrubbed environment and ANSI stripping) and adds tests that call _parse_subcommand_list. Whichever lands second rebases; the two changes do not conflict in intent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: dev-toolingCode area: dev-toolingbacklog: knownAlready tracked in docs/roadmapbugSomething isn't workingseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-10Found by the October 2026 code and documentation reviewwave: 3Roadmap wave 3: medium severity, module sweeps

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions