feat(python): declarative codegen config with a targets registry

[dmealing]

Intent

Ship GitHub #267: a declarative metaobjects.config.yaml (targets registry) for the Python codegen CLI. Python-only, additive — NO metamodel/vocabulary change, NO conformance-fixture change, NO coordinated release (npm/PyPI/NuGet/Maven unaffected). Closes #267.

What changed (branch fix/267-python-declarative-config, off main):

• NEW leaf module server/python/src/metaobjects/codegen/project_config.py (ProjectConfig/TargetConfig/load_project_config/ConfigError) + a published JSON Schema metaobjects-config.schema.json. Schema keys are IDENTICAL to the TS metaobjects.config.ts vocabulary: metadata / providers / targets..{outDir, generators, entities}. YAML not .py (Python's config surface is pure data; provider CODE stays in its module, referenced by module:symbol per 1.0: per-port CLI hook to load an app's custom provider (loaders already accept providers) #158).
• gen config mode in cli.py: no-arg "metaobjects gen" loads config+metadata once and runs EVERY target into its own outDir; --config picks the file (default ./metaobjects.config.yaml), --target scopes; providers resolve RELATIVE to the config dir (sys.path prepend — kills the PYTHONPATH= requirement); a cross-target duplicate-output-path guard (run_gen's guard is per-pass) that exempts the auto-emitted package init.py so targets may share an outDir with disjoint entities.
• verify --codegen config mode: no-arg per-target regenerate-to-temp + diff, aggregating the exit code; --target scopes; strict-by-default (ADR-0023) unless --lax.
• Back-compat is load-bearing and byte-identical: config mode triggers ONLY when no positional <metadata_dir> is given; an explicit <metadata_dir> + --out keeps today's flag path unchanged and does NOT consult the config.
• Docs: KNOWN_GAPS.md (targets-registry gap closed) + docs/features/cli.md (new config subsection).

Deliberate decisions a diff-only reviewer would not know:

• YAML (not an executable .py config) is intentional: unlike TS's metaobjects.config.ts (ADR-0034 owned generators / live providers), Python only needs the provider string reference (module:symbol, 1.0: per-port CLI hook to load an app's custom provider (loaders already accept providers) #158), so a data-only config is correct and avoids arbitrary code-exec at gen/verify time.
• Config mode is gated on "no positional metadata_dir" specifically to keep the existing flag path byte-identical (a hard back-compat requirement).
• The cross-target dup guard deliberately EXEMPTS the auto-emitted package init.py (byte-identical across targets) while STILL erroring on a real duplicate generated-file path — this is intended, tested both ways.
• The verify diff logic in _verify_one_target intentionally parallels the legacy _verify_codegen block (rather than refactoring a shared helper) to keep the legacy path provably untouched.
• Schema keys deliberately mirror the TS vocabulary (one shared schema, per-port file surface) per ADR-0021 D3 / FR-025; this does NOT introduce a single physical cross-port config file and does NOT touch TS/Java/C#.

Process already run before this gate: subagent-driven-development — 4 tasks, each a fresh implementer + an independent per-task spec+quality review (all Approved); a final whole-branch review (opus) surfaced 2 Important findings (a non-UTF-8/unreadable-config traceback leak; the cross-target dup-guard init.py false-positive), one fix wave fixed both, one scoped re-review confirmed all addressed with no new breakage. Full Python suite green (1631 passed). Public repo — no private names / home paths in committed files or commit messages.

This is a Python-only change; the relevant validation is the Python suite (cd server/python && uv run pytest). No TS/Java/C#/Kotlin product code changed, so their compile/conformance lanes are not exercised by this branch.

What Changed

• Adds a declarative metaobjects.config.yaml targets registry for the Python codegen CLI — a new project_config.py loader (ProjectConfig/TargetConfig/load_project_config/ConfigError) plus a published metaobjects-config.schema.json, with schema keys mirroring the TS metaobjects.config.ts vocabulary (metadata / providers / targets..{outDir, generators, entities}).
• gen and verify --codegen gain a config mode that triggers only when no positional <metadata_dir> is given: gen loads config+metadata once and runs every target into its own outDir (--config/--target, config-relative provider resolution, a cross-target duplicate-output guard exempting the auto-emitted __init__.py), while verify --codegen regenerates each target to a temp dir and diffs, aggregating the exit — the legacy explicit <metadata_dir> --out flag path stays byte-identical.
• Docs: new config subsection in docs/features/cli.md, a pointer in the Python port doc, and codegen/KNOWN_GAPS.md (targets-registry gap closed).

Closes #267.

Risk Assessment

✅ Low: The prior finding (verify false-positive drift on a gen-supported shared-outDir config) is correctly resolved by making verify symmetric with gen via one whole-selection regen + per-unique-outDir diff; the legacy flag path is byte-identical, the fix is thoroughly tested (bug repro + real-drift/stale-extra preserved + closure widening + dup-guard parity), and no new material issues were introduced.

Testing

Targeted validation of #267's Python declarative config passed completely. All 36 new #267 tests and 44 existing CLI back-compat tests are green (cd server/python && uv run pytest). I then ran the real metaobjects console script end-to-end against a multi-target config and captured a full CLI transcript (artifact below) demonstrating every config-mode path works as a user would experience it: no-arg gen running all targets with metadata loaded once, --target scoping, per-target verify--codegen diffing (shared outDir verified as one unit labeled core+extra — the per-unique-outDir diffing from the final commit), drift detection, the load-bearing back-compat flag path that ignores the config, the init.py-exempting cross-target dup guard (and its real-duplicate contrast), config-relative provider resolution with no PYTHONPATH, and clean no-traceback errors for malformed/non-UTF-8/missing configs. One out-of-scope observation (not a #267 defect): a target whose entity allowlist omits an M:N junction fails with a raw M2MDerivationError traceback — but I confirmed the legacy flag path (--entities Auth,Person) fails identically, so this is pre-existing entity-filtering + M:N behavior, unrelated to and unchanged by this additive config work. No actionable issues found; the change satisfies the stated intent.

Evidence: End-to-end CLI transcript (real metaobjects console script, 9 config-mode paths)

#267 Python declarative config — full demo of the metaobjects CLI. Key exits: [1] no-arg gen => wrote 8 file(s) across 3 target(s), exit 0; [3] verify --codegen => [models]/[other]/[types] in sync, exit 0; [4] hand-edit => drifted: Node.py, exit 1; [5] back-compat flag path gen meta --out flagout => 7 files flat (config ignored), exit 0; [6] shared outDir disjoint entities => [core+extra] in sync, exit 0; [6b] shared outDir duplicate entity => 'duplicate output path across targets', exit 1; [7] provider resolves with env -u PYTHONPATH, exit 0; [8] malformed config => 'target only must declare a non-empty outDir string', exit 1, no traceback; [8b] non-UTF-8 => 'cannot read config file: utf-8 codec...', exit 1, no traceback; [9] no metadata_dir + no config => exit 2 with hint.

================================================================================
#267 — Python declarative metaobjects.config.yaml (targets registry)
A CLI change. End-to-end demo running the real `metaobjects` console script.
Working dir: /tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo
================================================================================
metaobjects.config.yaml (keys mirror the TS metaobjects.config.ts vocabulary):
--------------------------------------------------------------------------------
     1	# metaobjects.config.yaml — declarative targets registry (#267)
     2	metadata: meta                   # metadata dir, relative to this config file
     3	targets:
     4	  models:                        # one target per output package
     5	    outDir: packages/models/generated
     6	    generators: [entity]
     7	    entities: [Program, Week]
     8	  other:
     9	    outDir: packages/other/generated
    10	    generators: [entity]
    11	    entities: [Node, Measurement]
    12	  types:
    13	    outDir: packages/types/generated
    14	    generators: [entity]
    15	    entities: [AllTypes]

================================================================================
[1] config-driven gen — NO positional <metadata_dir>: runs EVERY target
    into its own outDir; metadata loaded ONCE for the whole run.
================================================================================
$ gen
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/models/generated/Program.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/models/generated/Week.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/models/generated/__init__.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/Node.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/Measurement.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/__init__.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/types/generated/AllTypes.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/types/generated/__init__.py
metaobjects gen: wrote 8 file(s) across 3 target(s).
exit=0

Generated tree:
    packages/models/generated/__init__.py
    packages/models/generated/Program.py
    packages/models/generated/Week.py
    packages/other/generated/__init__.py
    packages/other/generated/Measurement.py
    packages/other/generated/Node.py
    packages/types/generated/AllTypes.py
    packages/types/generated/__init__.py

================================================================================
[2] --target scopes the run to ONE target's outDir (others untouched)
================================================================================
$ gen --target other
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/Node.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/Measurement.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/__init__.py
metaobjects gen: wrote 3 file(s) across 1 target(s).
exit=0

Only the `other` target was written:
    packages/other/generated/__init__.py
    packages/other/generated/Measurement.py
    packages/other/generated/Node.py

================================================================================
[3] config-driven verify --codegen — per-target regen-to-temp + diff
    First do a full gen so committed output exists, then verify (in sync).
================================================================================
$ gen
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/models/generated/Program.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/models/generated/Week.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/models/generated/__init__.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/Node.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/Measurement.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/other/generated/__init__.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/types/generated/AllTypes.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/packages/types/generated/__init__.py
metaobjects gen: wrote 8 file(s) across 3 target(s).
exit=0

(full gen re-run silently to populate committed output)

$ verify --codegen
metaobjects verify [models]: in sync (3 file(s)).
metaobjects verify [other]: in sync (3 file(s)).
metaobjects verify [types]: in sync (2 file(s)).
exit=0

================================================================================
[4] drift detection — hand-edit ONE target's committed file; verify flags it
================================================================================
(appended a hand-edit comment to packages/other/generated/Node.py)

$ verify --codegen
error: [other] generated code is out of sync with metadata.
  drifted: Node.py
regenerate (metaobjects gen) and commit the result.
metaobjects verify [models]: in sync (3 file(s)).
metaobjects verify [types]: in sync (2 file(s)).
exit=1

================================================================================
[5] BACK-COMPAT — explicit <metadata_dir> + --out uses the FLAG path and does
    NOT consult metaobjects.config.yaml (byte-identical to pre-#267 behavior).
================================================================================
$ gen meta --out flagout --entities Program,Week
flagout/Program.py
flagout/Week.py
flagout/program_router.py
flagout/week_router.py
flagout/program_filter_allowlist.py
flagout/week_filter_allowlist.py
flagout/__init__.py
metaobjects gen: wrote 7 file(s) to flagout
exit=0

Flag-path output is flat under flagout/ (config's per-target outDirs ignored):
    flagout/__init__.py
    flagout/program_filter_allowlist.py
    flagout/Program.py
    flagout/program_router.py
    flagout/week_filter_allowlist.py
    flagout/Week.py
    flagout/week_router.py

================================================================================
[6] shared-outDir with DISJOINT entities — two targets write ONE package.
    The auto-emitted __init__.py is byte-identical across targets and must NOT
    trip the cross-target duplicate-output-path guard (a deliberate exemption).
================================================================================
$ metaobjects gen --config shared.config.yaml
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/shared_pkg/generated/Program.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/shared_pkg/generated/__init__.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/shared_pkg/generated/Week.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/shared_pkg/generated/__init__.py
metaobjects gen: wrote 4 file(s) across 2 target(s).
exit=0

Both targets coexist in shared_pkg/generated with disjoint entity files:
    shared_pkg/generated/__init__.py
    shared_pkg/generated/Program.py
    shared_pkg/generated/Week.py

verify treats the shared outDir as ONE unit — no false 'extra' drift:
$ metaobjects verify --codegen --config shared.config.yaml
metaobjects verify [core+extra]: in sync (3 file(s)).
exit=0

================================================================================
[6b] contrast — same shared outDir but BOTH targets emit the SAME entity
     => the cross-target duplicate-output-path guard DOES fire (exit 1).
================================================================================
$ metaobjects gen --config dup.config.yaml
error: duplicate output path across targets: '/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/dup_pkg/generated/Program.py' written by both 'a' and 'b'.
exit=1


================================================================================
[7] config-relative PROVIDER — a custom subtype (validator.geocheck) is
    registered by a module:symbol provider sitting BESIDE the config. The CLI
    prepends the config dir to sys.path, so the provider imports with NO
    PYTHONPATH on the invoking shell (#158 / #267).
    (Metadata loads strict-by-default; the custom validator would be
    ERR_UNKNOWN_ATTR without the provider resolving.)
================================================================================
Invoking shell PYTHONPATH: '<unset>'  (unset => the win)

prov_demo/metaobjects.config.yaml:
     1	metadata: meta
     2	providers: ["geo_provider:geo_provider"]   # module:symbol, resolved RELATIVE to this file
     3	targets:
     4	  models:
     5	    outDir: generated
     6	    generators: [entity]

$ metaobjects gen --config prov_demo/metaobjects.config.yaml  (cwd=/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo)
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/prov_demo/generated/Place.py
/tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo/prov_demo/generated/__init__.py
metaobjects gen: wrote 2 file(s) across 1 target(s).
exit=0

Generated (the custom-validated entity model):
    prov_demo/generated/__init__.py
    prov_demo/generated/Place.py

================================================================================
[8] malformed config  =>  clean one-line ConfigError, exit 1 (NO traceback).
================================================================================
$ metaobjects gen --config bad.config.yaml
error: bad.config.yaml: target 'only' must declare a non-empty 'outDir' string.
exit=1

--------------------------------------------------------------------------------
[8b] non-UTF-8 / unreadable config  =>  clean 'cannot read' error, exit 1
     (the raw UnicodeDecodeError traceback leak a review found — now wrapped).
================================================================================
$ metaobjects gen --config nonutf8.config.yaml
error: nonutf8.config.yaml: cannot read config file: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
exit=1

================================================================================
[9] NO positional <metadata_dir> AND no config discoverable  =>  exit 2 with a
    clear hint (this is the only path that does NOT auto-engage config mode).
================================================================================
$ cd /tmp/empty-no-config  &&  metaobjects gen
error: no <metadata_dir> given and no metaobjects.config.yaml found. Either pass <metadata_dir> --out (flag mode) or create a metaobjects.config.yaml (or pass --config <path>).
exit=2


================================================================================
RESULT
================================================================================
Every config-mode code path on the real `metaobjects` console script works:
  [1] no-arg gen runs every target into its own outDir (metadata loaded once)
  [2] --target scopes to one target
  [3] verify --codegen (no arg) per-target regen+diff => 'in sync' per target
  [4] a hand-edited committed file => drift flagged for its target, exit 1
  [5] BACK-COMPAT: explicit <metadata_dir> --out ignores the config entirely
  [6] shared outDir + disjoint entities => success (__init__.py exemption);
      [6b] shared outDir + duplicate entity => dup guard fires, exit 1
  [7] config-relative module:symbol provider resolves with NO PYTHONPATH
  [8] malformed / [8b] non-UTF-8 config => clean one-line error, NO traceback
  [9] no metadata_dir + no config => exit 2 with a flag-vs-config hint
  +   verify --templates stays metadata_dir-gated (not config-driven)

Automated tests: 36/36 new (#267) tests + 44/44 existing CLI back-compat tests
green (cd server/python && uv run pytest).
================================================================================

- Evidence: Demo project (configs + metadata used to generate the transcript) (local file: /tmp/no-mistakes-evidence/01KZ5133694KTBSPT4K1J35CTE/demo)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅

• ⚠️ server/python/src/metaobjects/cli.py:750 - _verify_one_target regenerates a single target's entities into a temp dir and diffs against that target's committed outDir. Gen explicitly supports and tests two targets sharing an outDir with disjoint entities (test_gen_cross_target_shared_outdir_disjoint_entities_not_flagged → both Program.py and Week.py land in shared/). But verify's per-target diff computes extra = committed files not in expected (line 750): when target a regenerates only Program.py, the shared committed dir still holds Week.py (target b's file), so verify reports extra: Week.py as drift for target a (and symmetrically for b) — exit 1 on output that metaobjects gen produced correctly. Concrete path: SHARED_OUTDIR_DISJOINT_ENTITIES config → gen exits 0 → verify --codegen exits 1 with spurious extra drift on both targets. The per-target-diff model cannot distinguish 'another target's legitimate file' from a genuinely stale committed file when outDirs are shared. Resolution is a product decision: either make verify union-aware for shared outDirs, or document shared-outDir as gen-only / have verify warn — hence asking rather than auto-fixing. Primary disjoint-outDir paths (the common case) are correct and unaffected.

🔧 Fix: fix(#267): verify --codegen diffs per unique outDir, not per target
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• uv run pytest tests/codegen/test_project_config.py tests/codegen/test_cli_config_gen.py tests/codegen/test_cli_config_verify.py — the three new #267 test files (36 tests: loader unit + gen config mode + verify config mode)
• uv run pytest tests/codegen/test_cli.py test_cli_verify_subverbs.py test_cli_verify_strict.py test_cli_providers.py test_cli_registry.py test_cli_staleness_nudge.py — existing CLI suite (44 tests) confirming the flag path is untouched
• End-to-end CLI transcript on the real console script: (1) no-arg gen runs all targets into separate outDirs, metadata loaded once; (2) --target scopes to one target; (3) verify --codegen reports 'in sync' per target; (4) a hand-edited committed file is flagged as drift for its target (exit 1); (5) BACK-COMPAT: explicit <metadata_dir> --out ignores the config entirely (byte-identical flag path); (6) shared outDir + disjoint entities succeeds via the __init__.py exemption; (6b) shared outDir + duplicate entity trips the cross-target dup guard (exit 1); (7) a module:symbol provider resolves config-relative with env -u PYTHONPATH (no PYTHONPATH on the invoking shell); (8) malformed and (8b) non-UTF-8 config yield clean one-line ConfigError with no traceback; (9) no metadata_dir + no config yields exit 2 with a flag-vs-config hint
• verify --templates with no metadata_dir stays guarded (exit 2, 'not config-driven') — confirms config mode drives --codegen only
• Confirmed the M:N-through-junction resolution error when a target's entity allowlist omits the junction is PRE-EXISTING and identical on the legacy flag path (metaobjects gen meta --out <dir> --entities Auth,Person) — orthogonal to #267, not introduced by this change

✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 info

• ℹ️ server/python/pyproject.toml - mypy --strict and ruff are configured for server/python but carry pre-existing baseline errors (67 mypy across ~20 files, including cli.py:255 which exists at the base commit; 31 ruff across untouched files), so neither is an enforced clean gate. The new project_config.py module is mypy/ruff-clean and the 2 F841 the change introduced were fixed; cleaning the baseline so these become real gates is an out-of-scope follow-up (touches ~20 unrelated files).

✅ **Push** - passed

✅ No issues found.