feat(ctl): add infrahubctl schema format command#1189
Open
petercrocker wants to merge 9 commits into
Open
Conversation
Add an opinionated, offline formatter for Infrahub schema YAML files whose job is to normalise the ordering of keys within each node, generic, attribute, relationship and dropdown choice, so hand-authored schemas read consistently and produce small diffs. - New infrahub_sdk/ctl/schema_format.py with the pure formatting logic: canonical key orders, restricted-namespace filtering (core nodes only), list-item order preserved, a PyYAML dumper matching the schema-library layout, literal-block multiline handling, and a semantic-equality guard that aborts rather than risk changing a file's meaning. - New `format` subcommand in schema.py: in-place by default, plus --check (CI gate) and --diff, with warnings for comments that PyYAML cannot preserve. - Unit + CLI tests and the regenerated infrahubctl CLI reference.
github-actions
Bot
added
the
type/documentation
Deploying infrahub-sdk-python with
|
| Latest commit: |
68fb073
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://488becf4.infrahub-sdk-python.pages.dev |
| Branch Preview URL: | https://schema-format-command.infrahub-sdk-python.pages.dev |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Switch the schema formatter from PyYAML to ruamel.yaml round-trip mode so that reordering keys no longer discards comments. This also preserves quoting and inline (flow) sequences (e.g. `[manufacturer, name__value]`) for free, so the diff a format run produces is now purely key-ordering. - schema_format.py: reorder keys in place with move_to_end so the comments ruamel attaches to each key travel with it; keep the semantic-equality guard, restricted-namespace filtering, and canonical key orders. The header is preserved (or added when missing). - Drop the comment-drop warning and count_droppable_comments, which existed only because PyYAML lost comments. - schema.py: format from the raw file text; update the command help. - Add ruamel.yaml to the `ctl` / `all` dependency sets.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
The formatter's canonical key ordering is written against a known set of schema properties. When Infrahub adds or removes a property in the published JSON schema, that ordering may need updating (an unrecognised key is preserved, but not ideally placed). Track this without gating releases: - infrahub_sdk/ctl/schema_drift.py compares the live schema's property sets to a committed baseline (schema_properties.json) and reports added/removed properties. It never raises on drift. - `invoke schema-drift-check` emits GitHub ::warning:: annotations for any drift and always exits 0; `invoke schema-drift-update` refreshes the baseline. - .github/workflows/schema-drift.yml runs the check on release publish and manual dispatch, warn-only. - Baseline snapshot + offline unit tests for the drift logic.
Contributor
Author
|
Added a warn-only schema-drift check so upstream changes to the published JSON schema get surfaced without gating releases.
Rationale for baseline-vs-live (rather than diffing the formatter's key lists directly): the schema has internal fields the formatter deliberately doesn't order ( Note: because it's release-gated + |
…er detection Address code-review findings: - _format_entity: only iterate `attributes`/`relationships` when they are lists, so a parseable-but-malformed schema (e.g. `attributes: 5`) is left untouched instead of crashing. - _ensure_schema_header: detect a real `# yaml-language-server:` directive line via regex rather than an arbitrary substring, so the header is still added when the string only appears in a scalar or unrelated comment. - test_format_preserves_comments: assert exit_code == 0 so the test can no longer pass silently if the format command fails. Add regression tests for the malformed-section and substring-in-scalar cases.
CI lints with ruff 0.15.12 (develop's pinned version), which enforces pydocstyle D413; the branch's local ruff 0.15.0 did not, so this passed locally but failed in CI. Add the required blank line after the final docstring section in the schema formatter/drift modules and the schema format command.
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## develop #1189 +/- ##
===========================================
+ Coverage 82.65% 82.76% +0.10%
===========================================
Files 139 141 +2
Lines 12354 12606 +252
Branches 1851 1910 +59
===========================================
+ Hits 10211 10433 +222
- Misses 1575 1594 +19
- Partials 568 579 +11
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Sync the 361-commit gap so CI (which validates the merge with develop) uses the same ruff (0.15.12) and doc generator as develop.
_print_schema_diff used markup=False (needed so bracketed diff content like [manufacturer, name] stays literal) together with inline [green]/[red] tags, which then printed verbatim instead of colouring the line. Apply the colour with the style= argument instead.
…a format Three off-by-default transforms for `infrahubctl schema format`, keeping the base command purely key-ordering: - --strip-defaults: remove node/attribute/relationship keys whose value equals the schema default (context-aware; grounded in the published JSON-schema defaults). Consequential/internal fields (branch, state, inherited, display) are intentionally not stripped. - --sort-by-order-weight: sort attributes and relationships ascending by order_weight; items without one keep their authored order and go last. - --backfill-order-weight: give attributes/relationships lacking an order_weight a single constant value (1000). The semantic guard now neutralises exactly the requested transforms on both sides of the comparison, so an intended change is allowed while any unintended corruption still aborts. Verified guard-safe and idempotent across all schema-library files for every flag combination.
Contributor
There was a problem hiding this comment.
1 issue found across 6 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="infrahub_sdk/ctl/schema_format.py">
<violation number="1" location="infrahub_sdk/ctl/schema_format.py:370">
P2: `--sort-by-order-weight` rejects same-named items with different weights because the guard sorts by `name`, not the transform's weight key. Normalize with `_order_weight_sort_key` so the guard permits exactly the requested reorder.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| if isinstance(items, list): | ||
| items = [_normalize_item(item, defaults, options) for item in items] | ||
| if options.sort_by_order_weight: | ||
| items = sorted(items, key=lambda it: it.get("name", "") if isinstance(it, dict) else "") |
Contributor
There was a problem hiding this comment.
P2: --sort-by-order-weight rejects same-named items with different weights because the guard sorts by name, not the transform's weight key. Normalize with _order_weight_sort_key so the guard permits exactly the requested reorder.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At infrahub_sdk/ctl/schema_format.py, line 370:
<comment>`--sort-by-order-weight` rejects same-named items with different weights because the guard sorts by `name`, not the transform's weight key. Normalize with `_order_weight_sort_key` so the guard permits exactly the requested reorder.</comment>
<file context>
@@ -268,33 +343,89 @@ def is_schema_document(content: Any) -> bool:
+ if isinstance(items, list):
+ items = [_normalize_item(item, defaults, options) for item in items]
+ if options.sort_by_order_weight:
+ items = sorted(items, key=lambda it: it.get("name", "") if isinstance(it, dict) else "")
+ normalized[key] = items
+ return normalized
</file context>
Suggested change
| items = sorted(items, key=lambda it: it.get("name", "") if isinstance(it, dict) else "") | |
| items = sorted(items, key=_order_weight_sort_key) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
infrahubctl schema format— an opinionated, offline formatter for Infrahub schema YAML files. Its core job is normalising the ordering of keys (lines) within each node, generic, attribute, relationship, and dropdown choice, so hand-authored schemas read consistently and produce small diffs.Design
name/namespacefirst,attributes→relationshipslast,order_weightlast within each attribute/relationship, choices asname/label/description/color. Unknown keys are preserved (never dropped).RESTRICTED_NAMESPACESnamespace (Core, Builtin, Internal, Profile, Template, …) are left untouched.ruamel.yaml— comments (the# yaml-language-serverheader, standalone notes, and inline comments), quoting style, and flow-style sequences like[manufacturer, name__value]are all preserved. By default a format run produces a diff that is purely key reordering.Opt-in transforms (off by default)
Three flags go further and change file content; each is neutralised in the safety check so only its intended effect is allowed, and the base command stays purely cosmetic:
--strip-defaults— remove keys whose value equals the schema default, context-aware (e.g.optional: trueis stripped on a relationship but kept on an attribute). Grounded in the published JSON-schema defaults; consequential/internal fields (branch,state,inherited,display) are intentionally excluded to avoid coupling a schema to a default that could later change.--sort-by-order-weight— sort attributes and relationships ascending byorder_weight; items without one keep their authored order and go last.--backfill-order-weight— give attributes/relationships lacking anorder_weighta single constant value (1000).Schema drift tracking
The canonical ordering is written against a known set of schema properties. A warn-only check compares the live JSON schema against a committed baseline so upstream additions/removals are surfaced without blocking releases:
infrahub_sdk/ctl/schema_drift.py+ baselineschema_properties.json;invoke schema-drift-checkemits::warning::annotations and always exits 0;invoke schema-drift-updaterefreshes the baseline..github/workflows/schema-drift.ymlruns it onrelease: publishedand manual dispatch.Changes
infrahub_sdk/ctl/schema_format.py(new) — formatting logic (ruamel round-trip, in-place key reordering, opt-in transforms, transform-aware safety guard).infrahub_sdk/ctl/schema.py— theformatsubcommand (--check/--diff/--strip-defaults/--sort-by-order-weight/--backfill-order-weight).infrahub_sdk/ctl/schema_drift.py,schema_properties.json(new) — drift detection + committed baseline.tasks.py—schema-drift-check/schema-drift-updateinvoke tasks..github/workflows/schema-drift.yml(new) — warn-only drift check.pyproject.toml— addruamel.yamlto thectlandalldependency sets.tests/unit/ctl/test_schema_format.py,test_schema_format_app.py,test_schema_drift.py(new) — 36 tests, incl. comment/flow/quote preservation, malformed-input handling, the three opt-in transforms, and drift logic.docs/docs/infrahubctl/infrahubctl-schema.mdx+ changelog fragment.Verification
uv run pytest tests/unit/ctl/test_schema_format*.py tests/unit/ctl/test_schema_drift.py— 36 passed.uv run invoke format lint-code docs-validate— clean.A companion PR (opsmill/infrahub-skills#74) documents the command in the
infrahub-managing-schemasskill.