Report cross-version metamodel compatibility findings in integrated documentation

[AlexanderLanin]

TL;DR / Executive summary

Cross-module documentation integration combines independently released modules, while Bazel resolves one current validation baseline for the complete graph. A module can therefore be locally valid at release time yet trigger new metamodel findings during integration.

Introduce a compatibility-aware validation mode for cross-module integration only:

• Keep metamodel warnings from top-level documentation fatal.
• Report, but do not fail on, metamodel warnings originating in imported docs() targets that have already passed their standalone validation.
• Keep concrete version-mismatch warnings fatal when both endpoints are local; report, but do not fail on, them when at least one endpoint is external.

Non-fatal compatibility findings must remain prominent in console output and on the generated top-level documentation index. Standalone validation remains strict and unchanged.

Problem

S-CORE documentation is integrated from modules that evolve independently and are not upgraded in lockstep. As a result, different modules may support different versions of docs-as-code, the metamodel, and documentation capabilities.

During integration, Bazel resolves a single effective dependency graph. It selects the newest metamodel requested by any participating module and the newest set of feature requirements referenced by any participating module; that resolved set is applied to all features in the integrated build. The consuming documentation is therefore typically validated against a newer baseline than the one available when a supplying module was released. Consequently, a module that was valid when published may fail integration simply because the consumer applies newer validation rules. Although alignment processes exist, they are not yet applied consistently; and even with consistent use, their deliberately infrequent cadence—typically every two to eight weeks—means that temporary version skew remains expected.

Examples include attributes or links that became mandatory only in later metamodel versions, or changed version requirements. These are compatibility findings: validation findings that are expected and explainable from the producer/consumer version skew. They are classified separately so integration can continue while the compatibility gap remains clearly visible; they are not necessarily defects in the supplying module.

The existing behaviour effectively assumes a continuously integrated project landscape in which participating modules advance together. Definitions, processes, and alignment initiatives exist to synchronise independently released modules with different upgrade schedules, but they cannot be applied perfectly or eliminate version skew completely.

High-level solution

Introduce a compatibility-aware validation mode for cross-module documentation integration only.

The design assumes that every imported docs() target has already passed standalone validation against its own declared baseline; otherwise it would not be eligible for integration. Under that invariant, a metamodel finding discovered during integration is either owned by the top-level documentation and remains a fatal local error, or originates in an imported module and can be assessed as a producer/consumer version mismatch using its source provenance and declared baseline.

The mode distinguishes between:

• errors: violations that make the integrated documentation invalid and must fail the build; and
• compatibility findings: deviations explained by the supplying module's declared baseline being older than the consumer's resolved validation baseline. They are reported without blocking integration.

Standalone validation remains unchanged: a module built against its own declared baseline must still fail on genuine violations.

Compatibility findings must be impossible to overlook. They remain prominently visible in build output and are presented to readers on the landing page of the generated top-level documentation. This keeps integration robust while preserving a clear upgrade and remediation signal.

Solution details

1. Apply this classification only during cross-module integration. Standalone validation of every docs() target remains unchanged.
2. Classify metamodel-validation warnings by their source:
   • A metamodel warning originating in top-level documentation remains fatal.
   • A metamodel warning originating in an imported, standalone-validated docs() target is a non-fatal compatibility finding. This includes missing mandatory attributes, missing mandatory links, and all other metamodel checks: the supplying documentation was locally consistent against its own baseline.
3. Treat concrete version-mismatch warnings differently. They remain fatal when both endpoints involved in the comparison are local, because local documentation is expected to be consistent. When at least one endpoint belongs to an external module, report the mismatch as a non-fatal compatibility finding.
4. Keep non-fatal compatibility findings highly visible in console output, including the supplying module, affected need, source location, and check category.
5. Add a generated compatibility notice to the top-level documentation index. It should summarise the findings, identify the affected modules, and link to a detailed generated report and the affected needs.
6. Make the structured compatibility report available to CI so projects can track and eventually reduce compatibility debt. A future policy may add budgets or prohibit introducing new findings.

Open problems and risks

• Version declarations must be authoritative and unambiguous. If a module cannot state its supported baseline, the integration must use a safe default rather than silently classifying violations as compatible.
• The policy needs careful ownership. An overly broad compatibility category could conceal a genuine regression; an overly narrow one defeats the purpose of decoupled upgrades.
• Migrations may be non-monotonic: a module may conform to one model version while conflicting with another in more than one way. Reports must preserve both producer and consumer context.
• The top-level report must avoid double-counting a finding that appears through several dependencies and must retain the supplying module as provenance.
• Compatibility-aware validation must not weaken normal standalone validation.
• Open policy decision: should a version mismatch be non-fatal when at least one endpoint is external, or only when both endpoints are external? The initial proposal uses the former; the latter is a stricter alternative that needs evaluation.
• The presentation needs to be useful without overwhelming readers. The index should provide a concise summary; detailed findings belong on a separate generated report page.

[AlexanderLanin]

Proof of Concept

Website is created and contains the following warning box:

[AlexanderLanin]

Implementation ideas

We generally do not control the warning emitters. Sphinx, Sphinx-Needs, or another extension may expose no compatibility callback and only partial log metadata. We can still integrate through extension events, source-specific hooks, or narrowly scoped injection. Local findings, unknown sources, parser failures, and missing or ambiguous provenance must remain fatal; standalone validation remains unchanged.

Option 1: Classify at source-specific integration points

Invoke the compatibility policy before a warning is emitted. The integration may be inside owned code or injected through a third-party extension event. For metamodel checks, the reporter can be passed through CheckLogger. Sphinx-Needs version conditions can be classified at needs-before-post-processing, neutralizing only approved conditions.

Unknown warnings remain on native -W. Each new source needs a stable pre-emission integration point and focused tests. If an adapter mutates source data to prevent warning emission, tests must also prove that rendering and link semantics remain unchanged.

Option 2: Collect warnings and recreate -W

Add a handler at config-inited, disable native -W, and fail at build end for every observed warning that was not classified as an external compatibility finding.

Unknown messages and parser drift fail closed: the likely consequence after a dependency update is an unnecessary build failure, not an ignored warning. Warnings emitted before config-inited remain a gap.

Option 3: Wrap the source logger adapter

Replace or monkey-patch the relevant adapter's warning() method. Classify known findings and delegate every other call unchanged to native -W.

This retains structured call arguments where available. It depends on the adapter API and installation order, but upgrades are infrequent and focused tests can detect drift. This is a viable source-specific fallback when no extension event exists.

Option 4: Filter Sphinx's warning handler

After Sphinx configures logging, find its WarningStreamHandler and insert a custom filter before WarningSuppressor. Approved findings are recorded and removed from that handler; every other record reaches Sphinx's warning count and native -W.

This provides one common interception point, but depends on Sphinx's private handler type and filter order. Approved findings also need a separate output path because they no longer appear in the normal warning stream.

Option 5: Reconcile a collector with the warning inventory

Extend Option 2 by comparing classified records with Sphinx's total warning count or --warning-file. An early warning missed by the collector creates a mismatch and therefore fails the build.

This closes Option 2's early-warning gap without patching Sphinx startup, but adds global reconciliation logic for duplicates and warning formatting.

Option	Intervention point	Unknown warnings	Maintenance complexity
1. Source adapter	before emission, via source hook	native -W	medium core; medium per source
2. Collector	after emission, via added handler	our final gate	medium core; low-medium per source
3. Logger wrapper	at source warning()	native -W via delegation	medium core; medium-high per source
4. Handler filter	inside Sphinx's warning handler	native -W	high core; low-medium per source
5. Reconciled collector	Option 2 plus warning inventory	our reconciled gate	high core; low-medium per source

Recommendation

Start with Option 1: one small adapter at the nearest stable pre-emission integration point for each supported source. It preserves native -W for unsupported and unknown warnings and avoids replacing Sphinx's global failure behavior.

If a future source has no extension hook, prefer the narrow source-specific wrapper from Option 3. Use global Option 5 only when neither integration is possible, because it adds substantially more shared maintenance complexity.

Whichever mechanism is used, black-box tests should prove that unknown, malformed, local, and missing-provenance findings remain fatal.