DR8 integration workflow by PiotrKorkus · Pull Request #398 · eclipse-score/reference_integration

PiotrKorkus · 2026-10-13T11:56:11Z

DEMO

Demo PR exists in private fork, as to demonstrate it we need to push workflow to main branch first.

Overview

What changed. ref_int no longer runs every module's unit tests inside its own workspace. The old way used quality_runners.py unit tests, the generated rust_coverage/BUILD, and resolved_dependencies.py / module_patches.py to inject modules into MODULE.bazel. Now each target_sw module runs its own tests in its own workspace, but against the dependency versions ref_int resolves.

Flow

test_and_docs.yml
 ├─ module_tests (module_tests.yml)
 │   ├─ List modules: jq .modules.target_sw keys from known_good.json
 │   └─ Test <module>: one matrix job per module (fail-fast: false)
 │        python3 scripts/module_tests.py run <module>
 │        → uploads artifact module-tests-<module>
 └─ test_and_docs (needs module_tests, runs unless cancelled)
      ├─ feature + platform integration tests (as before)
      ├─ download module-tests-* → _module_reports/
      ├─ module_tests.py report → unit_test_summary.md,
      ├─ collect test.xml into tests-report/ (module results go under external/<module>+/)
      └─ docs build (source code linker reads tests-report/), release ZIP on tags

What module_tests.py run does (scripts/module_tests.py:281)

Resolves ref_int's dependency set. It runs bazel mod graph in ref_int with --lockfile_mode=error, so every job gets the same set from the committed lockfile.
Fetches the module source through ref_int, at the known_good commit with ref_int's patches applied. It copies the source to _module_tests//.
Makes one edit to the module: it appends include("//:unified_toolchains.MODULE.bazel"). This is needed because toolchains can only be defined in MODULE files.
Pins the module's direct deps to ref_int's versions. For each direct dep that ref_int also resolves, it writes a --override_module==<ref_int's fetched copy> line into an overrides bazelrc. Bazel resolves the deeper dependencies from there.
Wraps bazel with its own --output_base, plus --bazelrc=unified.bazelrc and --bazelrc=overrides.bazelrc.
Runs the module's own test command: bazel coverage <test_args from known_good.json> --config=unified [--config=unified-rust] --combined_report=lcov.
Collects the results: test.xml files, lcov.dat, resuldes.bazelrc go into _module_tests/reports//.
Fails if no test ran. Bazel exits 0 when every test is skipped, so a run with no test results counts as a failure.

Shared config

bazel_common/unified.bazelrc: the platform, the GCC 12.2 toolchain, the Ferrocene Rust toolchain, the ferrocene-coverage config and the ANDROID_HOME workaround. ref_int's .bazelrc imports it, and so does every module run, so both are built the same way.
bazel_common/unified_toolchains.MODULE.bazel: defines unified_gcc_x86_64_toolchain. The unified_ prefix keeps it from clashing with a module's own toolchain names.
known_good.json → metadata.test_args: each module's own CI test arguments (configs, filters, targets).

Running it locally

python3 scripts/module_tests.py run score_baselibs
python3 scripts/module_tests.py report _module_tests/rep

Squash of: - feat: two-stage DR-008 test workflow with per-module Stage 2 execution - fix: give central-mode test runs their own exclusion list - ci: split DR-008 test execution into its own workflow file, keep old job unchanged - fix: restore itf conftest.py - ci(stage2): isolate score_communication under a dedicated Stage-2 rc - fix(stage2): transport dependency patches into Stage 2 injection - style: apply ruff formatting - fix(stage2): add score_config_management to ref_int's GCC toolchain - chore: align DR-008 changes with upstream main Co-authored-by: Piotr Korkus <piotr.korkus.ext@qorix.ai>

- ci: add module tests workflow for PR testing - refactor: test target_sw modules via --override_module instead of MODULE.bazel injection - fix(module_tests): give module runs their own output base - fix(unified.bazelrc): clear ANDROID_HOME for module runs too - fix(module_tests): pin direct deps only and inject unified toolchains - fix(time): give time's ubuntu image pull its own name - fix(module_tests): fail runs without executed tests; patch time sanitizer flags - build: bump score_toolchains_rust to 0.10.0 - build: use score_tooling v2.2.3

score_docs_as_code@36cdc3f depends on score_devcontainer 1.11.1, which was yanked, breaking every module test. The override lives in unified_toolchains.MODULE.bazel as it is included by ref_int and by every tested module.

OliverHeilwagen

Really nice solution. Using --override_module and bazel fetch from ref_int context reduced the complexity drastically, also patching a include() for the unified toolchain solves the former open point.

I only see pinning of the whole module graph as required change (pinning of transitive dependencies) see suggestion line 181, all other comments are optional suggestions.

OliverHeilwagen · 2026-10-15T20:28:25Z

+        "USE_BAZEL_VERSION": (REF_INT / ".bazelversion").read_text().strip(),
+    }
+
+    shared = direct_deps(mod_graph(workspace, env)) & resolved


Please change to:

Suggested change

shared = direct_deps(mod_graph(workspace, env)) & resolved

shared = (graph_modules(mod_graph(workspace, env)) & resolved) - {module.name}

Only pinning direct dependencies can lead to version drifts compared to the dependency resolution done by reference_integration.
All modules in scope of reference integration version resolution should be enforced on the modules as long as they are a dependency of the given module under test.
Please also update autodoc description in line 18.

Note: direct_deps can be removed after this change.

Follow up: A override from reference integration may lead to changes is the mod graph of the module, which may require additional --overide_module=, i assume we have to run this step in a loop till we are stable.

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T20:46:16Z

+def override_rc(paths: dict[str, Path]) -> str:
+    return "".join(f"common --override_module={name}={paths[name]}\n" for name in sorted(paths))


👍 usage of --override_module simplifies the approach drastically, thanks for finding this.

OliverHeilwagen · 2026-10-15T21:01:48Z

+    A repo that fails to fetch (e.g. a dead mirror of a module no build ever touches) is just
+    missing from the result instead of failing the run.
+    """
+    bazel(["fetch", "--keep_going", "--lockfile_mode=error", *(f"--repo=@@{n}+" for n in sorted(names))])


👍 Using bazel fetch from reference_integration context also has the benefit that git_override / singe_version_override patches from known_good.json are applied automatically, again nice simplification of the approach!

Please add this effect to the docstring of fetch().

Should we guard agains local_path overrides?

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T21:13:42Z

+    work = args.work_dir.resolve()
+    workspace, out = work / module.name, work / "reports" / module.name
+    shutil.rmtree(workspace, ignore_errors=True)
+    shutil.rmtree(out, ignore_errors=True)
+    out.mkdir(parents=True)
+    source = fetch({module.name}).get(module.name)
+    if source is None:
+        sys.exit(f"could not fetch {module.name} in ref_int")
+    shutil.copytree(source, workspace, symlinks=True)
+    shutil.copy(UNIFIED_TOOLCHAINS, workspace)
+    module_file = workspace / "MODULE.bazel"
+    module_file.chmod(0o644)  # read-only when copied from Bazel's repo cache
+    with module_file.open("a") as f:
+        f.write(f'\ninclude("//:{UNIFIED_TOOLCHAINS.name}")\n')


nitpick (optional)
tiny refactor would further increase already good readability of the script:
_setup_working_dir(...)
_copy_fetch_module(...)
_append_unified_toolchain(...)

similar changes could be done for:
_collect_coverage(...)
_write_summary(...)

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T21:19:36Z

+    shutil.copy(UNIFIED_TOOLCHAINS, workspace)
+    module_file = workspace / "MODULE.bazel"
+    module_file.chmod(0o644)  # read-only when copied from Bazel's repo cache
+    with module_file.open("a") as f:


👍 again nice solution for ensuring identical toolchain configurations.

OliverHeilwagen · 2026-10-15T21:24:26Z

+    env = {
+        **os.environ,
+        "PATH": f"{wrapper.parent}{os.pathsep}{os.environ['PATH']}",
+        "USE_BAZEL_VERSION": (REF_INT / ".bazelversion").read_text().strip(),


Please extend comment stating that USE_BAZEL_VERSION overrules the modules .bazelversion file.

optional, change not required.

OliverHeilwagen · 2026-10-15T22:13:53Z

+
+    shared = direct_deps(mod_graph(workspace, env)) & resolved
+    pinned = fetch(shared)
+    rc.write_text(override_rc(pinned))


Should we not also write the extra_test_config from known_good.json here?

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T22:22:39Z

+    startup = f"--output_base={work / 'output_base' / module.name} --bazelrc={UNIFIED_RC} --bazelrc={rc}"
+    wrapper.write_text(f'#!/bin/sh\nexec {shutil.which("bazel")} {startup} "$@"\n')


Suggested change

startup = f"--output_base={work / 'output_base' / module.name} --bazelrc={UNIFIED_RC} --bazelrc={rc}"

wrapper.write_text(f'#!/bin/sh\nexec {shutil.which("bazel")} {startup} "$@"\n')

startup = [f"--output_base={work / 'output_base' / module.name}", f"--bazelrc={UNIFIED_RC}", f"--bazelrc={rc}"]

wrapper.write_text(f'#!/bin/sh\nexec {shlex.quote(shutil.which("bazel"))} {shlex.join(startup)} "$@"\n')

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T22:43:33Z

+    found = {}
+    for name in names:
+        # ponytail: "<name>+" is bzlmod's canonical name; a few built-ins (platforms) have none.
+        for repo in (external / f"{name}+", external / name):


Please document that multiple_version_override is excluded for now.
I also think this is fine, we can react once it is required.

Fixed. Comment can be resolved.

docs-as-code bump (#400) drops score_devcontainer from the graph, so the yanked-version override is obsolete and removed. Lockfile regenerated.

OliverHeilwagen

No findings remaining.
test_and_docs workflow issue should be resolved once PR is merged (pull_request_target).
build_and_test_qnx and _autosd issue was not investigated but appears to be unrelated to this PR.

Subramanian-K812 and others added 2 commits October 13, 2026 13:49

PiotrKorkus requested review from AlexanderLanin, FScholPer, MaximilianSoerenPollak, antonkri, dcalavrezo-qorix, lurtz, nradakovic, opajonk and pawelrutkaq as code owners October 13, 2026 11:56

PiotrKorkus mentioned this pull request Oct 13, 2026

DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2) #280

Closed

pawelrutkaq and others added 2 commits October 14, 2026 06:52

Ensure excluded test targets are still taken if present in known_good

3376fcd

build: override yanked score_devcontainer 1.11.1 with 1.11.2

1273b37

score_docs_as_code@36cdc3f depends on score_devcontainer 1.11.1, which was yanked, breaking every module test. The override lives in unified_toolchains.MODULE.bazel as it is included by ref_int and by every tested module.

OliverHeilwagen suggested changes Oct 15, 2026

View reviewed changes

PiotrKorkus added 2 commits October 19, 2026 09:50

review fixes

e369df1

Merge upstream/main: resolve MODULE.bazel.lock conflict

568f2f1

docs-as-code bump (#400) drops score_devcontainer from the graph, so the yanked-version override is obsolete and removed. Lockfile regenerated.

OliverHeilwagen approved these changes Oct 19, 2026

View reviewed changes

	shared = direct_deps(mod_graph(workspace, env)) & resolved
	shared = (graph_modules(mod_graph(workspace, env)) & resolved) - {module.name}

		def override_rc(paths: dict[str, Path]) -> str:
		return "".join(f"common --override_module={name}={paths[name]}\n" for name in sorted(paths))

		startup = f"--output_base={work / 'output_base' / module.name} --bazelrc={UNIFIED_RC} --bazelrc={rc}"
		wrapper.write_text(f'#!/bin/sh\nexec {shutil.which("bazel")} {startup} "$@"\n')

Conversation

PiotrKorkus commented Oct 13, 2026

DEMO

Overview

Flow

Shared config

Running it locally

Uh oh!

OliverHeilwagen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OliverHeilwagen left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants