Skip to content

CI: build the graphics configuration, and run the test suite - #66

Draft
msollami wants to merge 4 commits into
mainfrom
ci/graphics-and-tests
Draft

CI: build the graphics configuration, and run the test suite#66
msollami wants to merge 4 commits into
mainfrom
ci/graphics-and-tests

Conversation

@msollami

Copy link
Copy Markdown
Collaborator

Summary

Two CI jobs for two configurations that nothing currently builds or runs, plus one command that
does the same locally.

Issue #65 is the motivating case. The vendored stb_image in src/imageio.c collided with the copy
inside libraylib.a, and USE_GRAPHICS=1 on Linux failed to link with dozens of multiple definition of 'stbi_load'. It reached a user rather than CI because that configuration is built by
nobody:

  • macOS ld64 tolerates duplicate symbols across archives where GNU ld makes it a hard error, so
    the graphics link always succeeded locally;
  • the build job tries apt-get install libraylib-dev and is allowed to continue without it — and
    on these runner images it is never available (E: Unable to locate package libraylib-dev), so every
    run has built USE_GRAPHICS=0 and the placeholder renderer.

The second gap is the test suite. CI compiles the tree and runs two source-level gates; the 400+ test
binaries have never been run by anything but a person choosing to. A subset is indistinguishable from
the whole in a summary, and a pull request recently reported the suite as passing on the strength of
40 of 426 binaries.

Changes

build-graphics job — installs the X11/GL headers, builds raylib 5.5 from source (it is not
in the runner's package set, which is why the apt attempt kept failing), links the tree with
USE_GRAPHICS=1, and fails rather than degrading: a job that quietly builds the placeholder
renderer is what hid #65. It then asserts the regression directly — src/imageio.o must export no
stbi_* symbols (nm shows t, not T) — which catches this class even against a library other
than raylib. Verified against Sam's fix in main: 0 exported, 139 file-local.

tests job — runs tools/run_test_suite.sh, which builds every test target and runs every
binary in one pinned configuration, comparing the outcome against tests/known_failures.txt.

tools/run_test_suite.sh + make check-tests — the same thing locally, so the gate is
reproducible before pushing rather than only discoverable after. Three traps are handled explicitly,
each of which cost real time to diagnose:

  • the subset — every *_tests binary runs; nobody chooses;
  • the configuration — pinned and printed, because a tree left at USE_FLINT=OFF reports 36
    failures of which 32 are the missing library, and one target cannot even link there;
  • the timeouttimeout is GNU coreutils and absent on macOS, where an unguarded call exits 127
    and a pass/fail loop reads that as a failing test. During development this produced a "0 passed,
    444 failed" run that looked like catastrophe and was a missing binary. gtimeout is used when
    present, and its absence is reported rather than silently turning every result red.

tests/known_failures.txt — the standing failures, each with a one-line reason. A listed failure
does not fail the run; an unlisted one does; a listed test that starts passing is named so its line
can be deleted. Without this the gate would be red on the day it lands, and a gate that is red on
arrival teaches people that red means nothing.

Not in this PR, because it is already in main: flint_bridge_tests is built only when
USE_FLINT=ON (it calls flint_cyclotomic_gcd, which lives inside #ifdef USE_FLINT, so with FLINT
off the target fails to link and takes make in tests/ with it). Mentioned because the tests
job depends on that being true.

Testing

  • tools/run_test_suite.sh run against this branch (= main at 9ee372e): 437 passed, 7 failed,
    7 baselined, no unexpected failures.
    The baseline was seeded from that run, not from another
    branch — four entries are long-standing, and three are the NMinimize/FindMinimum tests added on
    2026-08-16/17, each failing deterministically across three runs, so they are convergence
    assertions the optimizers do not currently meet rather than flaky tests. Every line carries its
    reason; none is mine to fix, and they should be short-lived.
  • This is also the gate proving its own worth before it lands: those three failures are on main
    right now and nothing would have reported them.
  • nm src/imageio.o on this branch: no exported stbi_* symbols, 139 file-local.
  • Workflow YAML parses; sh -n tools/run_test_suite.sh clean.
  • make check-c99, check-packed-aware, check-image-packing, check-menu-ids unchanged and green.

Notes for review

  • The tests job builds 400+ binaries and will be the slowest job in the workflow. If PR turnaround
    matters more than simplicity, it shards cleanly over a matrix on the first letter of the binary
    name — happy to do that instead.
  • Marking tests and build-graphics as required status checks in branch protection is what
    turns them from advisory into a gate; that is a repository setting, not something this PR can do.

Two jobs for two configurations that nothing built or ran, plus one command that does the
same locally.

THE GRAPHICS LINK. Issue #65 — the vendored stb_image in src/imageio.c colliding with the
copy inside libraylib.a, dozens of "multiple definition of 'stbi_load'" under USE_GRAPHICS=1
on Linux — reached a user because that configuration is built by nobody. macOS ld64 tolerates
the duplicate symbols GNU ld rejects, so it always linked locally; and the `build` job tries
`apt-get install libraylib-dev` and is allowed to continue without it, which on these runner
images it always is ("E: Unable to locate package libraylib-dev"), so every run has built the
placeholder renderer. The new build-graphics job installs the X11/GL headers, builds raylib
5.5 from source, links with USE_GRAPHICS=1, and FAILS rather than degrading — a job that
quietly builds the placeholder is what hid the bug. It then asserts the regression directly:
src/imageio.o must export no stbi_* symbols, which catches this class against any library
that vendors stb, not just raylib. Verified against the fix now in main: 0 exported, 139
file-local.

THE TEST SUITE. CI compiled the tree and ran two source-level gates; the 400+ test binaries
have never been run by anything but a person choosing to, and a subset is indistinguishable
from the whole in a summary — a pull request recently reported the suite as passing on the
strength of 40 of 426 binaries. tools/run_test_suite.sh builds every target, runs every
binary in one pinned configuration, and reports against tests/known_failures.txt: a listed
failure is expected, an unlisted one fails the run, and a listed test that starts passing is
named so its line can go. `make check-tests` runs it locally; the tests job runs it on every
push and pull request.

Three traps are handled in the script because each cost real time to diagnose. The subset:
every *_tests binary runs and nobody chooses. The configuration: pinned and printed, since a
tree left at USE_FLINT=OFF reports 36 failures of which 32 are the missing library. The
timeout: `timeout` is GNU coreutils and absent on macOS, where an unguarded call exits 127
and a pass/fail loop reads that as a failing test — during development that produced a
"0 passed, 444 failed" run that looked like catastrophe and was a missing binary.

The baseline is MEASURED, not assumed: run against main at 9ee372e it is 437 passed, 7
failed, no unexpected failures. Four entries are long-standing; three are the
NMinimize/FindMinimum tests added on 2026-08-16/17, each failing deterministically across
three runs, so they are convergence assertions rather than flaky tests. Every line carries
its reason, because a baseline of bare names is a list of tests nobody will ever fix.

[claude-assisted]
@msollami
msollami marked this pull request as draft August 17, 2026 18:24
msollami and others added 3 commits August 17, 2026 11:24
The first CI run of the tests job failed at link time and the log was unusable: a -j build
interleaves concurrent compilers, so the error arrived shredded --
"/usr/bin/ld: flint_qqbar.c:(.text+0xmake[2]: *** [...] Error" names neither the symbol nor
the file, and the next move after reading it would have been a guess.

On failure the script now makes one serial pass and greps for the things that actually
identify a link problem (error:, undefined reference, multiple definition, cannot find -l).
A couple of minutes on a path that is already failing, in exchange for an error a human can
act on.
…when it was requested

The tests job failed with undefined references to fmpq_mat_det, fmpz_set_mpz and friends: USE_FLINT
was defined, the bridge compiled for real, and -lflint never reached the linker. The block linked
${FLINT_LIBRARIES} — bare names that depend on link_directories surviving into the link line, which
it did locally (FLINT 3.6 via Homebrew) and did not on the runner (FLINT 3.0.1 via apt). It now
prefers ${FLINT_LINK_LIBRARIES}, which are absolute paths and cannot be lost that way, and prints
what it resolved.

It also stops degrading silently when FLINT was explicitly requested. -DUSE_FLINT=ON is a statement
about what the caller wants covered; answering it by turning the feature off and printing a warning
into a scrolling log is how a configuration ends up untested while looking green — the exact shape of
issue #65. An explicit request that cannot be satisfied is now a configure-time error naming the
package to install; an unrequested default still falls back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants