CI: build the graphics configuration, and run the test suite - #66
Draft
msollami wants to merge 4 commits into
Draft
CI: build the graphics configuration, and run the test suite#66msollami wants to merge 4 commits into
msollami wants to merge 4 commits into
Conversation
Two jobs for two configurations that nothing built or ran, plus one command that does the same locally. THE GRAPHICS LINK. Issue #65 — the vendored stb_image in src/imageio.c colliding with the copy inside libraylib.a, dozens of "multiple definition of 'stbi_load'" under USE_GRAPHICS=1 on Linux — reached a user because that configuration is built by nobody. macOS ld64 tolerates the duplicate symbols GNU ld rejects, so it always linked locally; and the `build` job tries `apt-get install libraylib-dev` and is allowed to continue without it, which on these runner images it always is ("E: Unable to locate package libraylib-dev"), so every run has built the placeholder renderer. The new build-graphics job installs the X11/GL headers, builds raylib 5.5 from source, links with USE_GRAPHICS=1, and FAILS rather than degrading — a job that quietly builds the placeholder is what hid the bug. It then asserts the regression directly: src/imageio.o must export no stbi_* symbols, which catches this class against any library that vendors stb, not just raylib. Verified against the fix now in main: 0 exported, 139 file-local. THE TEST SUITE. CI compiled the tree and ran two source-level gates; the 400+ test binaries have never been run by anything but a person choosing to, and a subset is indistinguishable from the whole in a summary — a pull request recently reported the suite as passing on the strength of 40 of 426 binaries. tools/run_test_suite.sh builds every target, runs every binary in one pinned configuration, and reports against tests/known_failures.txt: a listed failure is expected, an unlisted one fails the run, and a listed test that starts passing is named so its line can go. `make check-tests` runs it locally; the tests job runs it on every push and pull request. Three traps are handled in the script because each cost real time to diagnose. The subset: every *_tests binary runs and nobody chooses. The configuration: pinned and printed, since a tree left at USE_FLINT=OFF reports 36 failures of which 32 are the missing library. The timeout: `timeout` is GNU coreutils and absent on macOS, where an unguarded call exits 127 and a pass/fail loop reads that as a failing test — during development that produced a "0 passed, 444 failed" run that looked like catastrophe and was a missing binary. The baseline is MEASURED, not assumed: run against main at 9ee372e it is 437 passed, 7 failed, no unexpected failures. Four entries are long-standing; three are the NMinimize/FindMinimum tests added on 2026-08-16/17, each failing deterministically across three runs, so they are convergence assertions rather than flaky tests. Every line carries its reason, because a baseline of bare names is a list of tests nobody will ever fix. [claude-assisted]
msollami
marked this pull request as draft
August 17, 2026 18:24
The first CI run of the tests job failed at link time and the log was unusable: a -j build interleaves concurrent compilers, so the error arrived shredded -- "/usr/bin/ld: flint_qqbar.c:(.text+0xmake[2]: *** [...] Error" names neither the symbol nor the file, and the next move after reading it would have been a guess. On failure the script now makes one serial pass and greps for the things that actually identify a link problem (error:, undefined reference, multiple definition, cannot find -l). A couple of minutes on a path that is already failing, in exchange for an error a human can act on.
…when it was requested
The tests job failed with undefined references to fmpq_mat_det, fmpz_set_mpz and friends: USE_FLINT
was defined, the bridge compiled for real, and -lflint never reached the linker. The block linked
${FLINT_LIBRARIES} — bare names that depend on link_directories surviving into the link line, which
it did locally (FLINT 3.6 via Homebrew) and did not on the runner (FLINT 3.0.1 via apt). It now
prefers ${FLINT_LINK_LIBRARIES}, which are absolute paths and cannot be lost that way, and prints
what it resolved.
It also stops degrading silently when FLINT was explicitly requested. -DUSE_FLINT=ON is a statement
about what the caller wants covered; answering it by turning the feature off and printing a warning
into a scrolling log is how a configuration ends up untested while looking green — the exact shape of
issue #65. An explicit request that cannot be satisfied is now a configure-time error naming the
package to install; an unrequested default still falls back.
# Conflicts: # makefile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two CI jobs for two configurations that nothing currently builds or runs, plus one command that
does the same locally.
Issue #65 is the motivating case. The vendored
stb_imageinsrc/imageio.ccollided with the copyinside
libraylib.a, andUSE_GRAPHICS=1on Linux failed to link with dozens ofmultiple definition of 'stbi_load'. It reached a user rather than CI because that configuration is built bynobody:
ld64tolerates duplicate symbols across archives where GNUldmakes it a hard error, sothe graphics link always succeeded locally;
buildjob triesapt-get install libraylib-devand is allowed to continue without it — andon these runner images it is never available (
E: Unable to locate package libraylib-dev), so everyrun has built
USE_GRAPHICS=0and the placeholder renderer.The second gap is the test suite. CI compiles the tree and runs two source-level gates; the 400+ test
binaries have never been run by anything but a person choosing to. A subset is indistinguishable from
the whole in a summary, and a pull request recently reported the suite as passing on the strength of
40 of 426 binaries.
Changes
build-graphicsjob — installs the X11/GL headers, builds raylib 5.5 from source (it is notin the runner's package set, which is why the apt attempt kept failing), links the tree with
USE_GRAPHICS=1, and fails rather than degrading: a job that quietly builds the placeholderrenderer is what hid #65. It then asserts the regression directly —
src/imageio.omust export nostbi_*symbols (nmshowst, notT) — which catches this class even against a library otherthan raylib. Verified against Sam's fix in
main: 0 exported, 139 file-local.testsjob — runstools/run_test_suite.sh, which builds every test target and runs everybinary in one pinned configuration, comparing the outcome against
tests/known_failures.txt.tools/run_test_suite.sh+make check-tests— the same thing locally, so the gate isreproducible before pushing rather than only discoverable after. Three traps are handled explicitly,
each of which cost real time to diagnose:
*_testsbinary runs; nobody chooses;USE_FLINT=OFFreports 36failures of which 32 are the missing library, and one target cannot even link there;
timeoutis GNU coreutils and absent on macOS, where an unguarded call exits 127and a pass/fail loop reads that as a failing test. During development this produced a "0 passed,
444 failed" run that looked like catastrophe and was a missing binary.
gtimeoutis used whenpresent, and its absence is reported rather than silently turning every result red.
tests/known_failures.txt— the standing failures, each with a one-line reason. A listed failuredoes not fail the run; an unlisted one does; a listed test that starts passing is named so its line
can be deleted. Without this the gate would be red on the day it lands, and a gate that is red on
arrival teaches people that red means nothing.
Not in this PR, because it is already in
main:flint_bridge_testsis built only whenUSE_FLINT=ON(it callsflint_cyclotomic_gcd, which lives inside#ifdef USE_FLINT, so with FLINToff the target fails to link and takes
makeintests/with it). Mentioned because thetestsjob depends on that being true.
Testing
tools/run_test_suite.shrun against this branch (=mainat 9ee372e): 437 passed, 7 failed,7 baselined, no unexpected failures. The baseline was seeded from that run, not from another
branch — four entries are long-standing, and three are the NMinimize/FindMinimum tests added on
2026-08-16/17, each failing deterministically across three runs, so they are convergence
assertions the optimizers do not currently meet rather than flaky tests. Every line carries its
reason; none is mine to fix, and they should be short-lived.
mainright now and nothing would have reported them.
nm src/imageio.oon this branch: no exportedstbi_*symbols, 139 file-local.sh -n tools/run_test_suite.shclean.make check-c99,check-packed-aware,check-image-packing,check-menu-idsunchanged and green.Notes for review
testsjob builds 400+ binaries and will be the slowest job in the workflow. If PR turnaroundmatters more than simplicity, it shards cleanly over a
matrixon the first letter of the binaryname — happy to do that instead.
testsandbuild-graphicsas required status checks in branch protection is whatturns them from advisory into a gate; that is a repository setting, not something this PR can do.