Repository navigation
Test framework, stage 3: llcommon's tests in Catch2 - #459
Conversation
llsdserialize_test.cpp held one group that runs Python's llsd package as an independent implementation, and so made the whole file a Python test. That group moves, with the helpers only it used, to llsdserialize_python_test.cpp, the Python test now, and the rest runs without an interpreter. The includes each half no longer needs go. The two files give exactly the results the one gave: 84 tests, 72 in llsdserialize_test and the 12 of the Python group. In stage 3 the Python file joins llleap and llprocess in llcommon_python_tests, and the rest llcommon_tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A TESTS(CLS, OTHER, N0..N3) macro made eight of llsingleton_test's fifteen tests, four for each order of A and B, each a copy of the same steps with the class names pasted into its expected log. A test a macro makes is one tut2catch.py cannot convert or map. The steps are now one helper, check_order<CLS, OTHER>, given the dependency flags and the two halves of the log it expects, and the eight tests are written out, each naming its own case. The eight give the same results under the same names, and still fail when an expected log is wrong. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LLURI declared operator!= alone, "required for tut", whose ensure_equals compared with !=. Catch2's CHECK(a == b) needs ==, and under C++20 a declared == gives != too, so == replaces it: two URIs are equal when their strings are, as before. llsd_test's helpers compare an LLURI as they compare every other type, and need it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llprocess's Windows tests launch the test executable itself as a helper
that owns child processes and then exits, to show the children die with
it. Under TUT the relaunched executable ran its tests until the first of
llprocess's fixtures saw the helper's environment and exited. In a
Catch2 binary of several files that would run the other files' tests in
the child first.
A test file now registers such a helper by name, AL_TEST_HELPER(name)
{ ...; return code; }, as it registers a test case, and launches its
binary with Helper::argument(name), --al-helper=<name>. The main finds
that argument before Catch2 sees any, sets the process up as for a test
run (APR, LLError, LL_ERRS throwing, LLTrace's recorder), runs that
helper alone and exits with its code. An unknown name exits 2; two
helpers of one name abort the binary as it starts.
test_support_tests gains a helper and two entries: one runs it and
expects its exit code, its report that APR is up and no test run, and
one names a helper that is not there. Each fails when the main skips the
setup or leaves the argument to Catch2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sites that never run
Two gaps llcommon's conversion found:
- Catch2 writes a value it prints into its XML report, \xNN for a
control character or a byte that is not UTF-8, but lets vertical tab,
form feed and a UTF-8-encoded surrogate through, which XML forbids
too. commonmisc's binary LLSD checks print all three, and its report
could not be read. The report is read through a filter that writes
those as Catch2 writes the rest, and that reads on when a surrogate's
bytes straddle two reads.
- Every assertion in a passing case's body must have run, but a few
legitimately never do on a host: in a loop that should not turn, or a
branch of an if constexpr the build discards (alsimd's AVX checks on
arm64). check --unrun names them in a JSON, {file: {line: why}}, each
reviewed, and they are listed with the reason rather than counted a
mismatch. Every other site that never ran still is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TM20, the third wave of stage 3: llcommon's 58 TUT files, 733 tests, become four Catch2 binaries, each file still run in a process of its own: - llcommon_tests, 53 files, alsimd at the tree's tier among them, and lleventcoro, which shares a process now that it keeps one app for it; - llcommon_python_tests: llleap, llprocess and llsdserialize_python, which drive a Python peer; - llsingleton_test and llapp_test alone: one deletes every singleton, the other makes and ends an LLApp, and LLCommon with it, in each case. alsimd at each other x86-64 tier joins the kernels in llmath's simd_tests. wrapllerrs.h, whose users are all here, fails a search of the captured log with FAIL rather than a tut::failure. The files are tut2catch.py's output, with the names the idiom pass that follows chose, and these hand resolutions of what it marked TM-CHECK, applied by a script and each reviewed: - ensure_equals() in code this Mac does not compile (lldate's parser boundary block, Windows-only code in llprocess, llstring_utf, stringize and llsdserialize_python), and in llsd's helpers, which compare an LLURI too: UNSCOPED_INFO and REQUIRE(a == b); - llinstancetracker's two set_test_name(desc) calls go, the names given through --names; - llleap's and llsdserialize_python's catches of tut::failure catch ll_test::Failure, which carries no message, and test_or_split's `throw e` is a rethrow; - lldependencies' display<std::string>, which only TUT's failure messages reached, is [[maybe_unused]]; - lleventcoro's one app, and llapp's fixture as it ends each app, tell the state guard what an LLApp changes on purpose, as lllogin's does. tut_parity.py, against the TUT executables' recorded results (732 tests: 729 ok, 3 known skips; one more is Windows-only), holds for all 58 files under --recounted, and plain for the 47 the script did not touch, with nine assertion sites reviewed as never running in a passing case here: alsimd's AVX checks under if constexpr, and three in loops that should not turn. All 62 entries pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The idiom pass over TM20's 58 converted files, as for llmath, TM18 and TM19: - every case named for what it checks: 627 of 733 names changed, 232 of them #N, and many taken from set_test_name or comments that ran long, were cut off, or carried commas, quotes or brackets; - messages that only restated their assertion are gone: 3,040 became 760, and 88 CAPTUREs say what string-building said before; - asserting helpers are named for what they do now (47 require_*), with a scoped INFO or CAPTURE in place of a message built by hand; - 261 REQUIREs that guarded nothing are CHECKs, and 11 CHECKs that guard a read after them are REQUIREs. No case was added, removed or moved, and no assertion: per case, each assertion compares what it compared before, in the same order. Some changed only in form: an UNSCOPED_INFO and REQUIRE(false) became a FAIL (six), a helper's name changed inside its call, casts and parentheses that printed nothing went. A few edits are in code this Mac does not compile (Windows-only branches of llprocess, llsdserialize_python, llstring_utf and stringize, and lldate's parser boundary block), each reviewed line by line. tut_parity.py --recounted holds for all 58 files, with eight assertion sites reviewed as never running in a passing case here, and all 62 entries pass. Bugs in the tests that the pass found are left for their own change, since a conversion keeps behaviour: - llstreamtools: a \r check labelled "space at end"; a message said a lone quote is kept where the expectation removes it; the get_word/skip_to_next_word comparison never reaches the hyphen it describes; fullread compares one byte short; return values assigned and never checked; an empty get_brace_count case. - llstring: startsWith/endsWith messages swapped; a branch tests s1 after format rewrote it; checks commented out. - alregex: a check that passes whenever firstByteRange returns false. - alrecoverystore: three front() calls with no size checked. - lleventdispatcher: three matchers with actual and expected swapped. - lazyeventapi: a case that passes trivially and leans on case 1. - llpounceable: an empty compile-fail case. - llsdserialize: the cross-compatible "map" case starts from an empty array; srand() reseeded on every pass; comments copied between cases. - llsdserialize_python: an ECHILD from waitpid passes without the exit code checked. - llsd: REQUIRE(true). - stringize: the Windows case checks nothing elsewhere. - llstring_utf: stale comments, and two deleted messages that were wrong about what their check showed. - llinstancetracker: a snapshot iterator dereferenced three times with its size unchecked. - llsingleton: an empty first case; a bare `throw;` outside a handler. - llunits and lluuid: comments about what is built, cast or tested that the code does not do. - lltimer: bounds of 0.04 s and 5 ms that a loaded machine could miss. - lldeadmantimer: an isExpired() call that drops the CPU arguments, so the two checks after it cannot fail. - lldate: the boundary block names macros that are commented out. - llfile: sizeof a const char*, eight bytes by coincidence. - commonmisc: &bin[0] on a vector not checked non-empty; a max_bytes shorter than the text it parses. - llprocess: one find() checked twice where the second meant an offset; a CHECK inside a try that only a failed REQUIRE reaches; polls that race a script's truncating write (in TUT too). - Assertions off the test's own path: llcond's two REQUIREs and lleventcoro's explicit_wait REQUIRE run inside coroutines, and llprocess's postend REQUIRE inside an event pump's listener. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A REQUIRE that fails throws, which inside a coroutine LLCoros catches and keeps to rethrow in the main fiber. The failure is already recorded, but the throw ends the coroutine. In llcond's two-coroutine case it ended it before it set the value the test then waits for, and the case hung until it was killed. In lleventcoro the throw would wait to be rethrown in whichever case came next. In llprocess's postend listener it would go back through LLProcess's own handling of a child's end, mid-post. Each of the four guards nothing after it, so each is a CHECK, which records the failure and lets the rest run. A wrong expectation in each now fails its case at once, and llcond's no longer hangs. No assertion in llcommon's tests runs on a thread of its own: the threads only work, and each case checks their results after joining them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The autokill tests launch the test executable again as a helper that starts children, writes their ids and exits, so the test can see Windows end the children with it. The relaunched executable ran its tests until llprocess's fixture, seeing the helper's environment, ran the helper and exited. In llcommon_python_tests that would run llleap's and llsdserialize_python's tests in the child first, or any case at all under a random order. The helper is registered as llprocess_autokill and launched with --al-helper=llprocess_autokill, so the main runs only it; the fixture no longer looks. This code is Windows-only and is not compiled here: it uses only what test_support_tests shows working on every platform, the registration, Helper::argument and the main's dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he process The harness already checks what every LL process has: the working directory, the environment, the floating-point state, the C locale, the APR pool, LLTrace's master recorder, the LLApp and LLError's fatal function. llcommon_test_checks, an OBJECT library in llcommon_tests and llcommon_python_tests, adds: - the live thread pools and work queues, by name: a case that leaves a worker running leaves it working, and counting LLSD objects, through every case after it; - LLError's default level. It also ends LLEventPumps after every case, as the plan has it, so a pump a case made, and the listeners still on it, cannot reach the next; lazyeventapi's cases did this for themselves already. One case left a thread running: alrecoverystore's first write makes ALRecoveryWriter, a singleton whose one thread the viewer stops in its cleanup. Its tests now make it once for the process, before the first case that writes would, and stop it in the harness's teardown, telling the guard, as lllogin's accessor does for its app. Each check was shown to name the case responsible: a case changed to leave a thread pool, a work queue or a lowered log level behind was named for each. Each binary passes 10 random orders and 20 repeats with no guard report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedReview was skipped as selected files did not have any reviewable changes. ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Summary
Merge Risk: ⚪ Minimal · up to
|
Description
The third wave of stage 3 of the test framework's move from TUT to Catch2 (TM20): llcommon's tests. 58 TUT files with 733 tests become four Catch2 binaries. Each test file still runs in a process of its own. It builds on #455's harness and tooling.
llcommon_testsllcommon_python_testsllsingleton_testllapp_testalsimd's builds for the other x86-64 tiers join the batch kernels in llmath's
simd_tests.Before converting, under TUT, each with the same results:
llsdserialize_python_test.cpp, so the rest runs without an interpreter. The two files give the 84 results the one gave.TESTSmacro, which the converter can't convert or map. They are written out over one helper.Production: 9a876de,
LLURIcompares withoperator==, in place of theoperator!=TUT needed. Under C++20,!=comes from it.Harness and tools:
--al-helper=<name>.AL_TEST_HELPER(name). The main finds the argument before Catch2 sees any, sets the process up as for a test run, runs that helper alone and exits with its code.test_support_testsentries cover it.tut_parity.py:check --unrun <json>lists reviewed assertion sites that never run in a passing case on a host, rather than counting them as mismatches.The conversion:
a7fad41, the conversion:
--names;c8b7e58, the idiom pass:
#N;CAPTUREs;require_*;No case and no assertion was added, removed or moved.
03a5b6d, assertions off the test's path:
94177ce, llprocess's Windows autokill helper runs through
--al-helper. This code is Windows-only and isn't compiled on the Mac I tested on.9f14a44, this wave's guard checks:
LLEventPumpsis ended after every case.I broke a test for each check on purpose, and each check named the case responsible.
Related Issues
Issue Link: none. This follows #455 (TM19).
Checklist
Additional Notes
ctest -LE manualpass;test_support_tests,llmath_tests,llmessage_testsandllcorehttp_testspass, their combined runs included.tut_parity.pyholds all 58 files to the TUT executables' recorded results (732 tests: 729 ok, 3 known skips; one more is Windows-only).--recounted, and plain for the 47 files the resolution script didn't touch.--recounted, with eight assertion sites reviewed as never running here (alsimd's AVX checks underif constexpron arm64, and two loops that shouldn't turn).~LLAppnever clearssApplication, soLLApp::instance()keeps pointing at a deleted app. Every caller I found runs while the app is alive, so the bug is latent.🤖 Generated with Claude Code