Skip to content

Latest commit

 

History

History
640 lines (556 loc) · 38.8 KB

testing_strategy.md

File metadata and controls

640 lines (556 loc) · 38.8 KB

Load, soak, and fuzz testing

morph's unit suite (tests/CMakeLists.txt, Catch2) is deterministic, single-shot, and in-process: every test picks a fixed input and asserts one outcome. Four additional, opt-in test categories complement it by exercising the framework over distributions of input and time, rather than hand-picked examples: a fuzz harness, a soak test, a load/latency benchmark, and an adversarial cross-socket run. None of these run in the default ctest sweep or the fast unit build — each lives behind its own CMake option, default OFF, so an ordinary cmake --build

  • ctest is unaffected by their existence.

Contents

Fuzz harness (tests/fuzz/)

Built only under -DMORPH_BUILD_FUZZERS=ON; requires Clang, since it uses libFuzzer (-fsanitize=fuzzer), the compiler-builtin coverage-guided fuzzing engine — no third-party fuzzing dependency or service. Configuring this option under a non-Clang compiler fails the configure step with a clear message rather than silently building nothing.

Two targets:

Target Exercises Invariant
fuzz_wire_decode morph::wire::decode (the outer envelope parse) and, for a decoded "execute" envelope with a non-empty body, ActionTraits<FuzzInnerAction>::fromJson(body) (the inner re-parse — see wire.md's "the body double-parse hazard") Every input either parses (at each stage) or throws std::runtime_error. Never crashes, trips a sanitizer, or hangs.
fuzz_dispatch_execute morph::backend::RemoteServer::handle end to end, against a live server with one pre-registered Fuzz_DispatchModel instance Every input yields a reply that itself decodes as a wire Envelope with kind "ok" or "err". Never crashes; never hangs (bounded by libFuzzer's own -timeout, not by anything in the harness).

Both binaries link -fsanitize=fuzzer,address (via the apply_fuzzer() CMake function in cmake/compiler_options.cmake), so a crash found while fuzzing is also an AddressSanitizer memory-safety finding, not just a caught exception.

Corpus. tests/fuzz/corpus/wire_decode/*.txt and tests/fuzz/corpus/dispatch_execute/*.txt hold hand-picked seeds drawn from the existing hardening tests (test_wire_hardening.cpp, test_server_limits.cpp) — a valid register/execute/deregister envelope, a duplicate-key envelope, a moderately-nested body, a malformed-UTF-8 body, and plain garbage. tests/fuzz/findings/wire_decode/ and tests/fuzz/findings/dispatch_execute/ hold the committed reproducers and are where any input that triggers a crash/hang/sanitizer report gets committed as a permanent regression case, once the bug it found is fixed — see "Known findings" below.

CI-integrated regression check. ctest -R fuzz_.*_replay runs each target once per file in its corpus + findings directories (libFuzzer's single-run replay mode — passing individual files, not a directory, so it replays and exits rather than starting a mutating fuzzing session) and fails if any input crashes. This is fast and deterministic, suitable for CI; it is not a fuzzing campaign.

The input globs are deliberately extension-agnostic (*, not *.txt): a fuzzer input is an arbitrary byte string, and a libFuzzer-minimized reproducer is conventionally saved as .bin. An extension-scoped glob silently skips whatever it does not match — the committed reproducers are .bin, so a *.txt glob would replay only the seeds and leave the regression cases unrun. An empty glob is a configure-time FATAL_ERROR: a replay invocation with no FILE arguments is an unbounded fuzzing run, not a regression check. The nightly linux-all-features job additionally asserts, after the run, that every file under tests/fuzz/findings/ was actually referenced — a guard that never fires is indistinguishable from one that works.

Running an actual campaign (manual / scheduled, not CI-per-commit):

cmake --preset clang-release -DMORPH_BUILD_FUZZERS=ON
cmake --build build/clang-release --target fuzz_wire_decode fuzz_dispatch_execute
./build/clang-release/tests/fuzz/fuzz_wire_decode -max_total_time=3600 tests/fuzz/corpus/wire_decode

Passing the corpus directory as the sole argument makes libFuzzer both seed from and write newly-discovered coverage-increasing inputs back into that same directory — review and keep genuinely new, interesting cases; move anything that crashes into findings/ instead, once the underlying bug is fixed (see below).

Known findings. The committed reproducers under tests/fuzz/findings/ each guard a property of morph::wire's glaze-based parsing that fuzzing showed can fail:

  • Decode never reads past the buffer (tests/fuzz/findings/wire_decode/skip_ws_heap_overflow.bin). Every glz::read<> call site that accepts an arbitrary string_view — morph::wire::decode, journal::fromJson, BRIDGE_REGISTER_ACTION's generated fromJson/resultFromJson, FileOfflineQueue's fromJson — sets .null_terminated = false. With glaze's default (true), skip_ws scans past the buffer's real end for a terminator a string_view never guarantees, and a 5-byte input ({"{k) is an AddressSanitizer heap-buffer-overflow any peer can send a RemoteServer. The option costs nothing beyond an optimization that never legitimately applied. session_auth.hpp's token-claims decode parses a const std::string&, which has the standard's null terminator, and needs no option.
  • Every reply and every body is valid JSON whatever bytes it carries (tests/fuzz/findings/dispatch_execute/err_reply_control_char_roundtrip.bin). Glaze's default writer escapes " and \ but not ASCII control bytes (0x00-0x1F, 0x7F), so a string carrying one produces invalid JSON — and, when the same string also holds a \ or ", glaze's chunked write path corrupts the byte into two 0x00 bytes while the output still decodes. Three layers close it:
    • err messages go through wire::makeErr, whose detail::sanitizeControlChars replaces each control byte with a printable \xHH placeholder. An err message echoes untrusted text (an unrecognized Envelope::kind, a caught exception's what()) and is log-bound, so substitution is right there: a raw 0x1B would carry an ANSI escape into the reader's terminal.
    • Every other envelope string — body, modelType, actionType, contextKey, typeId, the session's principal/token — carries caller data that must round-trip byte for byte, so wire::encode writes with wire::detail::EscapingWriteOpts (glaze's escape_control_characters), which is lossless in both directions.
    • Action and result bodies are written not by wire::encode but by ActionTraits<A>::toJson / resultToJson (registry.hpp's BRIDGE_REGISTER_ACTION), with the same instrument one layer down, model::detail::EscapingWriteOpts (see docs/spec/core/registry.md, "Control bytes in action and result bodies"). Action bodies are pure caller data (a paste's content, a chat message, a filename), so they are at least as exposed as the envelope; an application-ladder rung replays tests/fuzz/findings/ as stored user content to cover that round trip.

Each is also covered by a dedicated regression test in tests/test_wire_hardening.cpp ("Bug C" through "Bug G") in addition to the fuzz_*_replay findings above.

Soak tests (tests/soak/)

Built only under -DMORPH_BUILD_LOAD_TESTS=ON (requires MORPH_BUILD_TESTS=ON, since these link Catch2). Two Catch2 test cases in the morph_soak binary:

  • test_soak_switch_backend.cpp — cycles Bridge::switchBackend() between a LocalBackend and a fresh SimulatedRemoteBackend/RemoteServer pair every cycle, firing a burst of execute() calls each cycle. Asserts every issued completion eventually resolves (value or BackendChangedError — see backend.md), that the live model-instance count settles back down after the run (a leaked IModelHolder would show as monotonic growth), and — on Linux, where /proc/self/status is readable — that RSS growth across the run stays under a configurable bound. The final iteration always lands on a fresh LocalBackend before the last RemoteServer is dropped, so the bridge's active backend never outlives the object it references (see backend.md's "Lifetime & ownership").
  • test_soak_reconnect_churn.cpp — wires a real NetworkMonitor → ReconnectCoordinator → SyncWorker pipeline exactly as offline.md's "End-to-end integration" shows, and flips online/offline hundreds of times. Asserts the offline queue is fully drained after every flap and every onOnline() reconnects (this test's tryReconnect never fails).

Both are CI-sized by default (a few hundred cycles, seconds to run) and scale to a real multi-hour soak via environment variables, without touching code:

Variable Default Test
MORPH_SOAK_CYCLES 200 switch-backend churn
MORPH_SOAK_EXECUTES_PER_CYCLE 20 switch-backend churn
MORPH_SOAK_RSS_SAMPLE_EVERY 20 switch-backend churn
MORPH_SOAK_RSS_GROWTH_KB_MAX 102400 (100 MiB) switch-backend churn
MORPH_SOAK_FLAP_CYCLES 150 reconnect churn

A production morph::observe metrics seam already exists (observability.md) and instruments RemoteServer/ LocalBackend dispatch, SyncWorker::run(), and ReconnectCoordinator::onOnline() directly. These soak tests deliberately do not depend on a particular MetricSink being installed in the test process, though: they instrument themselves directly instead (a model-local instance counter, /proc/self/status, and locally-owned atomic call counters), so their pass/fail signal never depends on how (or whether) a host application has wired up observability.

Both morph_soak and morph_bench are sanitizer-instrumented when AF_SANITIZER is set. They are opt-in, so no default sanitizer leg pays for them; the reason to instrument them rather than exempt them is that churn over thousands of cycles is exactly the shape of test whose finding is a leak or a race and not a failed assertion. Under a sanitizer preset the benchmark's published numbers are not comparable with an ordinary run's and are not meant to be — there it is a correctness run over the dispatch path.

Load / latency benchmark (tests/bench/)

Built only under -DMORPH_BUILD_LOAD_TESTS=ON alongside the soak tests (same option; different target, morph_bench). bench_dispatch_latency.cpp drives a RemoteServer directly against a trivial echo model (isolating framework overhead from business logic):

  • Latency — 2000 serial (concurrency-1) round trips; reports p50/p95/p99 wall time in milliseconds (nearest-rank percentile over the sorted sample).
  • Throughput — a window at each of concurrency 1/2/4/8/16, reporting executes/second (MORPH_BENCH_WINDOW_MS, default 200 ms).
  • Both phases run MORPH_BENCH_TRIALS times (default 5), and the run reports the best, median and worst trial of every figure rather than one number — see below.
  • Writes bench_dispatch_latency.json into the build directory. The p50_ms/p95_ms/p99_ms/throughput keys carry the best trial; trials, latency_samples_per_trial, throughput_window_ms, p99_ms_median_trial, p99_ms_worst_trial, each throughput point's executes_per_sec_worst_trial, and a trial_detail array with every trial's own figures are additive. Successive runs are archived and diffed for a regression, and trial_detail is what a diff finer than the gates below has to read.
  • Enforces two coarse regression gates via CHECK: p99 <= MORPH_BENCH_P99_MS_MAX (default 50.0 ms) and concurrency-1 throughput >= MORPH_BENCH_MIN_THROUGHPUT (default 500.0 executes/sec). Both are environment-variable-overridable so CI hardware differences don't need a code change, and both read the best trial — see below.

The serial phase must not wait on a polling step, or it reports the poll. Waiting on each reply with morph::testing::WaitReply does exactly that: its await() calls waitUntil, which sleeps 5 ms between predicate checks. Measured that way, same binary, 20 processes per configuration, Release:

idle 16-way oversubscribed
p50 5.0562 / 5.0714 / 5.0740 ms 0.0166 / 0.0167 / 0.0216 ms
c=1 executes/sec 171467 / 175928 / 178855 311.9 / 1749.9 / 18627.1

A 302x swing in the headline figure, selected by machine load, and neither mode is the dispatch latency: the same idle processes report ~176k executes/sec at concurrency 1, i.e. a round trip of about 5.7 µs. An idle machine reports one whole sleep step per call; a busy one reports the case where the reply beats the caller's first predicate check. 311.9 executes/sec is also below the 500/sec floor this file enforces, so the gate would fire on machine load rather than on morph. The benchmark therefore uses a local condition-variable reply sink, and a blocking drain at the end of each throughput window for the same reason — a polling drain sits inside the window's own elapsed time.

It reports a distribution because a wall-clock figure has no regime to pin. morph_bench_alloc can pin its race, and an allocation count then comes out exact. Contention is not a mode, it is a tax, so this benchmark reports the spread over trials instead. The gates read the best trial: contention can only make latency worse and throughput lower, so the best of N is the least contaminated estimate of what the code costs, while a real regression moves every trial including the best. Measured, 20 processes per configuration, best-of-five figures:

p99 (ms) c=1 executes/sec
Release idle 0.0090 / 0.0096 / 0.0104 173169 / 176282 / 181641
Release loaded 0.0181 / 0.0210 / 0.0259 56825 / 61211 / 63479
Debug idle 0.0325 / 0.0343 / 0.0360 42560 / 43150 / 43996
Debug loaded 0.0598 / 0.0608 / 1.8717 1549 / 22684 / 22990

The Debug rows are the ones the defaults must hold for: the only CI leg setting MORPH_BUILD_LOAD_TESTS=ON is the nightly linux-all-features job (nightly-slow-checks.yml), on the gcc-debug / clang-debug presets, and its ctest --preset passes no label filter. The two ceilings are therefore unchanged, and that is a measurement rather than timidity: Debug-under-load still spans 28x on throughput and 57x on p99 for the same binary, so no pair of constants separates a regression from a contended runner. 50 ms is ~27x the worst best-trial p99 measured and 500/sec ~3.1x below the worst best-trial throughput — a dispatch path made three times slower passes both. Setting the floor from a 12-core box instead turns 3 of 20 Debug-under-load processes red, so that is not the way out of it either.

The property those loose ceilings look like they leave ungated is gated elsewhere, tightly. An allocation gate is what to weigh before tightening any wall-clock ceiling, and it exists: bench.alloc_budget (below) is set by rule at one allocation above the figure the benchmark measures — not a tolerance band — and that figure came out identical on every one of 20 processes across three toolchains, idle and loaded alike: a one-allocation margin, on a quantity machine load cannot move. The two numbers are deliberately not restated here. Both live in tests/bench/CMakeLists.txt — the ceiling as MORPH_ALLOC_BUDGET_PER_CALL, the measured figure in the comment that derives it — and they move together every time the dispatch path gets cheaper, while the argument they support does not; so this paragraph keeps the argument, and that file the current values. So "dispatch does not get more expensive" is already gated to a resolution no wall-clock constant on any host can approach. What the two ceilings gate is the residue: a regression that costs time without costing allocations — a spin, a syscall, a lock held longer. That is a real class, and it is the only class they are for.

Two things a future tightening needs, which the benchmark produces.

  • A cross-process distribution (bench_dispatch_latency.jsonl, beside the per-process .json, overridable with MORPH_BENCH_LEDGER, off to disable). MORPH_BENCH_TRIALS gives a distribution over trials within a process, which mitigates the spread but does not record it — one process still prints one triple and cannot say where it sits among others. Each run appends one line (pid, the headline percentiles, concurrency-1 throughput, load_1m, inject_delay_us) and prints its own rank among every line already there, so N unorchestrated runs produce the distribution a candidate ceiling would be read off. The load average is on the row and not only on stdout, because load sweeps put the same binary 28x apart on throughput between an idle box and a loaded one: a figure without its load is not comparable with another figure. Rows are appended O_APPEND in one write and a row that cannot be parsed is skipped, because a diagnostic ledger must never redden a benchmark.
  • An injectable regression (MORPH_BENCH_INJECT_DELAY_US), which spins for that many microseconds inside the model's execute — inside the round trip both phases measure. A spin and not a sleep: a sleep yields the core and would flatter the throughput phase. This is what makes "what size of regression does this gate catch?" a command rather than an argument, and it is what any candidate ceiling should be shown firing against before it lands. A run carrying an injection says so on stdout and on its ledger row.

Both ceilings fire, and the size they fire at is measured. Release, clang 22.1.8, 12-core x86-64 Linux, MORPH_BENCH_TRIALS=1, load average 11.9–12.8:

injected p99 (ms) c=1 executes/sec p99 gate throughput gate
0 0.0142 168862 pass pass
1000 µs 1.02685 989.2 pass pass
2000 µs 2.03428 495.5 pass FAIL
3000 µs 3.18618 331.2 pass FAIL
60000 µs 61.0339 16.7 FAIL FAIL

The baseline round trip is ~5.9 µs, so the throughput floor — the tighter of the two — first speaks at roughly a 340× regression and the p99 ceiling at roughly 8500×. The ratio, not the time, is the portable part: reproduce the table on any host with MORPH_BENCH_INJECT_DELAY_US.

What is not established, and is why the ceilings stay loose: nobody has characterised the CI runner. Both sets of figures above come from a workstation, which is exactly the misleading configuration to set a CI ceiling from. Guessing a second time from the same box would be the first mistake with a different number.

The echo model/action (BenchEchoModel/BenchEchoAction) are declared at file scope, not inside the file's anonymous namespace with its other local helpers — Glaze's reflection needs external linkage to mangle the type name, the same requirement tests/fuzz/'s harness fixtures document.

Allocation census (bench_dispatch_allocations.cpp, target morph_bench_alloc)

A second binary under the same option, and deliberately not a second case in morph_bench: it replaces the global operator new/delete, which is process-wide and would perturb any other measurement sharing the binary. It counts the heap allocations one Ping -> Pong round trip costs through LocalBackend — 50 warm-up calls excluded, every call waited out, no JSON and no socket — and prints the total, the bytes, and (with --attribute) the size of every allocation in one steady-state call.

It also runs a group of narrower censuses over morph's four registry lookups, each one taken over ids past libstdc++'s 15-character SSO buffer and again over ids inside it — the cost is entirely id-length-dependent, so a census over one side alone either measures zero or overstates the saving, and morph's real ids straddle the line ("CreateSwimlane" is 14 characters, one under):

census opaque key transparent key pure lookup?
ActionDispatcher::coalesce + requiredFieldsFor 2.00 / 0.00 0.00 / 0.00 yes
PayloadMigrationRegistry::find 2.00 / 0.00 0.00 / 0.00 yes
BridgeHandler::executeJson 24.07 / 21.06 21.07 / 21.06 no — a whole round trip
ModelRegistryFactory::create 2.00 / 1.00 2.00 / 1.00 no — constructs a holder; parked

The first two decode nothing and execute nothing, so what they allocate is exactly what looking a registry key up costs. The last two do more than look up, so their figure is a floor plus the key and what is comparable between runs is the difference between the long-id and short-id columns.

It exists because a scoping number goes stale the moment anything on the path changes, and re-deriving one from a prose description of how it was once taken is how a fix ends up built against a figure nobody re-checked. The census re-takes it on demand instead.

It is both an instrument and a control, and which one it is at any moment depends on the flags. Run bare, it asserts nothing and the number it prints is the whole output; a green run of it in that mode proves nothing. Given --budget=<n> and/or --lookup-budget=<n> it fails when the figure exceeds the ceiling, and it is registered with ctest in exactly that form, as bench.alloc_budget.

The reason it took a per-toolchain budget to get there is that an allocation count is standard-library specific, not only morph-specific: std::function's inline buffer and std::string's SSO threshold differ between libstdc++, libc++ and MSVC's STL. So the ceiling is the cache variable MORPH_ALLOC_BUDGET_PER_CALL, defaulted to a figure measured on the toolchains that actually build this target in CI (Linux clang-debug and gcc-debug), left unset on MSVC where nothing has measured it, and settable to 0 to disable the gate. tests/bench/CMakeLists.txt carries the measurements the default was chosen from and the headroom argument.

The --lookup-budget half takes no headroom at all: with transparent keys a pure registry lookup allocates nothing, for ids of any length, and that is a property which either holds or has regressed. The ctest case passes --lookup-budget=0, and it covers ActionDispatcher and PayloadMigrationRegistry.

--id-length-budget covers the third registry, ActionExecuteRegistry, which cannot be probed on its own — it dispatches whatever it finds — so what is gated is the gap between an executeJson over long ids and one over short ones. 0.5 rather than 0, and the 0.5 is measured: the figure reads 0.01 because of a deterministic two-allocation one-off across a census's 200 calls, shown to follow census order rather than id length by swapping the two censuses, and identical in all of 10 processes. Making either half of that key opaque again takes the gap to 3.00 (the Key's two std::strings, plus the one executeJson builds from ModelTraits<Model>::typeId()), so 0.5 separates the residue from the thing guarded with wide margin on both sides. Like the lookup half it needs no per-toolchain default: it is a difference between two runs of one build.

All three ceilings were checked against a reverted fix. Removing PayloadMigrationRegistry's PairKeyEqual takes --lookup-budget to 2.00 and turns the case red; restoring ActionExecuteRegistry's Key{...} temporary takes --id-length-budget to 2.01; restoring executeJson's std::string{typeId()} alone takes it to 1.01.

Both halves are checked against a reverted fix rather than only against themselves: each was run with the transparent keys taken back out, and each went red. A budget that has never been seen to fail is the control that measures nothing, which AGENTS.md ("Verify rather than assert") warns about.

The dispatch census pins a race, and without that pin it is not comparable between runs. A dispatch and the handlers attached to it race: win, and each handler joins a vector the settle drains; lose, and each takes CompletionState's attach-after-ready path, which costs differently. Ungated, that makes the headline figure bimodal — measured, interleaved, 20 processes per configuration: ~16.95 on an idle machine, ~13.06 with the machine 16-way oversubscribed, same binary. The benchmark therefore holds the worker across execute() and both attaches, so it always measures the attach-before-settle regime: the more expensive of the two, and the one a real GUI client is in.

With the race pinned, x86-64 Linux, clang 22.1.8 / libstdc++ 16.2.1, the opaque-key build costs 17.06 allocations per local round trip and 2.00 allocations per registry lookup for ids past the SSO buffer; the transparent-key build costs 14.06 and 0.00 per lookup — exactly 3.00 removed, in every one of 40 processes, idle and loaded, on clang Release, clang Debug and gcc Debug alike. A figure from an ungated benchmark is not comparable with either.

Concurrent-user benchmark (scripts/scenario/bench_bookmarks.py)

morph_bench measures one dispatch inside one process. This benchmark measures a whole server from outside it: a ladder_bookmarks_server process, its transport, its authorizer and its model lifecycle, under thousands of simulated users who each connect, sign in, do 5-10 actions with think time, and disconnect. Its question is how many concurrent users one server carries, and which thread runs out first. scripts/scenario/README.md has its flags.

It measures the framework rather than the rung when the server runs with the in-memory store (--store memory): each model instance then keeps its own bookmarks in a container, journals nothing and touches no database, so an action costs the framework plus a container operation. --workload framework goes further and replaces every action with a Login round trip.

What it reports per run, and why each figure is there:

  • Server CPU per thread, from /proc/<pid>/task. A server whose total CPU is low can still be saturated, on one thread; the per-thread split is what shows which one.
  • Client CPU, so a run limited by its own load generator says so.
  • Other processes' CPU on the host, from /proc/stat less the server and the clients. The machine this runs on is shared, and a run that competed for it warns rather than reporting a figure that is not comparable with a quiet one.

It is a manual tool, not a CI gate: its figures depend on the host, and its value is in comparing two runs on the same quiet machine.

Adversarial cross-socket run (tests/qt/test_qt_websocket_adversarial.cpp)

Built under MORPH_BUILD_QT=ON (one more file in the morph_qt_tests binary). A hostile QWebSocket client, talking to a real QtWebSocketServer over loopback TCP (same-process, like most of test_qt_websocket.cpp — see that file's dedicated "Process separation: ..." cases for genuine OS-process separation), drives four scenarios:

  1. An oversized frame (body past wire::kMaxEnvelopeBytes).
  2. A rapid flood of 5000 execute messages with no per-message wait.
  3. A duplicate-top-level-key envelope (the wire.md smuggling caveat, over a real socket).
  4. A connection that opens and then never sends a single frame.

After each, an honest QtWebSocketBackend client registers a model and executes an action against the same server and must succeed — the assertion in every scenario is that the server keeps serving honest clients. RemoteServer::LimitPolicy and QtWebSocketServerConfig (see backend.md) are implemented, each with its own dedicated, precise unit coverage (tests/test_limit_policy.cpp, tests/qt/test_qt_websocket.cpp) — this file does not re-derive that coverage. Every server here uses the default, unconfigured QtWebSocketServerConfig, under which maxConnections/ messagesPerSecond/handshakeTimeout/idleTimeout are all still unbounded/disabled (0), but maxMessageBytes defaults to wire::kMaxEnvelopeBytes (not 0) — so, concretely: scenario 1 (oversized frame) is actively rejected by the default config and this file asserts that directly (an err reply mentioning maxMessageBytes), while scenarios 2 and 4 (flood, stall) are not capped by default and this file asserts only that the server keeps functioning under them, not that they are rejected.

Test type naming: avoid file-scope name collisions

tests/**/*.cpp files are written independently, and each is free to declare its own local model/action types (OrderModel, Widget, Report, …) for its scenario. A type declared at file scope — not inside an anonymous or named namespace — has external linkage. If two different test files declare a same-named, differently-defined type at file scope, that is a One Definition Rule (ODR) violation: undefined behavior that neither the compiler nor the linker diagnoses, since each translation unit only ever sees its own definition. Which definition the linker keeps for a given call site is link-order dependent, so the bug can pass locally and fail in CI (or the reverse), with no diagnostic pointing at the cause: a bare OrderModel stub in one file can silently win over another file's real OrderModel, so the real one's onBackendChanged() is never invoked and its offline queue never drains.

Convention: wrap local model/action types in an anonymous namespace by default. Only lift a type to file scope when a macro that requires external linkage (BRIDGE_REGISTER_MODEL, BRIDGE_REGISTER_ACTION) needs it — and in that case, prefer a short, file/feature-specific prefix over a generic name (StepILOrderModel, not OrderModel; see tests/test_remote_step_interleaving.cpp). A type declared inside a named namespace is also safe, since its linker symbol is namespace-qualified (e.g. namespace registration::models { struct Report { ... }; }, which is how a test that needs BRIDGE_REGISTER_MODEL/BRIDGE_REGISTER_ACTION at namespace scope keeps its types distinct).

Why UndefinedBehaviorSanitizer (clang-ubsan, cmake/compiler_options.cmake's -fsanitize=undefined) does not catch this. UBSan instruments individual operations within a translation unit at compile time — signed overflow, an invalid enum load, a misaligned access, a bad vptr cast — and its runtime checks fire while that instrumented code executes. An ODR violation across two TUs has no such moment: each TU's own code is entirely well-formed and runs exactly as instrumented: a real OrderModel calls its own onBackendChanged() correctly when compiled and run alone, and a bare stub of the same name in another file is equally well-formed on its own. The defect exists only in what the linker does across the two .o files — silently keeping one TU's definition of BackendChangedNotifiable<OrderModel> for both — which happens before any sanitizer runtime is even loaded. No sanitizer (ASan, UBSan, TSan, MSan) is designed to catch cross-TU ODR violations; this is a structural blind spot of the whole compile-time-instrumentation family, not a configuration gap.

The linker's own ordinary duplicate-symbol diagnostics don't fire either, for a related reason: the affected symbol (BackendChangedNotifiable<OrderModel>, a template instantiation) is implicitly inline, and the linker treats multiple same-mangled-name inline/template definitions across TUs as expected COMDAT folding — the normal, correct case when two TUs instantiate the same template identically. It has no way to distinguish that from "two different definitions that happen to share a mangled name," which is exactly this bug, so it silently keeps one arbitrarily in both cases. (GNU gold's --detect-odr-violations is the closest built-in linker feature, but it is gold-specific, heuristic on this exact template-instantiation shape, and not wired into any preset here.)

So only a source-level check could catch it — the one layer that sees both TUs' declarations before they are compiled down to symbols a linker or sanitizer could reason about. No such check runs in CI: the convention above is kept by review.

Install / export consumability (scripts/check_install_export.sh)

Every other suite in this document builds morph from inside the tree, where include/ is on the include path because the build put it there. That hides a whole class of defect: cmake --install can exit 0 having installed Glaze's headers and a working glazeConfig.cmake — Glaze carries its own install/export rules and gets them for free through FetchContent — while installing zero morph headers and no morphConfig.cmake. Unless a CI leg installs morph, nothing notices. A consumer following the standard CMake workflow then gets a prefix holding someone else's dependency and none of the library they meant to install.

CI-enforced by a dedicated job (install-export in ci.yml), which self-tests the checker first, then runs it. The checker installs morph to a scratch prefix and compiles and runs a consumer project outside the tree — find_package(morph CONFIG REQUIRED COMPONENTS net) plus target_link_libraries(consumer PRIVATE morph::morph morph::net) over one translation unit including morph/core/bridge.hpp, morph/forms/forms.hpp, morph/util/quantity.hpp, morph/util/rational.hpp and morph/version.hpp.

Compiling is the point, not find_package succeeding. Three defects in the install rules produced a prefix that find_package accepted and that no filename inspection would have questioned:

  • installing only the declared FILE_SET HEADERS omits morph/detail/fixed_string.hpp and morph/detail/quantity_equation.hpp, which public headers include, so the installed tree resolves in the build directory and nowhere else;
  • install(EXPORT … NAMESPACE morph::) prefixes the target name, so morph_net arrives as morph::morph_net while every in-tree alias says morph::net;
  • without find_dependency(glaze) in the package config, morphTargets.cmake names an imported target nobody created.

The checker also builds morph_verify_interface_header_sets, which is not part of all and which nothing else in CI builds. That phase is what keeps INTERFACE_HEADER_SETS_TO_VERIFY honest: the detail/ header set is installed but deliberately outside the verified set, because morph/detail/quantity_equation.hpp is included partway down morph/util/quantity.hpp and is not self-contained. It is the one slow phase, and --skip-header-set-verification drops it for the self-test's fast cases.

Two vacuity guards, because a consumability check that measures nothing looks exactly like one that passes: the run fails if no optional component was installed (leaving the exported-name check with nothing to read), and a consumer asking for a component that was not installed must fail at find_package rather than succeeding and breaking at link time.

Self-tested by scripts/test_check_install_export.sh, which reintroduces each defect into a scratch copy of the tree one at a time and requires the checker to catch each one for the stated reason — the checker is tested before it is trusted.

Rung bundles load without warnings

Each rung's screen tests hold one case, <rung>_bundle_loads_without_warnings (bookmarks, bank, lims), in examples/<rung>/screens/tests/. It builds the bundle the rung's server serves and loads every screen and app document in it against the catalog facts the bundle carries, which include each action's effects and records (examples/common/testkit/bundle_warnings.hpp). The case fails on any refusal and on any load-time warning (Document::warnings): a mutation whose action declares no effects and after which nothing refreshes, a refresh declaration the effects already cover, or a watch of a query that runs a mutation with effects. It is what keeps the rungs' actions declaring their effects and their screens free of refresh declarations. It runs in the ordinary ctest -L ladder sweep.

Declined techniques, and why

These are scope decisions about what this library is, not gaps.

  • Crash / power-loss simulation. SQLite ships a storage engine of its own and must survive an OS crash mid-write; morph does not own storage — its offline queue (include/morph/offline) delegates durability to SQLite itself (sqlite_offline_queue.hpp) or to plain file I/O (file_offline_queue.hpp), and a crash-durability claim about those belongs to SQLite's own test suite and the filesystem's fsync contract, not to a simulation morph would have to build and maintain. Declined as out-of-scope for what this library is, not as a gap.
  • A proprietary, dedicated coverage harness (SQLite's TH3). SQLite's headline coverage figure is produced by TH3, which is not shipped with SQLite and is not free to obtain. Citing that number as a model without the harness that produces it would make the claim aspirational rather than checkable. morph's coverage figures are produced entirely by tools already in this repository (scripts/coverage.sh, llvm-cov/llvm-profdata), so anyone can reproduce them.
  • 100% MC/DC as a stated target. This repository measures line and branch coverage, not MC/DC, and does not claim to. Adopting the number without the instrument that verifies it would be exactly the mistake the TH3 point above describes.

Cross-references

Spec Relationship
wire.md decode, kMaxEnvelopeBytes, and the body double-parse / duplicate-key caveats fuzz_wire_decode and the adversarial run target.
backend.md RemoteServer::handle (fuzz_dispatch_execute's target), switchBackend/cancelPending (the switch-backend soak test), the transports the load benchmark drives, and LimitPolicy/QtWebSocketServerConfig (the resource limits the adversarial run's default-config scenarios exercise).
offline/offline.md NetworkMonitor/ReconnectCoordinator/SyncWorker, the pipeline test_soak_reconnect_churn.cpp drives through thousands of flaps.
security.md The hardening tests (test_wire_hardening.cpp, test_server_limits.cpp, test_qt_websocket.cpp) these suites generalise from single-shot examples to distributions/time/a real adversary.
core/observability.md The morph::observe metrics/trace seam already instruments the same dispatch paths these suites exercise; the soak tests deliberately instrument themselves directly rather than depend on a MetricSink being installed in the test process — see "Soak tests" above.