Skip to content

06-testing.md

Latest commit

 

History

History
853 lines (753 loc) · 52.8 KB

06-testing.md

File metadata and controls

853 lines (753 loc) · 52.8 KB
Raw

06 — Testing Standards

Status: Active Derives from: ADR-0070 Owned Tenant BFF Sessions, ADR 0003 — Tenant Isolation Defense in Depth, ADR 0010 — Cross-Module Communication. Public-read additions derive from ADR-0052. Public-renderer controls derive from ADR-0053 and ADR-0054. The seeded browser/audit proof derives from ADR-0056. Native admission, stock-browser fallback and HMR proof controls derive from ADR-0055. The current CI execution windows derive from ADR-0082, which narrowly supersedes the predecessor window clauses in ADR-0075. The public-demo outer-window exception derives from ADR-0085; its implementation and complete hosted qualification remain separate gates. Version-qualified framework cleanup and generated-output proofs derive from ADR-0083. UI localization controls derive from ADR-0027 and the Accepted P02d-6 package. Protected native page/form lifetime and scoped UI controls derive from ADR-0078 and ADR-0079. Their qualification and packet status live in Phase 03.

Test pyramid, conventions, and what every change must cover.

Tenant BFF proofs follow ADR-0070: unit/actual-native route ownership and planted negatives, real warm Keycloak → BFF → app-role AccountSite lifecycle/event evidence, and a separate native HTTPS browser proof of both hosts' cookie storage/send/logout/history. Realm issuance alone proves no BFF session; Phase 06 retains the full browser OIDC journey. TenancyIdentity/TenantBffSessionTests.cs owns the real chain and invokes verify-bff-session.mjs through a bounded private pipe. Its separate stock-browser case invokes verify-bff-browser.mjs against native HTTPS with fake grants/API; natural BFCache observations and the explicit persisted-event counter remain distinct. ADR-0078's separate real entry-pagination case uses more than twenty actual app-role Membership records and the real API; a stock browser imports the real callback's owned entry cookies, follows continuation links and Back, and observes HTML/URL/history confidentiality before second-page selection. This proves navigation after the real code flow, not browser execution of that flow. Only closed scalar observations leave its browser action. Every fixture compiles the copied current native/Next sources, joins owned processes/listeners and exports no raw tokens, cookies, callback/logout URLs, provider text or HTML. Ordinary CI compiles native web/authenticated SDK entries before test discovery/execution within its existing whole-job budget. BFF success also requires unforced group settlement and native exit 0 without a terminating signal. Parent failure closes the private lifetime pipe first and verifies its own temporary directory is empty; forced/incomplete cleanup fails.

Test Pyramid

---
title: LearnStack Test Pyramid
---
flowchart TB
  e2e[End-to-end / Playwright<br/>handful of golden flows]
  contract[Contract & API tests<br/>OpenAPI + provider fakes]
  integration[Integration tests<br/>Testcontainers Postgres + pinned Keycloak issuance]
  arch[Architecture tests<br/>module boundaries + tenant invariants]
  unit[Unit tests<br/>domain + application + UI logic]

  unit --> arch --> integration --> contract --> e2e
Loading

Text fallback (for renderers without Mermaid support — pyramid base → top):

  • Unit tests (base layer, widest) — domain + application + UI logic.
  • Architecture tests — module boundaries + tenant invariants.
  • Integration tests — Testcontainers Postgres and pinned Keycloak issuance. Other providers arrive with their actual consumers (ADR-0035).
  • Contract & API tests — OpenAPI + provider fakes.
  • End-to-end / Playwright (top, narrowest) — handful of golden flows.

We invest most at unit + integration. Architecture tests are zero-flake. E2E covers only what cannot be proven below.

Backend Test Types

Type Project Tool
Unit LearnStack.Tests.Unit xUnit, FluentAssertions
Integration LearnStack.Tests.Integration xUnit + WebApplicationFactory (Docker-free host tests) and, from Packet 6, Testcontainers marked [Trait("Requires","Docker")] — CI runs the two halves in separate jobs by that trait
Architecture LearnStack.Tests.Architecture xUnit with NetArchTest (Mono.Cecil), reflection and source scans
API contract LearnStack.Tests.Contract The published OpenAPI document; the snapshot-diff gate activates in Phase 02d, per § API Contract Tests
End-to-end Shared demo lifecycle (P02d-7) Playwright library + Lighthouse; Phase 06 owns the broader golden-flow suite, per § End-to-End Tests

Unit Tests

  • Cover domain aggregates, use cases (with mocked repos), value objects, pure helpers.
  • No I/O. No HttpClient. No database.
  • One scenario per test. Arrange / Act / Assert.
  • Naming: MethodName_Scenario_ExpectedOutcome.

Integration Tests

LearnStack.Tests.Integration holds four populations, and which one a test belongs to is decided by what it needs, not by what it is about:

  • Host tests — a real HTTP pipeline through WebApplicationFactory, no Docker. Everything that is a property of the API surface lives here: routing, the error shape, idempotency, limits, the tenancy edge. These run in the required backend CI job alongside the unit suite.
  • Data tests — real Postgres via Testcontainers, connected as learnstack_app, one database per test collection — cases that write either roll back or clean up after themselves, because the fixture's seeded row counts are what the isolation assertions compare against. Not Valkey and not SeaweedFS: nothing the backend runs calls either, and both sit behind the gated compose profile (ADR-0035). Everything that is a property of the schema lives here, and every schema-level tenant-isolation invariant does. Packet 6 shipped the fixtures, the four-role provisioning suite, both migration chains, the policies, a two-tenant seed and the schema-level isolation suite; Packet 7 re-runs those cases one layer up — the third population below. All of them connect as learnstack_app, because a test run as the owner or as a BYPASSRLS role passes against inert policies. They run in the ordinary/native Docker execution jobs behind the required backend integration (Testcontainers) aggregate, split from Docker-free tests by [Trait("Requires","Docker")].
  • Request-level isolation tests — a WebApplicationFactory<Program> pointed at a Testcontainers PostgreSQL. Packet 7 introduces the population, because it is the only shape in which the real middleware chain, the real EF query filters and the real RLS policies run against one another in one request, connected as learnstack_app. Packet 7 originally drives test-only controllers because it ships no production endpoint. P02d-4 adds actual public Education GET/HEAD cases in PublicEducationHttpTests, including mapped-host scope and uniform hidden resources. Test-only probes remain separate from the production endpoint set. These live under Database/ with the rest of the Docker-bound suite and carry the same trait.
  • Identity-provider tests — P02b-6's TenancyIdentity/KeycloakFixture imports the committed realms into Keycloak 26.6.2 with its own PostgreSQL database. SharedKeycloakRealm shares a baseline provider, not an immutable or read-only store: real login changes provider sessions, and the actual reconciler runs against it with independently checked zero-write expectations. Cases in that collection serialize. Warm/mutated realm variants own private providers; tests changing shared owner state restore it before returning. The provider uses owned HTTPS with a generated fixture CA, explicit custom-root trust and normal hostname verification; no host trust-store mutation or certificate-validation bypass is allowed. Code/S256 login and signed tokens are real; capturing the registered HTTPS callback without following it does not prove a browser/BFF session. Application isolation still requires the separate learnstack_app fixture. Provider/admin credentials and tokens never enter failure output. These cases carry the Docker trait and enter ordinary shards. TenantRealmApiTestGroup separately combines the provider with the actual PublicReadFixture and preserves its physical read-only/application-role observations. It is nonparallel within its process; do not add that database fixture to the provider-only collection. TenantRealmProfileProtectionTests owns a class fixture rather than changing the shared baseline's profile. Offline realm tooling controls run in meta with fresh named-case reports and zero skips/todos.

Where a Docker-bound test lives, and what it takes, is what routes it. Every_Database_Test_Carries_The_Docker_Trait scans the whole of LearnStack.Tests.Integration and requires the trait on every test class under Database/ and on every one elsewhere that takes a container fixture — SchemaFixture, PostgresFixture, PublicReadFixture, KeycloakFixture, or the SharedSchema / SharedKeycloakRealm / TenantRealmApiTestGroup collections, as ApiHandlerCompositionTests does. It scanned Database/ alone until Packet 9's fifth review, which left a class beside the HTTP suites unchecked. A missing trait does not fail the class, it runs it in the backend job: the hosted runners also carry a Docker socket, so the container starts and the test passes — nothing goes red, and the Docker suite stops being where the Docker tests live. Its companion, The_Docker_Trait_Sweep_Can_Actually_Fail, feeds the predicate the shapes it must and must not flag. Both backend aggregates reject an empty or incomplete assigned population. All Docker executions and their aggregate additionally require their exact discovered case identities, complete results and zero skips: a filter matching nothing exits 0 without those guards.

Host tests use real application module configuration. Data and request-level isolation tests use real PostgreSQL plus application module configuration. Identity-provider tests use real Keycloak with the committed realm configuration; combined provider/API cases also use real application module configuration. All four use no mocked repositories and cover happy paths and edge cases.

Under ADR-0062, scripts/ci-native-tests.py owns the complementary Docker populations: Requires=Docker&NativeCrashProof=true and Requires=Docker&NativeCrashProof!=true. The existing source-trait rule also restricts the shared native trait to the two complete native proof classes and rejects an unclassified proof caller. Actual VSTest discovery and final TRX IDs prove the disjoint complete population; display names alone can collide across theory rows. Private discovery diagnostics are not exported. Incremental execution events retain actual assembly/case IDs, outcome and duration even when interruption prevents final TRX. Each selected project uses its own private diagnostic stream with the pinned VSTest BatchSize=1 setting; empty project populations remain covered by discovery. Final results reconcile those records with TRX, including all summary counters. A display-name-only progress record cannot distinguish colliding theory rows.

For the six enumerated live Keycloak test classes, failed execution measurements retain the fixture's closed startup stage, reason and cleanup result, plus the startup cancellation state when supplied. Stages identify the last reached SDK milestone, not the exact failing operation. The cancellation state is captured when failure is caught, before cleanup; it does not identify the provider cause. Older diagnostic sentences leave that state unknown. Unknown, ambiguous or oversized messages supply no diagnostic fields; raw messages, exception types and provider output remain private. These fields neither establish quiescence nor replace Docker absence, complete results or a successful execution receipt. Completed-case streams and retained TRX contain no raw startup failure text.

Native alone uses the exclusive disposable Ubuntu 24.04 VM and closed scripts/ci-native-provisioning.py owner. Setup verifies a supported collector or writes only literal core; it never restores a pipe. The unchanged child fixture still qualifies its own preconditions and actual native exits. The required aggregate checks every producer outcome and matching complete artifacts. Offline provisioning/launcher controls run in meta. The CI acceptance and implementation record owns actual review and hosted qualification; local controls do not establish VM ownership, no-dump proof or either execution's whole-job budget fit.

Complete bounded CI assignments

ADR-0063 extends that execution owner to base, Architecture and ordinary selector-group shards. The tracked plan is the count/weight source of truth; the finite workflow matrix must agree before launch. Actual Release discovery supplies assembly, case ID and FullyQualifiedName. Selected filters are independently rediscovered before execution; all theory rows stay with their selector. Advisory weights never define membership or a wall bound.

scripts/ci-test-shards.py and scripts/ci-test-shard-results.py implement the two always-running aggregates under the unchanged backend/integration required names. Each verifies current-attempt job outcomes through the read-only Actions API, downloads immutable artifact IDs, verifies archive hashes and recomputes assignment and complete exact case/TRX unions. Both scopes compare their full manifest/toolchain to the same uniquely named, immutable current-attempt base artifact. Integration additionally requires base as this common witness; it does not count base's non-Docker cases in its Docker union. Base also requires the actual Release build, format and Hangfire distribution/source/replacement gates. Matching worker digests alone cannot replace API or result evidence.

Base/Architecture use only the approved non-root child and owned systemd/cgroup boundary on exclusive Ubuntu 24.04 VMs: no Docker census/disposal, collector access or crash inventory. Ordinary retains Docker ownership; native alone retains collector provisioning and independent crash assertions. The existing positive-VM predicate recognizes GitHub's documented standard AWS/Azure pool as clarified by ADR-0062 Amendment 4; container/identity/topology and authority checks remain mandatory. ADR-0094 gives verified Architecture and ordinary Docker producers a forty-five-minute whole-job ceiling (ADR-0082 set twenty-five and thirty). Base/native remain twenty minutes, both aggregates five and meta five. ADR-0085 selects only the public-demo exception: a 22-minute job and 19-minute first-step work deadline, with its separate 180-second nominal reserve and internal 720-second proof unchanged. All other jobs retain their existing ceilings. The backend 203-second reserve and all test-specific deadlines stay fixed. The closed (mode, expected job, plan counts) policy selects API, receipt and enclosing workload/project observation bounds: 2,700 Architecture and ordinary, 1,200 base/native seconds. Receipt partitions and global limit expansion confer no authority. Individual observations allow the inclusive ceiling; disjoint receipt intervals sum strictly below it. Nested observations are not additional disjoint time. Offline plan/result controls retain their shared thirty-second meta allocation (27-second TERM plus three-second kill); extensions must be re-measured against that unchanged allocation. Actual SDK/console/loaded-adapter hashes, disjoint phase intervals, first-result gaps and scoped resource observations accompany every successful execution. Uninstrumented fixture/container measurements remain unknown.

Native inventory allocations — derives from ADR-0091. The verified native owner selects the initial phase before owned-test launch; cleanup selects the final default. Exact private phase strings are preflight and final; unknown shapes refuse. No environment/receipt/CLI selects a duration.

Inventory owner Allocation / sublimit
Native preflight file/image inventory 60 s
Native final file/image inventory 20 s total
image_layer_snapshot read-only commands, both phases and all rechecks 10 s each
Other commands, including the file-format probe existing bounds; default 5 s

Each phase keeps one absolute shared deadline clipped by the execution/cleanup owner; every command is also clipped by its sublimit. Final metadata uses ten seconds per command within twenty seconds total. Neither sums of inner maxima nor actual retirement are reserved. Preflight consumes execution allowance, with SCAN_SECONDS=20 still in the unchanged 203-second reserve ledger. Positive current image provenance, exact proved-directory pruning, complete rescan and fresh consistency checks remain mandatory. No filename/format, partial query or unreadable/changed/timed-out evidence supplies a clean inventory. Actual final fit remains the phase's source-bound hosted qualification obligation.

Use Re-run all jobs, not failed-job or single-job reruns. All producer siblings must belong to the current run attempt. Before rerunning, retain the completed attempt's original producer packages, aggregate receipts and workflow metadata: a rerun can replace its Actions artifacts. Recovered job-log rows are labelled recovery evidence, never original artifact packages. Odd/even attempts select the two deterministic layouts; final qualification requires genuinely distinct co-membership, including SharedSchema methods, on the same source, plan, toolchain and complete case population. Plan, algorithm or fixture-lifetime changes require both layouts. Complete this qualification before merging. GitHub can omit a merged same-repository PR from a run's REST projection; the aggregate refuses that missing identity even after a complete rerun. Further PR qualification uses an open follow-up PR, without weakening identity checks or combining receipts from different attempts. Normal future pushes execute one complete assignment. The CI follow-up record owns actual hosted fit and the two complete attempts; local controls do not establish either.

An isolation test connects as learnstack_app. One that runs as learnstack_migration, learnstack_platform or learnstack_outbox_admin passes even when every policy is inert, and therefore proves nothing.

Architecture Tests

Non-skippable: no architecture test carries Skip, and the suite runs in the required backend job on every PR. The rules — canonical names, what each asserts, and whether it runs yet — are the architecture-test catalogue. This section keeps no second list: the table that stood here named checks, such as a Hangfire payload scan and a provider-SDK reflection rule, that are registered for later phases and were read as running.

API Contract Tests

  • Validate every endpoint against the published OpenAPI.
  • Breaking changes require a version bump.
  • P02d-4 Step 4 delivered: the contract suite compares the production composition's served document with backend/openapi/v1.json, preserving array order. Eight GET/HEAD operations, exact parameters/statuses and recursive DTO allowlists have clean and planted offending controls. The required frontend job regenerates the locked SDK and refuses drift; fake transports exercise all four generated GET wrappers. Pinned CLI fixtures cover ADR-0024 compatibility, with explicit companions for the tool's open-enum gaps. Snapshot admission controls fail closed on missing/deleted snapshots and failed base verification. The packet record records executed counts, reviews and the live required-check rollout.

End-to-End Tests

P02d-7 Step 3 implements the narrow seeded catalog → course → lesson browser proof over the running production API/native renderer. Its delivery record records the passing local nine-page walkthrough and 27 Lighthouse audits; both independent Step 3 reviews are complete. The packet's Step 4 record owns real CI receipts and required-check activation; local proof alone does not establish activation. The proof uses Playwright's library and managed Chromium, without a separate Playwright Test project. It supplements G33's human walkthrough and preserves P6's recorded manual smoke; it does not replace either with automated accessibility scores.

Phase 06 owns the broader MVP vertical slice (tenant → page → course → enrollment → live session), approximately ten high-signal golden flows and the full Playwright/axe harness. P02d-5/6's real-API HTML/RSC integration cases remain separate from browser evidence. P02d-7's proof adds actual linked navigation, computed themes, ordered fields, TLS refusal controls and bounded audits over unchanged seeded product routes.

Run the shared proof from a fresh checkout without root .env, web .env.local or artifacts/public-demo, with Docker and the documented toolchain available:

(cd frontend && pnpm install --frozen-lockfile)
(cd frontend && pnpm exec playwright install --with-deps chromium)
make demo DEMO_MODE=ci

The lifecycle provisions a unique tmpfs-backed Compose project, generates private disposable TLS/configuration, seeds through normal commands and stops its Keycloak before browser navigation. Fixed application/Compose ports must be free. Review the sanitized artifacts/public-demo/receipt.json, all 27 JSON/HTML report pairs and browser screenshots; proof and owned cleanup must both pass. Existing artifacts must be moved aside before a repeat run. Private environments, certificates, keys, browser profiles and raw process logs are excluded from artifacts. Human foreground make demo retains manual hosts/CA/leaf setup and persistent local data. The local test workflow owns the complete setup and artifact instructions.

The shared CI launcher observes a clean Git commit and tree before its own dependency restoration or builds, verifies the owned repository root, and rechecks that identity on entry to the browser proof and before success. Workflow browser tool bootstrap precedes that launcher checkpoint. Inherited Git overrides do not select another repository. Tracked, staged or nonignored untracked changes refuse qualification; ignored runtime/build outputs remain permitted. These clean checkpoints are source observations, not hermetic build attestation: ignored configuration/dependencies and an edit restored between observations are not certified. Foreground local development retains its existing dirty-tree workflow.

Each report receipt binds the JSON and HTML byte digests to the same invocation, commit/tree, build, URL/run, configuration and browser identity. A missing, corrupt or substituted member refuses the pair. Closed operation diagnostics distinguish actual child exit, spawn, deadline, cancellation and daemon failure; cleanup never borrows a signal sent intentionally during teardown. Raw provider messages and private configuration do not enter these receipt diagnostics.

The frontend workspace pins Playwright and Lighthouse as development tools. Lighthouse's transitive development axe-core@4.14.0 does not supply a standalone application axe dependency or the full Phase 06 axe suite. There is no pnpm test:e2e, pnpm test:a11y or @playwright/test project.

Public renderer fixture ownership

P02d-5 remediation Step 3 implements test-owned process/tree cleanup, positive child environments, explicit TLS verification and sticky bounded private-output scanning. Setup and termination share one owner; SIGINT/SIGTERM, explicit control closure and EPIPE must stop owned children before removing private files. Fixed port collisions fail; a healthy response alone cannot establish owned readiness. Cleanup joins final child output before its containment verdict. An unresolved process/listener cleanup retains the private tree and fails; it cannot claim confirmed teardown. Envelope boundaries at chunk ends require a delimiter or EOF.

Controls must fail for omitted cleanup, early split leaks followed by large output, disabled trust verification, stale listeners and unclosed upgrades. Healthy builds receive nonempty private configuration; a real public-only Client Component and a successfully built leaking mutant make emitted-asset containment falsifiable. The remediation record owns execution/review evidence. These are socket/build controls, not P7 browser or Lighthouse delivery.

Frontend Test Types

Type Tool
Unit (logic) Vitest
Component Testing Library + Vitest
Server actions Vitest + Next.js test helpers
E2E P02d-7: Playwright library; Phase 06: broader Playwright Test harness
Visual regression Chromatic / Playwright snapshot
Accessibility P02d-7: Lighthouse automated score; Phase 06: full axe-core via Playwright suite

Rules:

  • Components with non-trivial state have unit tests.
  • Snapshot tests only for stable visual primitives.
  • Playwright covers the same golden flows the backend E2E covers, from the user's perspective.
  • Visual regression covers the public renderer and page-builder block output.

Protected page and form proofs follow Frontend Architecture § Native Entry and Administration. Keep the following evidence distinct:

  • Pure/component controls cover all seven selected views, current capability and exact-target scope, string versions, independent form keys, complete-array Role/grant buttons, kernel restrictions and release adoption. Missing exact own profile and disabled preferences use the declared fallback; a viewed member's locale never changes the editor's UI. Stale/key-reuse recovery preserves bounded original intent without a resubmit form or visible internal IDs. Unknown-outcome recovery preserves the exact original retry body/key. Escaping and private-field absence are assertions, not snapshot-review assumptions.
  • Actual native/Next HTML and Flight controls assert the finite page-read set, no-store behavior and nonempty authorized documents across hosts/locales. Exercise a passive GET separately from HEAD and captured prefetch: the former still prepares page data without idle/refresh activity; the latter produce no private page snapshot. Null passive reads emit no private page/title, not a fabricated fallback. A passing blank speculative render cannot stand in for a positive full-document proof.
  • Lifetime controls reserve actual snapshot credits and hold provider/API task promises, the handler, registered waitUntil work and native write callbacks. Exercise disconnect, timeout, expiry, logout/scope/revision change, concurrent hosts, overflow and shutdown. Credits remain held until actual registered cleanup settles, including nested registrations. Assert refusal before headers and connection closure after headers without invented backpressure or completed callbacks. Native entry and mutation/account-change recovery documents use the same output fence; test late recovery output as well as successful page output. Real HTTP ServerResponse controls destroy its socket before close and attempt immediate write/end: no new callback credit may strand. An earlier issued callback held separately must still retain the frame/snapshot until released. A fake transport that always invokes remembered callbacks cannot refute this Node behavior.
  • The source fence refutes added application awaits, schedulers, API/IO aliases, transitive imports and framework background hooks in the selected RSC closure. It permits only the passive page/provenance seam. This is bounded source analysis; the lifetime owner joins registered work and does not claim to discover arbitrary detached asynchronous work. Keep planted dirty twins alongside each exception.
  • DOM-local dirty-form controls assert a visible localized notice and a cancellable navigation warning only while an editable form is dirty. Submitting one edited form must not erase another form's warning; sign-out, cancelled submission, expiry, replacement and unmount have separate controls. No field-value cache or automatic submission is introduced by the warning mechanism.

The native HTTPS stock-browser proof uses separate host cookie jars and the real five new page routes, native forms and session/history controls with fake grants and API responses. Each page requires its exact ready view and visible specific content; the shared protected wrapper and h1 are insufficient. An unavailable-view counter retains that wrapper/heading and must fail the same positive guard. Its JavaScript-disabled context visits both hosts/locales, observes visible profile controls, submits the native form, follows its fresh GET and signs out. Assert the visible no-script notice by a DOM locator and exact text; a text-selector engine that excludes noscript is not evidence that the browser hid that notice. This fixture proves browser behavior, not the real provider/API chain or manual assistive-technology conformance. An unavailable stored profile preference has no preselected replacement: test name-only invalidity and absent locale form data, deliberate empty/default or enabled choice, and preservation when that language becomes available again. The no-script journey also exercises the required choice through the native form; its closed receipt distinguishes this witness from ordinary enabled/default input.

Keep natural Back navigation and the explicit persisted-event counter separate. A stock browser can perform fresh network navigation with zero natural BFCache restores. Do not report the synthetic event as a natural restore. Native POST error history must navigate to its checked GET recovery rather than repeat the POST; the GET history dirty twin must still exercise the actual reload branch. No-script success establishes usable forms/recovery, not script-based expiry/history withholding. Record actual browser/version and observed counters with the owning phase evidence. Keyboard/focus, reflow/zoom, contrast and real assistive-technology checks retain their separate evidence requirements below.

P02d-6 G38(b,c)/G43 — Steps 1–4 implemented, 2026-10-10. Vitest covers synchronous views and pure mappings; all 31 active recommended JSX accessibility rules have blocking actual-config dirty/clean controls. Async pages have production HTML/RSC proofs against unchanged product routes in the existing required real API/Next Docker fixture; keep P5's synthetic transport probe in a separate mode so its catch-all cannot shadow product routes. Preserve learnstack_app, RLS, read-only transactions, native HTTPS, closed hop headers and fixture containment. Step 4 adds deterministic overlapping host/locale lanes, authored RTL content, whole-theme fallback, next-document freshness and one-shot content-failure modes. The delivery record owns execution, review evidence and the completed manual accessibility smoke.

Assert visible HTML/DOM separately from Flight: status, lang/dir, headings, theme, attribution, both hosts/locales, cross-tenant absence, overlapping concurrent host/locale requests, freshness in one process, controlled failures, both opaque pagination surfaces, exact schema swaps and inert authored values. Test-owned inventory exceeds the page bound; test-owned enabled RTL content proves direction and labelled UI fallback without rewriting historical seed inventory. Verify the approved local 307→404 chain, changed status URL, no direct internal bypass and HTTP 200 noindex known content-failure states. Under ADR-0055, completed product documents cost two API calls, fixed public status documents one and followed missing documents three. Metadata/layout/page/UI reuse adds no bootstrap; the next document re-reads. HEAD/RSC/prefetch and Flight fallback need separately proven counts. Protected native snapshots have their own finite read sets and are not covered by the public Education call budget. The Step 3 record records passing production replacement proofs; both independent Step 3 review rounds are complete. These HTTP/HTML/RSC proofs are not browser E2E evidence.

ADR-0055's separate test-only stock-Next navigation probe observes a failed Flight followed by a fresh document request and counts that chain separately. It requires an installed stock Chromium browser, uses an isolated disposable profile with certificate-specific public-key trust, and fails if the browser is absent. The required CI integration job checks /usr/bin/google-chrome and its version. Development HMR controls separately check recompilation and fresh admission in one native process. The Step 3 record owns their actual execution and review status; P7's browser demo and Lighthouse, and Phase 06's full Playwright/axe, remain separate.

Applicable jsx-a11y rules run at error severity with actual-config planted controls. Catalogue checks cover nonempty/equal key sets, valid ICU and argument agreement, checked callsite coverage and bounded formatter failure, with planted offenders. Canonical names/statuses live in the architecture-test catalogue; acceptance does not mark them Implemented. Manual keyboard, focus, 320 CSS px reflow/zoom, long strings, contrast and real screen-reader smoke evidence must name commit, environment, both hosts/locales and assistive technology. If pending, P6 completion is not claimed. Full Playwright Test/axe remains Phase 06; the narrow seeded demo/browser/Lighthouse proof is delivered by P02d-7. G38(a)'s transport proofs and G38(d)'s guarded runner are already delivered.

Version-qualified Next response cleanup

ADR-0083, ADR-0084 and Standard 07 require distinct installation, generated-output and actual-owner evidence. Acceptance registers these obligations; it does not establish their execution.

  • Verify the actual Node and selected pnpm executables at their supported boundaries. Wrong patch/major versions, correct declarations paired with wrong executables, missing or partial patches, changed census bytes and stale maps must refuse. Verify equivalent semantics in every transformed copy and execute actual supported development/production runtime families; an unused copy is no exemption.

  • Prove clean generation and its input/output manifest. Old or changed .next beside a corrected package, changed generated native output, mismatched sources or producer, and unsupported development cache/compile paths refuse before runtime import or listener creation. Copied fixtures consume the same verified correction and generation provenance without broader live-source exemptions.

  • Use real healthy and peer-close HTTP/HTTPS controls with identity/gzip/deflate, HEAD, empty, 204/304, static/direct-end and streamed responses. Include early header flush with unknown length and an empty producer. Observe actual encoding selection, compressor allocation/close, handler settlement, headers and wire bytes separately. Zero body bytes or a bodyless status cannot certify no encoder. Include actual native fence refusal after middleware allocation: preserve the empty no-store 503 and removed encoding headers, observe real finish then close, and join even a never-ended encoder. Final headers or prior finish cannot erase allocation. A premature-close-only owner must fail this retained-resource case. Preserve healthy encoding, bytes and backpressure. The terminal-only dirty control must expose retained compression after its pipe settles.

  • Exercise a real initialized router and actual pipe rejection before headers or encoding selection, abort before selection, synchronous construction failure, early stream fault and rejected end work with an existing stream. Each boundary transitions before joining, seals late allocation and joins every real resource without waiting for its caller's future error response. Primary error identity and separate cleanup faults survive, including combined failure and faults after headers/finish; no unhandled rejection, second response or rollback is permitted. Duplicate, missing or late required owner registration refuses; duplicate installation cannot allocate another compressor.

  • Cover already-terminal attachment, close during blocked write/end, separately held real write/end callbacks, drain and exact listener retirement. Actual finish or close resolves terminal observation; premature close preserves expected-abort classification without certifying delivery. No invented event, callback, timeout or diagnostic disposal supplies positive cleanup.

  • Hold actual end work and independent native handler, registered framework, provider/API and snapshot-pool work through transport/compressor closure. Reservations remain held until their own required work settles. Preserve native publication fences, refusal before headers, cancellation after headers and the existing shutdown failure boundary. Observe actual native development/production browser journeys separately from direct framework controls.

  • Under ADR-0084, prove owner and child startup stop latches before handshake, before shutdown registration, between producer stages and during held real app.prepare(), with SIGTERM, SIGINT and parent disconnect/death. No late start/ready clears stop or permits listen. The one five-second watchdog starts at the first stop, even before registration; neither registration nor a later stop resets it. Expiry fails child and retains parent ownership until actual family retirement; existing ten-second operation/body bounds remain unchanged.

  • Hold a real descendant after normal, failed and SIGKILL direct-child exit. Refuse replacement until actual group absence. Prove initialized development static-paths worker retirement and normal production/development restart. Preserve actual admission/request/session/Next shutdown order. Runtime generic broadcast-preload and group-kill mutants fail. Compiler/installer cancellation still fails, including existing SDK cancellation controls and child-zero stop.

  • Check the distinct runtime result matrix: only actual listener-ready plus requested graceful child-zero/no-signal stop and actual group absence succeeds. Pre-ready/unsolicited zero, signal exits, faults, forged ready without binding and parent loss fail. Handshake/IPC/log output never substitutes for bind plus real HTTP/TLS checks. No new startup timeout is introduced.

  • Verify exact parent records and independent child graph/cache metadata without child lease acquisition or cache preparation. Challenge/record/incarnation confusion, wrong parent/mode, stale record, missing channel and direct child entry refuse. Check owner builtins import closure and all boundary input mutations, including copied literal child instrumentation before genuine build.

  • Capture immutable positive owner/runtime/producer family handles before proof readiness. Owner-group absence alone does not release runtime roots. Exercise parent loss, missing/stale handles, changed records, observation uncertainty, preserved failure and positive all-family fixture reconciliation. No cached group signal, forced cleanup, timer or receipt replaces actual retirement.

  • Migrate ingress/BFF bind, preparation, version-fetch, observe, advance, render-control, sanitized native/sink upgrade, proof-listener and bounded proof-shutdown schemas through the copied finite relay. Deliver each schema to its existing host and retain forged/malformed/omitted/lost-frame, wrong-sender, never-binds and runtime-replacement negatives. HMR/freshness/publication/session proofs compare actual child incarnation, not only unchanged outer PID.

  • Deliver the public-admission copied profile's seven fixed modes, observe and settled requests, exact correlated count/immutability replies and one actual admission-shutdown count frame through its own closed relay. settled awaits the real native completion observer. Test malformed/wrong-profile/replayed values and ensure a fabricated reply cannot satisfy actual completion. Preserve wire/context/late/HEAD/abort/active-shutdown/browser/HMR witnesses; no BFF count clamp or new production capacity bound is inferred for admission counters.

Frontend type/lint/full guarded tests and public browser/Lighthouse proofs remain required. Complete final-source hosted A followed by full B uses the same source, toolchain, plan and discovered population: all 27 successful jobs, 20 natural VM owners, 25 original ZIPs and exact zero-skip unions. The existing CI windows, 203-second reserve and per-test deadlines remain unchanged. Retain and audit A before B; local controls, failed subsets and diagnostic disposal replace neither. Manual accessibility evidence retains its separate maintainer-owned disposition in Phase 03.

Tenant Isolation Tests

Mandatory for every PR touching tenant-owned data:

Use the shipped examples:

  • PublicEducationHttpTests resolves actual mapped hosts and asserts exact-locale public catalog/detail behavior, foreign/draft/restricted hiding and sibling-organization refusal. Course_draft_branch_foreign_and_unknown_are_uniformly_hidden and Organization_host_never_serves_a_sibling_organization exercise the production public GET/HEAD contract.
  • EducationIsolationTests independently tests raw app-role RLS and EF-filter behavior on all four Education tables, including unresolved scope, insert scope, reporting reads versus writes, immutable organizations and parent constraints.

Fixtures apply every chain through MigrationChains, which owns current chain/history discovery. SchemaTableClasses independently checks the complete applied-table census; tests must not copy a fixed partial chain list. Follow add-integration-test for fixture selection and transaction-scoped context.

Rules:

  • Every module owning tenant-scoped data has at least one such test.
  • Failures block PRs.
  • Regression tests are kept indefinitely.

Test Data

  • Builders / factories for entity construction.
  • No global mutable fixtures.
  • Seed only what a test needs.
  • Factories live beside the suite that uses them — SchemaFixture and the seeder's composition for the integration suite. There is no shared test-kit project; one earns its place when two suites need the same builder.

Coverage Targets

  • Domain code: ≥ 90% line, ≥ 80% branch.
  • Application code: ≥ 80% line.
  • Infrastructure adapters: ≥ 70% line.
  • UI components: behavior coverage, not lines.

No CI job collects coverage today, so these numbers are a local target and gate nothing. The architecture, isolation and contract suites are the hard gates. Collection lands with the release pipeline in Phase 11; until it does, a statement that coverage "blocks" anything is false.

Test Speed

  • Unit suite < 30 s locally.
  • Single-module integration suite < 2 min locally.
  • Full CI < 15 min.

If a test gets slow, fix the test before the suite.

Flaky Tests

  • A flaky test is a bug. Triage immediately.
  • Quarantined tests re-enabled within one sprint or deleted.
  • A Skip fails CI: the ci-test-shards.py / ci-native-tests.py execution owners and ci-test-shard-results.py aggregates require complete discovered case-ID unions with every outcome Passed (No_Architecture_Test_Is_Skippable). P02d-5 G38(d) extends the runner to frontend workspaces: pnpm test from frontend/ discovers test packages and checks actual nonempty Vitest JSON outcomes, rejecting failures, skipped/todo cases, missing scripts/reports and omitted discovered files. Symlinked workspace packages and test sources fail explicitly rather than disappearing from discovery. Real clean/planted runner controls prove refusal.

Frontend guarded execution

Derives from: ADR-0092, ADR-0090, ADR-0089, ADR-0088, ADR-0087, ADR-0086 evidence erratum.

The frontend guard reconciles discovered test-file paths with reported files. It verifies nonempty, internally consistent Vitest counters and all reported cases Passed, with zero skipped/todo outcomes. It does not independently enumerate expected scenario IDs or detect a case conditionally never registered inside a reported file. Native registration/callback source controls provide separate narrower checks. Backend case-ID discovery/reconciliation is unchanged.

Test owner Timer / observation setting
Canonical apps/web aggregate, two file workers 900 s
Canonical packages/sdk aggregate 360 s
Other package aggregates 180 s
SDK cold typecheck/lint stop-request timer 150 s
Its asynchronous Vitest failure threshold 160 s
Selected ownership matrix outer and private group timers 180 s each, distinct starts
runFrameworkOwnershipGroupCore subprocess 210 s
Startup/workers/recovery Vitest cases 220 s
Per-runtime event observation schema inclusive 0–120,000 ms from that process's start
Closed 31-control router Node subprocess signal threshold 20 s; five siblings retain 10 s
Its combined synchronous Vitest failure threshold 90 s
Provenance 112-case Node subprocess signal threshold 90 s
Its synchronous Vitest failure threshold 100 s
Two sequential pnpm discovery probe signal thresholds 15 s each
Frontend CI job cancellation allowance 30 min

execFileSync/spawnSync timeout settings request a signal and may keep waiting for child exit; they are not synchronous return-time maxima. Vitest cannot preempt a blocked synchronous callback. Its setting is a failure threshold; 20 + 5 × 10 + 20 explains the combined allowance without guaranteeing a reserved interval or completion. The separate asynchronous package runner and workflow job retain outer expiry/cancellation ownership, without granting physical-cleanup credit. No hard inner watchdog is implied.

The SDK cold-prefix stop request begins after private preparation; Vitest counts the whole case. Two stop settings total 300 seconds and two case thresholds total 320 before preparation, joins and other work. The SDK aggregate of 360 seconds reserves neither sum nor a forty-second cleanup interval. With today's two tested packages, 900 + 360 = 1,260 leaves 540 nominal seconds inside the 1,800-second frontend job for setup/build/other proofs; neither sum guarantees fit. The uncached guard must accept; the cache-enabled counterfactual must refuse output_unowned, with the same source/ownership/cleanup assertions (direct control owner).

All unlisted settings remain, including other Node controls, initialized-router proofs, separate initial/router 120-second drivers, provider startup and runtime HTTP/readiness/shutdown. Three 180-second group settings sum to 540 before router/source controls and wrapper work; 900 seconds is not a reservation of every nested maximum or a hosted-fit claim. Canonical paths select the package setting, not manifest names/environment.

Package expiry invokes its existing owned-process-group SIGKILL and fails. Neither this invocation nor its child-process-double control proves retirement of independently detached native families, reservation release or root deletion. Actual native family handles/receipts/absence checks remain separate; uncertain physical cleanup receives no credit. This is not a claim that every timeout leaks. These aggregate native-proof runs are distinct from the pure-unit speed target above.

Live Classroom Testing

  • Token issuance — correct identity, role, TTL, grants.
  • Authorization — only authorized users can join.
  • Provider abstraction contract — fake provider validates the interface.
  • Webhook handling — signature verification, idempotency, tenant scope.
  • Attendance computation — synthetic event streams.
  • Recording policy — consent flow asserted; recording does not start without consent.

Real LiveKit is not required in CI; a fake provider satisfies the contract.

Background Jobs

  • Hangfire jobs are unit-tested by invoking the job class directly.
  • Tenant context propagation asserted in integration test for every job.
  • Idempotency: same job invoked twice produces the same effect.

CI Pipeline

  1. Restore + build, warnings as errors.
  2. Static analysis — the Roslyn analyzers run in the build; ESLint and TypeScript in the frontend job.
  3. Unit, architecture and contract tests, and the Docker-free integration tests — the base/Architecture executions behind the required backend aggregate.
  4. Complete ordinary and native Docker executions — required together by the backend-integration aggregate, with its existing check name unchanged.
  5. Frontend unit and component tests — the frontend job.
  6. Narrow seeded browser/Lighthouse proof — the public demo + lighthouse job invokes the shared make demo DEMO_MODE=ci lifecycle, per § End-to-End Tests. It retains real admission budgets and never retries a 429 or audits a refusal page. The packet's Step 4 record owns the real PR receipt and seventh required check activation. Phase 06 owns the broader Playwright Test/axe golden flows; any infrastructure-flake retry there requires logged justification.
  7. Coverage report — none yet, per § Coverage Targets.

A failure in any step that runs fails the pull request. Which checks block a merge is a repository setting, and CONTRIBUTING § Branch protection lists them.

Forbidden

  • Tests depending on real external services (Stripe, LiveKit Cloud, Postmark) without sandbox + recorded interactions.
  • Tests sharing mutable state.
  • Tests depending on execution order.
  • Tests sleeping > 1 s without a polling assertion.
  • Tests asserting on log output strings.
  • Tests catching and ignoring exceptions.

Test Names Read Like Specs

Good:

  • PublishCourse_WhenDraftHasNoLessons_ReturnsValidationError
  • JoinSession_WhenLearnerNotEnrolled_Returns403

Bad:

  • Test1, ShouldWork, CourseTests_Publish