Skip to content

Trim the test suite, close coverage gaps, and parallelize CI - #834

Merged
TheGreatAxios merged 18 commits into
mainfrom
cl-7514-trim-content-pin-and-cosmetic-tests-from-the-suite
Sep 10, 2026
Merged

Trim the test suite, close coverage gaps, and parallelize CI#834
TheGreatAxios merged 18 commits into
mainfrom
cl-7514-trim-content-pin-and-cosmetic-tests-from-the-suite

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Summary

  • Tests pin only behavior: content, brand, and animation-frame pins plus duplicate coverage are gone, and the freed budget funds coverage on load-bearing paths. The suite drops from 6,263 to 6,252 cases while adding tests for the auto shell policy wrapper bypasses, torn session journal recovery, exec close cancellation, session queue ordering, and MCP reconnect schema drift.
  • The three flaky subagent mount-gate tests (plus a structurally identical fourth) are fixed at the root: they slept through a hardcoded retry backoff against a refused connection; a local 401 server fails the send on attempt one, dropping each probe from ~1.8s to ~0.3s with zero fixed sleeps.
  • CI runs one static-analysis job (format, lint, typecheck), a decoupled build job, and a two-shard test matrix that still goes through the projects-dir guard with the same seed. Critical path drops from ~118s to ~95-100s.
  • A lint rule (corbits/no-content-pin-tests) with its own behavioral tests keeps content-pin tests from regrowing; AGENTS.md states the bar: a test earns its place only if a real behavior change can fail it.
  • The check gate now pins the shard path union and the seeded shard script, so sharding can never silently drop part of the suite.

Verification

  • bun run check passes: lint (including the new rule), typecheck, build, and the guarded suite at 6,252 pass / 0 fail (seed 424242).
  • Baseline from recent runs on main: 114-123s total, critical path 111-118s. Restructured: shards of ~78s and ~25s, critical path ~95-100s.
  • Flake note: a pre-existing family of load-dependent 5s timeouts in git/filesystem and TUI I/O tests rotates run-to-run on a loaded machine; every flagged test passes in isolation. Not introduced by this branch.
  • CI job names change (prettier, eslint, typecheck, build-and-test become static-analysis, build, test); main has no branch protection configured, so no required-check migration is needed.

Fixes CL-7514
Fixes CL-7515
Fixes CL-7513
Fixes CL-7516

@linear-code

linear-code Bot commented Sep 8, 2026

Copy link
Copy Markdown

CL-7514

CL-7515

CL-7513

CL-7516

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greybeard · Comment

The bar is sound (tests pin behavior, not copy); the lint heuristic and the trim both over-own it.

Findings

  • scripts/eval-capability.test.ts / scripts/eval-public-swe-one.test.ts — unique CLI coverage (parseArgs, mapPool, effort validation) deleted because those files were outside bun run test path list. Suite membership owns "this test runs," not deletion. Enroll them under ./src, ./tests, or ./evals (or add ./scripts to the suite). Do not delete the contracts.
  • tests/unit/tui/tool-formatter-web-brand.test.ts (deleted) / src/tui/tool-formatter.ts:54setActiveWebProviderBrand into humanizeToolName (branded web tool names; non-web tools unchanged) is product behavior. No remaining test covers it. Restore.
  • scripts/eslint-rules/no-content-pin-tests.ts / AGENTS.md:38 — error-level shape match is the wrong owner of a semantic bar. It cannot tell copy from plugin identity, frontmatter flags, or fork-boundary policy; its own tests treat manifest.id === "corbits-skills" as a wording pin (tests/unit/eslint-rules/no-content-pin-tests.test.ts:32). Evidence it already distorts: tests/unit/corbits-skills-catalog.test.ts was rewritten into violation collectors to evade expect(skill).not.toContain(...). AGENTS.md already owns the bar. Drop the rule, or keep it as a documented heuristic that is not the definition of "behavioral."
  • AGENTS.md:52 — still says bun run test is "the same suite CI runs." CI now shards via test:paths (package.json:36, .github/workflows/ci.yml test matrix). Path-union (tests/unit/check-gate.test.ts:47) is not the same isolation domain: mock.module leaks across ./src vs ./tests no longer fail in CI (CL-6967). Tell the truth: local bun run test remains the one-process suite; CI shards for wall clock.
  • package.json:36test:paths with zero args is bun test over the whole tree, including vendor/. The vendor-exclusion constraint lives only on the test script. Fail closed (require paths, or alias to test).

Notes

Palette hex/ANSI pins, splash copy, and animation-frame pins were correctly dropped. Duplicate faremeter / inference-abort / consumeStream copies were real duplicates. The 401-not-sleep mount-gate fix owns flake at the retry classifier. New coverage (git-global-config wrappers, torn base JSONL, exec close, session-queue race, MCP schema drift) is the right spend of the freed budget. YAML paths: completeness is fail-closed today and brittle the moment another paths: key appears.

@TheGreatAxios
TheGreatAxios force-pushed the cl-7514-trim-content-pin-and-cosmetic-tests-from-the-suite branch from dd1c5a4 to d42187d Compare September 9, 2026 23:35
The suite pinned packaged skill prose, palette hexes, ANSI-256
indexes, display-name branding, and animation frames, and carried
duplicated copies of faremeter, stream-consumer, and inference-abort
coverage. Those broke on every wording or palette edit without
guarding behavior, and two eval test files were never run at all.
Manifest, inventory, frontmatter-flag, and behavioral invariants
stay.
sh -c, xargs, transparent env prefixes, and quoted program/flag spellings previously had no coverage for the git-global-config rule; only the plain forms were pinned. Assert they all still ask, and that scoped read-only git config forms stay unflagged.
Existing torn-tail tests only tear a tail segment behind a clean turns.jsonl. A session small enough to never roll over tears the base the isogit store parses first, which takes the resilient recovery path instead. Pin that the base-only tear recovers committed turns, extra segments survive it, and the next write heals the file.
Pin that disposeExecRuntime cancels all live workers with the close reason before agent teardown starts, and that a rejected agent close neither blocks toolset disposal nor rejects the teardown.
Cover the enqueue-during-await race: a rotation queued while a delivery is still awaiting its send must wait for the delivery to settle, and a failed delivery must not block the rotation behind it.
The connect mock served one static tool payload, so reconnect tests could not observe a server whose tools changed between generations. Parameterize the payload and pin that reconnect mounts exactly the drifted set: renamed schema on the same tool, a new tool, a single definition per name, the stale client closed, and the drift announced through onToolsChanged.
The mount-gate probes await a full runSubAgent cycle whose inference send to an unreachable host classifies as retryable, so each test slept through the client's full backoff schedule (three attempts, 500ms plus 1000ms fixed) before asserting mount counts. Under suite load that crossed the 5s per-test timeout. Point the send at a local server answering 401, which classifies credential_failure and never retries, and give the four full-runtime probes a 15s timeout to absorb construction-time load spikes.
Tests that pin an asset's literal wording or an exact palette value fail
on copy and design edits while catching no behavior regression, and an
audit found most low-value tests shared that shape. A local eslint rule
now rejects the known shapes in test files; it is a heuristic shape
match, calibrated to stay silent on behavior-string assertions like
command parsing, approval display, and permission tokenization.
Unique CLI and web-brand contracts were deleted because they sat
outside the suite path list. Membership owns that, not deletion.
AGENTS.md owns the behavioral bar; an error-level shape-match
lint rule does not. test:paths now requires paths so a zero-arg
run cannot scan vendor/.
@TheGreatAxios
TheGreatAxios force-pushed the cl-7514-trim-content-pin-and-cosmetic-tests-from-the-suite branch from 13967db to 4e53901 Compare September 10, 2026 03:14
846 reaps the toolset before cancel/close and fail-closes a rejected agent.close. The coverage added here now asserts that order.
protect-main still requires prettier, eslint, typecheck, and build-and-test. Alias jobs go green only after static-analysis / build / the test shards succeed.
@TheGreatAxios
TheGreatAxios merged commit 5968d46 into main Sep 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant