Skip to content

fix(ci): the round tests had been skipping in CI since they were written - #477

Open
gHashTag wants to merge 7 commits into
feat/queen-supervisorfrom
fix/queen-round-tests-run-in-ci
Open

gHashTag wants to merge 7 commits into
feat/queen-supervisorfrom
fix/queen-round-tests-run-in-ci

Conversation

@gHashTag

@gHashTag gHashTag commented Sep 6, 2026

Copy link
Copy Markdown
Owner

The round tests have been skipping in CI since they were written

queen-round.test.ts guards its behaviour cases with it.if(present), where present means "the policy binary exists on this machine". Nothing in the workflow ever built it. From the log of a green run:

(skip) queen round, lease lost > dispatches nothing once the lease has moved
(skip) queen round, lease lost > says so when it stands down mid-round
(skip) queen round, lease lost > dispatches the same issue while the lease is held
(skip) queen round, send-backs counted > returns a first failure for a second pass
(skip) queen round, send-backs counted > names the pass it is actually asking for
(skip) queen round, send-backs counted > escalates at the ceiling instead of returning for ever
(skip) queen round, send-backs counted > increments only on a send-back, in the statement that records it

Seven of them, every run, green.

This is the file that exists because nothing in the repository called runRound — the one whose own header records a critic deleting the watch.held && guard, the stand-down warning, and the heartbeat sweep, one at a time, and watching all 364 tests stay green through every deletion. The lease guard is the one that matters: every write below the choice is unfenced, so a round that has lost its lease and keeps dispatching writes bees onto the legitimate Queen's board.

In CI it has been proving nothing since the day it merged.

Two changes, because either alone leaves the hole open

1. Build the binary — in the same image the Dockerfile uses (swift:6.0-jammy), so the thing under test is the one Railway ships rather than a lookalike compiled by whatever toolchain a runner happens to carry.

The step asserts the artifact, not the exit code:

test -x queen-core/.build/release/queend

A build that prints an error and still exits 0 would otherwise leave the tests skipping exactly as before, under a step that looked like it had worked.

2. Make the skip loud — a build step can be removed, renamed, or start failing quietly a year from now.

The existing first case asserts the path string, which is true whether or not anything is at the end of it, so it cannot notice this. The new one fails when the binary is absent and CI is set. A laptop without it is a fair place to skip — that is what the guard is for — but CI is the only machine whose green anybody reads as coverage. It names the command that fixes it rather than merely refusing.

What this unblocks

The .catch() added in #476, which lets a round dispatch even when review or reaping throws, currently has no test — I said so in that commit. With this merged, runRound can be driven with a pool that fails the review, and the dispatch asserted. That test follows on #476.

Honest note

I claimed in #476 that testing this needed a live database. That was wrongrunRound is exported and takes a pool double, GitHub is stubbed at fetch, and the harness has been there all along. I had not read the file before asserting its limits.

`queen-round.test.ts` guards its behaviour cases with `it.if(present)`, where
`present` means "the policy binary exists on this machine". Nothing in the
workflow ever built it. Read from the log of a green run:

  (skip) queen round, lease lost > dispatches nothing once the lease has moved
  (skip) queen round, lease lost > says so when it stands down mid-round
  (skip) queen round, send-backs counted > escalates at the ceiling instead of
         returning for ever
  (skip) queen round, send-backs counted > increments only on a send-back, in
         the statement that records it

Seven of them, every run, green. This is the file that exists BECAUSE nothing
in the repository called `runRound` - the one whose header records a critic
deleting the `watch.held &&` guard, the stand-down warning and the heartbeat
sweep one at a time and watching 364 tests stay green through every deletion.
In CI it has been proving nothing since the day it was merged.

TWO CHANGES, because either alone leaves the hole open.

Build the binary, in the SAME image the Dockerfile uses, so the thing under
test is the one Railway ships rather than a lookalike compiled by whatever
toolchain a runner happens to carry. The step asserts the ARTIFACT rather than
the exit code: a build that prints an error and still exits 0 would otherwise
leave the tests skipping exactly as before, under a step that looked like it
had worked.

And make the skip loud, because a build step can be removed, renamed or start
failing quietly a year from now. The existing first case asserts the PATH
STRING - true whether or not anything is at the end of it - so it cannot notice
this. The new one fails when the binary is absent AND `CI` is set: a laptop
without it is a fair place to skip, which is what the guard is for, but CI is
the only machine whose green anybody reads as coverage. It names the command
that fixes it rather than merely refusing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the fix label Sep 6, 2026
gHashTag added a commit that referenced this pull request Sep 6, 2026
The `.catch()` in the commit before this one had no test, and I said so - along
with a reason that was wrong. I claimed driving `runRound` needed a live
database. It does not: `runRound` is exported, takes a pool double, and
`queen-round.test.ts` has had a recording fake, a stubbed `fetch` and the real
policy binary since it was written. I asserted the file's limits without
reading it.

So here is the test. `FROM queen_dispatch d` is the review's SELECT and the
only query in the round using that alias, so failing it targets the review and
nothing else. The round is then expected to reach `dispatchBee` anyway and
record its INSERT. Reverting the `.catch()` turns this red: the round stops
before the dispatch and the INSERT never appears.

The shared fake gained a `throwOn` parameter rather than the test monkey-
patching `pool.query`, which needed a cast through `unknown` to compile and
would have left the next reader wondering which of the two queries recorded.
The failing statement is recorded BEFORE it throws, because a query that throws
is still a query the round issued.

HONEST LIMIT, and it is why this is a separate commit: these cases are guarded
by `it.if(present)` and the policy binary is not built in CI, so this test
SKIPS there today along with the other seven. #477 builds it and makes the skip
loud. Until that lands, this is proven on a machine with `queen-core` built and
nowhere else - which is exactly the state #477 exists to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

❌ Tests failed — 1/2089 failed

Suite Passed Failed Skipped
agent 87/87 0 0
build 9/9 0 0
cdp-protocol 5/5 0 0
eval 93/93 0 0
server-agent 272/272 0 0
server-api 975/975 0 0
server-browser 6/6 0 0
server-integration 10/11 0 1
server-lib 273/273 0 0
server-pglive 3/3 0 0
server-root 68/68 0 0
server-skills 31/31 0 0
server-tools 241/242 1 0
shared 14/14 0 0
Failed tests
  • server-toolsinput tools > blocks coordinate-based actions with ACL intent rules

View workflow run

gHashTag and others added 6 commits September 6, 2026 14:13
The build step worked and every test still failed: `queend exited 127: error while loading shared libraries: libswiftCore.so`. Built inside the Swift image, the binary links against a runtime a bare runner does not have.

So it is built with --static-swift-stdlib, and the step now checks that it EXECUTES rather than merely that a file exists - the exact distinction the artifact check was added for, one level further in.

Same sources, same compiler, different linkage from the deployed binary. Said plainly in the step rather than left as an implied equivalence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The static build worked and the step still failed - on my own run check. `queend` answers a JSON question on stdin, so `--help` and a bare `{}` both exit non-zero by design, and a gate demanding a clean exit was testing the question rather than the linkage.

Narrowed to the one code that means what the check is about: 127, the shared-library failure the first attempt hit. Any other code means the process started, which is the whole claim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--static-swift-stdlib made the binary run on a bare runner and quietly changed it: the static toolchain links libFoundationEssentials.a, a different Foundation from the dynamic one, and four daily-cap cases that pass on a dynamic build failed under it.

A gate that changes the thing it measures is measuring itself. So the build is dynamic - the linkage Railway ships - and the runtime is lifted out of the same image and put beside the binary, with LD_LIBRARY_PATH exported for the test step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…elf-diagnosing

Three assertions in queend-choose.test.ts pinned message text the policy has since reworded. They reproduce on macOS with origin sources, so this is test rot, not a platform effect: 'a worker has it' is now 'a worker already has it', and 'expected back' is now 'claimed, but no worker is attached yet'. The behaviour assertions beside them - chosen is null - passed throughout, so the intent held and only the wording moved.

The daily-cap cases are a different animal: they PASS on macOS and fail on Linux, which is the divergence Package.swift warns about in its own header. `expect(answer.allowed).toBe(false)` reports only 'Received: undefined' - the least useful half of the fact when the policy answers from a platform this checkout cannot reproduce. Matching the whole object makes the next CI log say what the binary actually returned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four daily-cap cases passed on a Mac and failed on Linux. The Linux binary said why, once the assertion was widened to print it: {"kind":"error","error":"could not decode the question: ... Expected date string to be ISO8601-formatted."} - it decoded nothing and chose nothing.

`spentTask` built its timestamps with new Date().toISOString(), which always carries milliseconds, and queend decodes with Swift's .iso8601 strategy, which does not accept a fractional second.

The product has been right about this for months. isoSeconds in queen-tick.ts strips the fraction before any task reaches the policy, and its comment records the identical error at codingPath [tasks, Index 67]. The fixture was reproducing a defect that had already been fixed, and nothing noticed because these cases had never run in CI.

A fixture that builds a shape production never emits tests a program that does not exist. This one now builds what boardTask builds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
'ignores spend from another day' overrode updatedAt with a raw toISOString() and put the milliseconds straight back after the shared fixture had been fixed - one case red for the reason the other seven had been.

The valve control pinned 'browseros-ai#1316: a worker has it or is expected back (rejected)'; the policy now says 'it is rejected - claimed, but no worker is attached yet'. Same wording drift as the three in queend-choose, and the behaviour assertion beside it - queend chose nothing - passed throughout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gHashTag

gHashTag commented Sep 6, 2026

Copy link
Copy Markdown
Owner Author

Result: Tests / server-api is green, with the round tests actually running

(pass) queen round, lease lost > has the policy binary wherever green is read as coverage
(pass) queen round, lease lost > dispatches nothing once the lease has moved
(pass) queen round, lease lost > says so when it stands down mid-round
(pass) queen round, lease lost > dispatches the same issue while the lease is held
(pass) queen round, send-backs counted > returns a first failure for a second pass
(pass) queen round, send-backs counted > escalates at the ceiling instead of returning for ever
(pass) queen round, send-backs counted > increments only on a send-back, in the statement that records it
(pass) queen round, boundary checked > names the files a bee committed outside its boundary
...
973 pass

What it took, and what each attempt taught

attempt result
build in the Swift image built fine — every test then failed exit 127: libswiftCore.so
--static-swift-stdlib ran — but links libFoundationEssentials.a, a different Foundation
assert a clean exit from the binary broke the step: queend answers a JSON question, so {} exits non-zero by design
dynamic build + runtime lifted from the image production linkage, binary starts, 8 real failures surfaced

The static fix was the tempting one and it was wrong: a gate that changes the thing it measures is measuring itself. The linkage now matches Railway's, and the runtime travels beside the binary.

The 8 failures were two causes, neither a product defect

Four wording assertions had rotted. The policy now says a worker already has it where the test pinned a worker has it, and it is rejected - claimed, but no worker is attached yet where it pinned expected back. These reproduce on macOS with origin sources, so it is drift, not a platform effect — and the behaviour assertions beside them (chosen is null) passed throughout.

Four fixtures reproduced a bug the product had already fixed. They passed on macOS and failed on Linux. Widening one assertion to print the whole answer got the binary to say why:

{"kind":"error",
 "error":"could not decode the question: ... codingPath: [tasks, Index 0],
          \"Expected date string to be ISO8601-formatted.\""}

spentTask built timestamps with new Date().toISOString(), which always carries milliseconds; queend decodes with Swift's .iso8601, which does not accept a fractional second. The product has been right about this for monthsisoSeconds in queen-tick.ts strips the fraction before any task reaches the policy, and its comment records the identical error at codingPath: ["tasks", "Index 67"].

I checked production before assuming: the registry board carries 2026-08-21T21:32:54Z, no fraction. The deployed Queen is not affected. The fixture was testing a program that does not exist.

Remaining red

  • Tests / server-toolsAssertionError: Expected a point for #submit-btn, a browser-suite assertion. This change touches no browser code; a comment-only control branch passed this job earlier, and a re-run of an identical commit also passed, so it is intermittent. Stated as measured, not diagnosed.
  • cla — the signing bot, on every PR here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant