Repository navigation
Licence cap and false-PASS came back in agent-written acceptance scripts: the toolkit harness is optional, so lessons from #287 don't reach them #567
Description
What happened
In a cook-off, two arms built the same app (ProcureFlow slice 1, Opus). Both lost a test run to the trial-licence session
cap, even though #287 is closed and the toolkit's helpers.js logout() is correct (xas POST, status 200).
- mxcli arm (wired to this toolkit): the acceptance scripts the agent wrote (
acc-t2) did not sign out. The run
stopped at a "Maximum number of sessions exceeded" refusal, and one scenario was retaken (project A-37, METRICS 12:02). - Second arm (another model-writing tool, same process): the agent wrote its own Playwright harness from scratch.
- Its sign-out did not free server sessions: 138 refusals and a ~20 min stall.
- Its
step()recorded a step that returned a defect string as PASS. That gave 5 false-PASS rows, caught by hand.
The bug
The toolkit's harness (helpers.js login/logout, journey-runner.js) is correct but optional. When an agent writes
acceptance scripts, nothing pushes it onto the harness, so the session-cap and false-pass lessons are learned again
in every project.
To research
- Should acceptance or journey work be required to go through
helpers.js(or journey-runner)? For example, a gate
could flag*.jsunder tests/ that callpage.goto+login.htmlwithout requiring helpers.js. - Add to the false-green register (
testing-shape.md): "a step function that returns a defect string, where the
runner only treats a throw orfalseas a failure, so it is recorded PASS". - A recovery recipe for when the cap is hit mid-run:
delete from "system$session"(Postgres, run --local) instead of a
restart. It currently lives only in one project's RESUME.md.
Evidence: cook-off projects, local (A-37 in the mxcli arm; F-19, F-24 in the other arm). No customer data.
Activity
MendixMau commentedon Oct 23, 2026
Triage: DISCUSS track. Item 1 would add a new FINDING or FAIL for hand-written test scripts in stages 5–6, in every entry mode that builds acceptance tests, and in à-la-carte assurance. Items 2 and 3 are docs only. Verified against the code; the cook-off evidence is local and UNVERIFIED.
Problem: The harness is correct.
project-tests/e2e/helpers.js:375-392signs out with an xaslogoutPOST, andtesting-shape.md:221already lists "a logout helper that reports success without ending the session" in the false-green register. Nothing makes an agent use either. An acceptance script written from scratch re-learns the session cap and false-PASS lessons in every project. Two gaps are confirmed:falseas a failure.delete from "system$session", Postgres,run --local) is nowhere in the toolkit.measured-claims.md:128-145diagnoses the cap but gives no recovery.Value: High. Each occurrence costs a stalled run (about 20 minutes and 138 refusals in one arm) or false-PASS rows that someone has to catch by hand. It recurs per project, because nothing carries the lesson.
Options:
testing-shape.md§4 register. That's about 60 words in a baseline file (test, gate and review roles), which must be stated in the PR.measured-claims.md, which is on-demand.tests/**/*.jsthat callpage.gotoandlogin.htmlwithout requiringhelpers.jsor going through journey-runner. It's reported inverify-module.shas a FINDING, not a FAIL, and costs one grep per run (no mxcli calls). Synced projects with their own harness see a new FINDING line. M effort.helpers.jsor journey-runner. It's the strongest fix, but it refuses legitimate custom harnesses and changes a gate verdict for existing projects. M effort.Recommendation: Option 2. Warn for one release, then decide on blocking from how often it fires. Optional extra: put the recovery recipe behind the error text
Maximum number of sessions exceeded. The trigger-lookup experiment on branchexperiment/task-routed-rules(bin/rule-lookup.sh) would then inline it exactly when the error shows up, at zero always-loaded words.Decision needed from you: Should hand-written test scripts that skip the harness be flagged (option 2), refused (option 3), or only documented (option 1)? Reply "go option N" to build it.
Generated by Claude Code
new(harness-scan): flag test scripts that sign in without the toolkit…
learn(testing-shape): returned-defect-string false-green register row (…
fix(verify-module, tests): portability markers on the guarded pgrep; …
test(behaviour): commit the baseline-adds A/B tasks and scorer so the…
Bug sweep 2026-10-26: gates, runners, doctor, lint receipt, git hooks…
MendixMau commentedon Oct 26, 2026
Fixed in #578 (squash a088b4e), option 2:
harness-scan.shlists hand-written scripts that sign in and never sign out, or sign out only with a GET/logout. verify-module shows it as an INFO row and no verdict changes. On three real scripts it flagged exactly the one that caused the cap. The false-green row in testing-shape.md is in too, A/B-tested.Generated by Claude Code