[field 2026-10-17] UNVERIFIED rows (mxbuild exit 3) and deploy-stage failures with no recorded reason #448

Field report from a field migration project (Next.js + PostgreSQL to Mendix 11.15, 12 modules), Stage 5. mxcli v0.27.0. Related: the watcher race issue and the exec evidence issue filed from the same deep dives. Filed separately because the 10 UNVERIFIED rows exit 0 and have no recorded reason.

What we were doing

Stage 5 parallel build. Scripts were replayed on the real model with bash bin/exec.sh mdlsource/<NN-script>.mdl. The gate is a full mxbuild --target=deploy run by _common.sh. A watcher (./mxcli run --local --watch -p App.mpr) served the same app, and from about 22:57 local, private-copy authors ran more mxbuild and gradle processes at the same time.

Symptom (BUILD-LOG rows, verbatim)

| 2026-10-16T19:23:10+0800 | `07-domain-ModuleA.mdl` | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: Compilation of the app bundle failed.) - model not verified |
| 2026-10-16T19:32:21+0800 | `11-domain-ModuleB.mdl` | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: An error occurred while cleaning excluded jar files & copying files.) |
| 2026-10-16T19:35:25+0800 | `12-domain-ModuleC.mdl` | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: Compilation of the app bundle failed.) |
| 2026-10-16T20:05:26+0800 | `18-layouts.mdl`          | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: An error occurred while compiling Theme files) |
| 2026-10-16T23:47:55+0800 | `55-render-helpers.mdl`   | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: An error occurred while compiling Java actions) |

Plus one mxbuild exit 139 (Segmentation fault: 11 at _common.sh:586, under load 40 to 53) and the "gate could not run" rows (mxbuild exit 1).

Repro

Run two mxbuild --target=deploy processes against the same project directory at once, or run one while a mxcli run --local --watch rebuild is in progress. Intermittent. No single-process repro was found, so no mxcli bug is asserted.

Expected vs Actual

• Expected: serialised builds or a clear "build dir busy" message; a real error text when the deploy stage fails.
• Actual: one build exits 1 or 3 with a generic compile message. exec.sh logs only mxbuild exit N and deployment/log is overwritten by the next build, so the real cause is unrecoverable. UNVERIFIED exits 0 (exec.sh around lines 1907 to 1908), so it looks like success while the model stays unverified and the stamp is cleared.
• The earlier JDK 25 plus Gradle 8.5 hypothesis was ruled out: the deploy log shows Gradle 9.5.1 compiling under JDK 25. Failures are intermittent: 18-layouts failed UNVERIFIED at 20:05:26 and passed clean at 20:06:36, and one domain script failed then passed with no model change.

Frequency

BUILD-LOG (115 data rows): 19 gate could not run plus 10 UNVERIFIED rows, 29 runs, about 26 percent of the exec runs. Exit 3 messages: "Compilation of the app bundle failed." x5, "An error occurred while cleaning excluded jar files & copying files." x3, "An error occurred while compiling Theme files" x1, "An error occurred while compiling Java actions" x1.

Cost

19 gate-not-run at 1216 s plus 10 unverified at 717 s of gate time, about 32 minutes, plus the blocked-retry cascade (about 29 minutes, see the evidence issue) and about 12 minutes of manual re-verification of UNVERIFIED rows.

Proposed fix

• exec.sh copies the mxbuild stdout and the deployment/log tail into .claude/loop/gate-failures/<timestamp>.log and prints it.
• Take the same lock the watch build uses (or refuse while it runs); retry the deploy once when the errors file is absent before declaring "could not run" or UNVERIFIED.
• Make UNVERIFIED exit non-zero (or a distinct code) so it cannot read as success, and require verify-model.sh before the next exec.
• Open an mxcli issue only if a single-process repro appears.

Files

• bin/exec.sh
• bin/_common.sh (mxbuild invocation, around the --target=deploy call)
• bin/model-stamp.sh
• skills/iterative-build-loop.md