Skip to content
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m

## Unreleased

- fix(gate-check, gate-cache): **the closeout verdict cache also keys on file CONTENT, so an overwrite that keeps an old mtime (`cp -p`, `rsync -t`, `tar x`, `unzip`, `touch -d`) can no longer replay a stale verdict**: a signed triage.md copied over a FAIL one still printed the cached Stage 0 FAIL (PR #589 review); the cache now stores one `find | sort | xargs cksum | cksum` signature of the project files (skipping .git, node_modules, .mxtk, mprcontents, deployment, *.mpr, files over 1 MB) and any difference, or a signature that cannot be computed, re-evaluates; new `tests/wave2/test-gate-cache.sh` (hit, edit/delete/waiver miss, cp -p miss, TTL 0, 'not evaluated' line); `MXTK_GATE_CACHE_TRACE=1` prints `gate-cache: hit|miss` on --closeout. Field run: ab-581/base copy (197 MB, 633 hashed files) - signature 28 ms, cache hit 1464 -> 1498 ms vs 2567-2786 ms full evaluation - PR #589 reviewer, OlafFede (#583) — OlafFede (#583)
- perf(gate-check, closeout): **a Stage close starts about 70% fewer processes (Stage 3 on a small project: `--closeout` 820 -> 127 starts, `gate-check <dir> 3` 560 -> 275, no-stage run 626 -> 338; verdict lines unchanged)**: (1) `--closeout` replays the evaluation of the stage query that just ran (variables snapshotted in `<project>/.mxtk/gate-cache`, valid only while nothing under the project or the toolkit is newer than that run and the key matches; any doubt re-evaluates; `MXTK_GATE_CACHE_TTL_MIN=0` turns it off); (2) a stage query no longer evaluates obligation/artifact rows owed from later stages and says how many it skipped (the summary line is labelled the same way); (3) the register is parsed once per run instead of one awk per lookup; (4) the toolkit `git fetch` runs at most once per 15 min (`MXTK_FETCH_TTL_MIN`, `MXTK_NO_FETCH=1` unchanged); plus pure-bash trimming in the close-out block and the obligation reverse check. Reported by OlafFede (#583). — OlafFede (#583)
- new(doctor, speed-setup): **`bin/doctor.sh --bench [<project>]`** times 50 process starts, git status, python/node start, one gate-check, an mxcli check and (on a COPY of the model) mx check against measured reference constants, prints one verdict line and writes `.claude/loop/bench-<timestamp>.tsv` for comparing machines; a **Machine health** section (normal run and `--bench`) warns on leftover mxbuild `deno` helpers and free memory under 2 GB (mxbuild is killed with exit 137 when memory runs out), `--cleanup` stops exactly those helpers after a typed y (no tty: prints the command); both runs end with ONE **Recommended fix** block (Gate mode: check, else WSL, dev container, or Claude in the cloud; antivirus-exclusion advice removed, it needed IT); new `skills/speed-setup.md`. Field run: ab-581/gk on Linux 4 cores, 46 orphan deno helpers found, bench 10.4 s vs 10.9 s reference — Maurits (mauritscgvisser)
- perf(exec): **~55% fewer forks and ~60% fewer process starts per exec** (Windows: ~100 ms each; follow-up to #584/#585). Measured on a two-module Purchasing/Approval app (Mendix 11.13, `Gate mode: check`), Linux, `strace -f` counting clone+execve for one `./bin/exec.sh`: forks 733 → 335, execs 350-375 → 135, 3 runs each; verdicts, exit codes, output and the BUILD-LOG row are byte-identical. What changed: **exec.sh** resolves the model, platform, Python and toolkit root once and hands them down as exported memos (`MXTK_MPR_MEMO`, `MXTK_MODEL_DIR_MEMO`, `MXTK_PLATFORM`, `MXTK_PY_MEMO`), uses `EPOCHSECONDS` / `printf %(…)T` / `${x##*/}` instead of `date` / `basename` / `dirname`, reads PROJECT.md and CLAUDE.local.md fields in-shell, and parses lint-gate's output with ONE awk instead of ~10 grep/sed/tr/sort/paste; **_common.sh** waits for an unbounded gate with `wait` instead of a `sleep 1` process per second (10 on a warm build, 150+ on a cold Windows one), reads the model version with one Python start instead of two, and no longer starts basename/cut/sed/grep to match a Studio Pro folder to a version; **model-stamp.sh** asks git once (it asked 6 times), keeps the file list in an array instead of mktemp/paste/rm, reads the stamp once, and `clear` answers before any set-up; **snapshot-mpr.sh**, **look-ledger.sh**, **script-owners.sh**, **metrics-ledger.sh**, **row-groups.sh**, **build-plan-status.sh**, **lint-gate.sh**, **exec-approval.sh**, **mdl-scan.sh** drop their dirname/basename/date/ls|wc|tr/find|wc forks; lint-gate skips its rule-receipt Python while no rule file, receipt or disabled-list is newer than the last clean check. `MXTK_PROFILE=2` traces the scripts exec.sh starts as well (=1 counts only exec.sh's own commands, so a started script is one line there), and the profile lines now name the file. Reported by mkr87. — mkr87 (#584, #585)
- new(doctor, init): **`bin/doctor.sh <project>` measures performance, not just presence**: a "Performance (measured)" section reads the last 20 `steps:` rows of the project's BUILD-LOG and reports the median exec time, the median gate and the slowest step against a normal machine (17-34 s per exec, gate 9-20 s), warning above 60 s per exec or 45 s gate with the lever to pull; on Windows with slow process starts it recommends (never requires) WSL2 or a dev container (Docker Engine in WSL2, Rancher Desktop, Podman Desktop). `Gate mode: check` is the default for every project, existing ones included after sync (`Gate mode: full` in PROJECT.md keeps a full build per exec). Windows field run (ExpenseTracker, Windows 11, Git Bash, 103 ms per process start): check-only execs took 90-103 s against 138-139 s with the full build (gate 60 s → 12-14 s); the 60-70 s of non-gate overhead is the next target. Reported by mkr87. — mkr87 (#584, #585)
Expand Down
50 changes: 48 additions & 2 deletions bin/gate-check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1947,6 +1947,42 @@ protocol_is_relevant() {
return 1
}

# Evaluation cache (#583): `--closeout` right after a stage query replays that query's variables
# instead of re-evaluating. Everything from here to the "Regenerate index.html" block is the
# evaluation; see bin/lib/gate-cache.sh for exactly when a hit is allowed.
. "$TOOLKIT_DIR/bin/lib/gate-cache.sh"
GC_CACHE_HIT=0
[ "$CLOSEOUT" = "1" ] && gc_cache_try
[ -n "${MXTK_GATE_CACHE_TRACE:-}" ] && [ "$CLOSEOUT" = "1" ] && echo "gate-cache: $([ "$GC_CACHE_HIT" = 1 ] && echo hit || echo miss)" >&4
if [ "$GC_CACHE_HIT" = "1" ]; then
eval "$GC_CACHE_SNAPSHOT"
else
gc_cache_begin

# The fetch below runs at most once per MXTK_FETCH_TTL_MIN minutes (default 15; 0 = every run) per
# toolkit clone (#583: a git fetch is a network round trip plus four processes on every query, and a
# Stage close runs three queries in a row). The stamp lives inside the clone's git dir, written only
# after a fetch SUCCEEDED, so a failed or skipped fetch is never remembered as a good one. A fresh
# stamp reuses the remote-tracking ref the last fetch left, which is exactly what a successful
# fetch would have compared against, minutes later. A worktree or submodule checkout has a .git
# FILE ("gitdir: <path>"), read here without a process; a .git that is neither means there is
# nowhere safe to put the stamp, and every run fetches as before.
GC_FETCH_STAMP=""
if [ -d "$TOOLKIT_DIR/.git" ]; then
GC_FETCH_STAMP="$TOOLKIT_DIR/.git/mxtk-last-fetch"
elif [ -f "$TOOLKIT_DIR/.git" ]; then
_gc_gd=""; read -r _ _gc_gd < "$TOOLKIT_DIR/.git" 2>/dev/null || true
case "$_gc_gd" in /*|[A-Za-z]:*) ;; ?*) _gc_gd="$TOOLKIT_DIR/$_gc_gd" ;; esac
[ -n "$_gc_gd" ] && [ -d "$_gc_gd" ] && GC_FETCH_STAMP="$_gc_gd/mxtk-last-fetch"
fi
gc_fetch_is_fresh() {
local ttl="${MXTK_FETCH_TTL_MIN:-15}"
case "$ttl" in ''|*[!0-9]*) return 1 ;; esac
[ -n "$GC_FETCH_STAMP" ] && [ "$ttl" -gt 0 ] && [ -f "$GC_FETCH_STAMP" ] || return 1
[ -n "$(find "$GC_FETCH_STAMP" -mmin "-$ttl" 2>/dev/null)" ]
}
gc_fetch_stamp() { [ -n "$GC_FETCH_STAMP" ] && : > "$GC_FETCH_STAMP" 2>/dev/null; return 0; }

TOOLKIT_HEAD="$(git -C "$TOOLKIT_DIR" rev-parse --short HEAD 2>/dev/null || echo "unknown")"

# Resolve the reference tip. Fetch failure must not silently degrade to the old self-comparison:
Expand All @@ -1964,8 +2000,10 @@ if [ "${MXTK_NO_FETCH:-0}" = "1" ]; then
TOOLKIT_REF="$TOOLKIT_HEAD"
SYNC_REF_LABEL="local HEAD — UNVERIFIED, fetch skipped (MXTK_NO_FETCH=1)"
elif git -C "$TOOLKIT_DIR" rev-parse --abbrev-ref '@{u}' >/dev/null 2>&1 \
&& GIT_TERMINAL_PROMPT=0 GIT_SSH_COMMAND="${GIT_SSH_COMMAND:-ssh -o BatchMode=yes -o ConnectTimeout=5}" \
git -C "$TOOLKIT_DIR" -c http.lowSpeedLimit=1000 -c http.lowSpeedTime=8 fetch --quiet 2>/dev/null; then
&& { gc_fetch_is_fresh \
|| { GIT_TERMINAL_PROMPT=0 GIT_SSH_COMMAND="${GIT_SSH_COMMAND:-ssh -o BatchMode=yes -o ConnectTimeout=5}" \
git -C "$TOOLKIT_DIR" -c http.lowSpeedLimit=1000 -c http.lowSpeedTime=8 fetch --quiet 2>/dev/null \
&& gc_fetch_stamp; }; }; then
TOOLKIT_REF="$(git -C "$TOOLKIT_DIR" rev-parse --short '@{u}' 2>/dev/null || echo "unknown")"
else
TOOLKIT_REF="$TOOLKIT_HEAD"
Expand Down Expand Up @@ -2643,6 +2681,8 @@ printf "Open questions (raised?): %s — %s\n" "$OQ_STATUS" "$OQ_NOTE"
# INFORMATIONAL here, not blocking: the verify loop is optional on small projects, a module can
# legitimately have no pages (an integration module), and a project that joined at Stage 5 owes
# nothing below its adoption point. What must not happen is silence. Stage 6 still blocks.
# Rows for stages after the requested one are not evaluated (#583); see the two libs.
case "$REQUESTED_STAGE" in P|p|[0-7]) export MXTK_ROWS_UPTO="$REQUESTED_STAGE" ;; *) unset MXTK_ROWS_UPTO ;; esac
if [ -r "$TOOLKIT_DIR/bin/lib/obligation-check.sh" ]; then
# shellcheck source=lib/obligation-check.sh
. "$TOOLKIT_DIR/bin/lib/obligation-check.sh"
Expand Down Expand Up @@ -3020,6 +3060,9 @@ case "${LEDGER_OWED:-}" in
*) _roll="$_roll Source files: ${LEDGER_OWED} owed (pending/fault/missing/not inventoried)"
_owed=$(( _owed + ${LEDGER_OWED:-0} )) ;;
esac
# A stage query skipped the rows owed from later stages (#583); the counts above are for the
# requested stage and earlier, and must not read as "nothing is owed anywhere".
[ -n "${MXTK_ROWS_UPTO:-}" ] && _roll="$_roll (obligation/artifact rows for stages after ${MXTK_ROWS_UPTO} not evaluated)"
printf " %s\n" "$_roll"
if [ "$_owed" -gt 0 ]; then
echo " These do not affect the exit code and never have. They are passes and files"
Expand Down Expand Up @@ -3055,6 +3098,9 @@ if [ "$N_WAIVED" = "0" ] && { [ "$N_PENDING" -ge 4 ] || [ "$N_FAIL" -ge 3 ]; };
echo " $0 --waive look/<Module> --reason \"...\" $PROJECT_DIR (one PASS on one module → WAIVED)"
fi

gc_cache_end
fi # GC_CACHE_HIT

# Regenerate index.html from these exact results — subject to three conditions, all of which
# exist because this write has destroyed a real file:
# 1. Only on a full informational run, unless --html says otherwise. A stage query is a
Expand Down
106 changes: 102 additions & 4 deletions bin/lib/artifact-check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,12 @@ _art_register_lines() {
_art_field() {
local reg="$1" want="$2" out
[ -f "$reg" ] || return 1
# Loaded register (see _art_kv_load): a lookup is parameter expansion, not an awk process.
if [ "$_ART_KV_REG" = "$reg" ]; then
_art_kv_get "$want" || return 1
printf '%s\n' "$_ART_KV_VAL"
return 0
fi
out="$(awk -v want="$want" '
/<!--/ { c = 1 } c { if ($0 ~ /-->/) c = 0; next }
{
Expand All @@ -102,11 +108,84 @@ _art_field() {
printf '%s\n' "$out"
}

# The register parsed ONCE per run (#583). _art_field above is called two or three times per
# manifest row, and each call was an awk process that re-read the whole register: 58 awk
# starts in one Stage 3 query, each ~100 ms under Git Bash. _art_kv_load runs the very same
# awk body once, emitting "key<TAB>value" lines (key lower-cased and trimmed, empty values
# dropped, comments skipped, first line per key wins at lookup) into one string; _art_kv_get
# answers from it with parameter expansion. Loaded only between mxtk_artifacts_report /
# mxtk_closeout_report entry and exit, so a caller that edits the register between two bare
# _art_field calls still gets the awk path and a fresh read.
_ART_KV=""; _ART_KV_REG=""; _ART_KV_VAL=""
_art_kv_load() {
local reg="$1"
_ART_KV_REG=""; _ART_KV=""
[ -f "$reg" ] || return 0
_ART_KV="$(awk '
/<!--/ { c = 1 } c { if ($0 ~ /-->/) c = 0; next }
{
l = $0; gsub(/^[ \t>*_-]+/, "", l); gsub(/\*/, "", l)
i = index(l, ":"); if (i == 0) next
key = tolower(substr(l, 1, i - 1)); gsub(/^[ \t]+|[ \t]+$/, "", key)
val = substr(l, i + 1); gsub(/^[ \t]+|[ \t]+$/, "", val)
if (val == "") next
if (index(key, "\t")) next # no lookup label contains a TAB; keep the record format unambiguous
print "\n" key "\t" val
}' "$reg")"
# Leading newline so every record, the first included, is preceded by one.
_ART_KV=$'\n'"$_ART_KV"
_ART_KV_REG="$reg"
}
_art_kv_unload() { _ART_KV_REG=""; _ART_KV=""; }
# _art_kv_get <lower-cased label> — sets _ART_KV_VAL; rc 1 when absent. Values never contain a
# newline (awk works per line), so "up to the next newline" is exactly the value. A key never
# contains a TAB it could be confused with: split at the first colon, trimmed.
_art_kv_get() {
local want="$1" rest
case "$_ART_KV" in
*$'\n'"$want"$'\t'*) ;;
*) return 1 ;;
esac
rest="${_ART_KV#*$'\n'"$want"$'\t'}"
_ART_KV_VAL="${rest%%$'\n'*}"
[ -n "$_ART_KV_VAL" ]
}

# Lower-case a label without a process in the common case (sets _ART_LC). Manifest ids are
# lower-case ASCII; only the stage letter ("P") needs mapping, and tr stays as the fallback
# for anything else with an upper-case letter.
_ART_LC=""
_art_lc() {
case "$1" in
*[A-Z]*) _ART_LC="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" ;;
*) _ART_LC="$1" ;;
esac
}

# _art_waiver <register> <artifact-id> <stage> — prints the reason, exits 0 if waived.
# Case-insensitive for the same reason obligation-check documents: a waiver that silently
# missed on capitalisation is one the author believes is in force and the gate does not.
# Loaded-register form: sets _ART_WAIVER_REASON, prints nothing, forks nothing. Same two
# lookups, same order, same text as the printing form below.
_ART_WAIVER_REASON=""
_art_waiver_set() {
local reg="$1" id="$2" st="$3" lst="$3"
_ART_WAIVER_REASON=""
[ "$_ART_KV_REG" = "$reg" ] || return 2
case "$st" in P) lst=p ;; esac
_art_lc "waived artifact $id"
_art_kv_get "$_ART_LC" && { _ART_WAIVER_REASON="$_ART_KV_VAL"; return 0; }
_art_lc "waived stage $lst"
_art_kv_get "$_ART_LC" \
&& { _ART_WAIVER_REASON="with its whole stage (Waived stage $st: $_ART_KV_VAL)"; return 0; }
return 1
}
_art_waiver() {
local reg="$1" id="$2" st="$3" r
if [ "$_ART_KV_REG" = "$reg" ]; then
_art_waiver_set "$reg" "$id" "$st" && { printf '%s\n' "$_ART_WAIVER_REASON"; return 0; }
return 1
fi
r="$(_art_field "$reg" "$(printf 'waived artifact %s' "$id" | tr '[:upper:]' '[:lower:]')")" \
&& { printf '%s\n' "$r"; return 0; }
r="$(_art_field "$reg" "$(printf 'waived stage %s' "$st" | tr '[:upper:]' '[:lower:]')")" \
Expand Down Expand Up @@ -178,8 +257,10 @@ _art_register_rank() {
# names it), mtime as the fallback for dateless names. Returns 0 for yes.
_art_newer() {
local ad bd am bm
ad="$(basename "$1" | grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' | head -1)"; ad="${ad:-0000-00-00}"
bd="$(basename "$2" | grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' | head -1)"; bd="${bd:-0000-00-00}"
# Pure bash: this ran basename + grep + head per file, twice, per comparison.
ad="${1##*/}"; bd="${2##*/}"
if [[ "$ad" =~ [0-9]{4}-[0-9]{2}-[0-9]{2} ]]; then ad="${BASH_REMATCH[0]}"; else ad="0000-00-00"; fi
if [[ "$bd" =~ [0-9]{4}-[0-9]{2}-[0-9]{2} ]]; then bd="${BASH_REMATCH[0]}"; else bd="0000-00-00"; fi
[[ "$ad" > "$bd" ]] && return 0
[[ "$ad" < "$bd" ]] && return 1
# GNU first, BSD second — on GNU `stat -f` succeeds with the wrong meaning.
Expand Down Expand Up @@ -270,6 +351,15 @@ mxtk_artifacts_report() {
return 0
fi

# Parse the register once for every lookup below (#583); unloaded at the end.
_art_kv_load "$reg"

# Stage limit (#583): rows for stages AFTER the one asked about are not evaluated. They can
# not change a verdict for the requested stage, and on a Stage 3 query they were most of the
# 120-row manifest. gate-check.sh exports MXTK_ROWS_UPTO (a stage token); unset = every row.
local limit_rank=99 skipped_n=0
[ -n "${MXTK_ROWS_UPTO:-}" ] && limit_rank="$(_art_rank "$MXTK_ROWS_UPTO")"

local adopted adopted_rank=0 mode
adopted="$(_art_adopted_stage "$reg" 2>/dev/null || true)"
[ -n "$adopted" ] && adopted_rank="$(_art_rank "$adopted")"
Expand Down Expand Up @@ -318,6 +408,8 @@ mxtk_artifacts_report() {
MXTK_ART_STATUS="FAULT"; MXTK_ART_N_FAULT=$((MXTK_ART_N_FAULT+1)); continue
fi

if [ "$r" -gt "$limit_rank" ]; then skipped_n=$((skipped_n+1)); continue; fi

# Entry mode: excused stages are said out loud, never skipped silently. An unknown mode
# (young register, no line yet) excuses nothing.
if [ -n "$mode" ]; then
Expand Down Expand Up @@ -352,8 +444,8 @@ mxtk_artifacts_report() {
fi

local reason
if reason="$(_art_waiver "$reg" "$id" "$stage")"; then
printf 'Artifact %-24s WAIVED — %s\n' "$id" "$reason"
if _art_waiver_set "$reg" "$id" "$stage"; then
printf 'Artifact %-24s WAIVED — %s\n' "$id" "$_ART_WAIVER_REASON"
continue
fi

Expand Down Expand Up @@ -385,12 +477,18 @@ mxtk_artifacts_report() {
fi
done < "$ARTIFACT_TSV"

if [ "$skipped_n" -gt 0 ]; then
printf 'Artifacts: %s row(s) for stages after %s not evaluated — a stage query reads its own stage and earlier; run without a stage for every row\n' \
"$skipped_n" "$MXTK_ROWS_UPTO"
fi

# The one line for everything on offer and untaken — said once so it is not silent, said
# once so eight optional rows do not read as eight things missing.
if [ -n "$_art_offered" ]; then
printf 'Artifacts optional, not opted in: %s — offered at each stage open by gate-check.sh --closeout; opt in with a register line "Opt-in artifact <id>: <why>"\n' "$_art_offered"
fi

_art_kv_unload
return 0
}

Expand Down
Loading
Loading