diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ae88844..c2e5509b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- perf(exec): **~55% fewer forks and ~60% fewer process starts per exec** (Windows: ~100 ms each; follow-up to #584/#585). Measured on a two-module Purchasing/Approval app (Mendix 11.13, `Gate mode: check`), Linux, `strace -f` counting clone+execve for one `./bin/exec.sh`: forks 733 → 335, execs 350-375 → 135, 3 runs each; verdicts, exit codes, output and the BUILD-LOG row are byte-identical. What changed: **exec.sh** resolves the model, platform, Python and toolkit root once and hands them down as exported memos (`MXTK_MPR_MEMO`, `MXTK_MODEL_DIR_MEMO`, `MXTK_PLATFORM`, `MXTK_PY_MEMO`), uses `EPOCHSECONDS` / `printf %(…)T` / `${x##*/}` instead of `date` / `basename` / `dirname`, reads PROJECT.md and CLAUDE.local.md fields in-shell, and parses lint-gate's output with ONE awk instead of ~10 grep/sed/tr/sort/paste; **_common.sh** waits for an unbounded gate with `wait` instead of a `sleep 1` process per second (10 on a warm build, 150+ on a cold Windows one), reads the model version with one Python start instead of two, and no longer starts basename/cut/sed/grep to match a Studio Pro folder to a version; **model-stamp.sh** asks git once (it asked 6 times), keeps the file list in an array instead of mktemp/paste/rm, reads the stamp once, and `clear` answers before any set-up; **snapshot-mpr.sh**, **look-ledger.sh**, **script-owners.sh**, **metrics-ledger.sh**, **row-groups.sh**, **build-plan-status.sh**, **lint-gate.sh**, **exec-approval.sh**, **mdl-scan.sh** drop their dirname/basename/date/ls|wc|tr/find|wc forks; lint-gate skips its rule-receipt Python while no rule file, receipt or disabled-list is newer than the last clean check. `MXTK_PROFILE=2` traces the scripts exec.sh starts as well (=1 counts only exec.sh's own commands, so a started script is one line there), and the profile lines now name the file. Reported by mkr87. — mkr87 (#584, #585) - new(doctor, init): **`bin/doctor.sh ` measures performance, not just presence**: a "Performance (measured)" section reads the last 20 `steps:` rows of the project's BUILD-LOG and reports the median exec time, the median gate and the slowest step against a normal machine (17-34 s per exec, gate 9-20 s), warning above 60 s per exec or 45 s gate with the lever to pull; on Windows with slow process starts it recommends (never requires) WSL2 or a dev container (Docker Engine in WSL2, Rancher Desktop, Podman Desktop). `Gate mode: check` is the default for every project, existing ones included after sync (`Gate mode: full` in PROJECT.md keeps a full build per exec). Windows field run (ExpenseTracker, Windows 11, Git Bash, 103 ms per process start): check-only execs took 90-103 s against 138-139 s with the full build (gate 60 s → 12-14 s); the 60-70 s of non-gate overhead is the next target. Reported by mkr87. — mkr87 (#584, #585) - new(exec): **check-only gate by default (`Gate mode: full` in PROJECT.md, or `MXTK_GATE_MODE=full`, opts out)**: each exec runs `mx check` (beside mxbuild, same version) instead of the full deploy build, converting its JSON into mxbuild's errors shape so the delta gate, rollback and report are unchanged. The full build still runs when Java/JavaScript/theme/widgets changed since the last full build, when the project has Java actions and the script drops or renames something (`drop … if exists`, the re-create idiom, does not count), on `--patch`, before any full build is recorded, and after 10 check-only passes in a row (`MXTK_CHECK_MAX`). The other full builds sit at the work's natural boundaries, as mxcli itself advises (`mx check` after every exec, mxbuild when packaging): a new pre-push hook and `module-close.sh` run `bin/verify-model.sh` when the stamp says the model was only `mx check`ed (the stamp now records `kind: check|full`; `model-stamp.sh check --full`), and `bin/verify-model.sh` closes a batch on demand. Commits stay allowed on a check stamp. Field run (purchase-requests app, Linux, Mendix 11.13.0): gate 20 s → 9-10 s; a push after check-only execs ran the full build (20 s) and re-stamped the model as full; a dropped attribute caught as 6 × CE1613 by `mx check` on the broken model. Golden input: captured `mx check -j` output in `tests/wave2/fixtures/mx-check/`. For #584/#585 (Windows, mkr87). — mkr87 (#584, #585) - perf(exec, doctor): **Windows exec overhead (#584/#585)**: exec.sh resolves mxbuild and the model version once per run and exports them (`MXBUILD_PATH`, `MXTK_MODEL_VERSION`) instead of re-probing per call; the build-plan page refresh runs in the background after the verdict (`MXTK_SYNC_REFRESH=1` waits); the first full build after an hour says a cold start can take 2-3 min; `MXTK_PROFILE=1` traces every command with a timestamp and prints the 15 slowest lines plus the step times; `bin/doctor.sh` measures process-start cost and above 30 ms names the antivirus exclusions to ask for (or WSL2/dev container); a missing lint baseline now says `bin/lint-gate.sh --update-baseline` instead of "lint could not run (exit 2)". Reported by mkr87. — mkr87 (#584, #585) @@ -101,7 +102,6 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m - fix(exec): **a script restated from a stale DESCRIBE now WARNs instead of silently reverting an earlier exec** (#261). A script drafted from a DESCRIBE taken before an earlier script of the batch ran put four old messages back, with check, mxbuild and lint green. New `project-bin/describe-baseline.sh stamp` writes `-- baseline: sha256=` (plus `-- baseline-mxcli: `) from a DESCRIBE at draft time; `exec.sh` runs `verify` only for a script that carries the header, so other scripts cost one grep and no mxcli call. It reports DRIFT with the first lines of the diff, STALEHEADER, and RESTAMP after an mxcli upgrade (DESCRIBE text may change, so that is not called drift). A re-run by the script's own last recorded write (the #269 ledger) and a fresh model are not drift. Nothing is refused and the exit codes do not change. Preflight STOP 21 gains one clause (`-- supersedes:`, `stamp`, exec.sh only warns) and its trigger names `create or modify` (+24 words). Not-found DESCRIBE and `--version` output are captured from mxcli v0.26.0. Lands after the #269 PR; with it, supersedes draft #311. **Behaviour change to check in your project:** `exec.sh` output can carry `WARN: stale draft: …` lines, and the BUILD-LOG detail cell `WARN stale baseline: `; `MXTK_RESTATE_GUARD=off` skips the check. — AlvinnYeo (#261) - fix(exec): **re-running an older `mdlsource/` script over a newer owner's live work now WARNs instead of passing silently** (#269). Three field apps had 99/579, 130/357 and 38/159 documents created by more than one script, and a replay of an older one reverted the newer with exit 0 and every gate green. New `project-bin/script-owners.sh` maps each microflow, nanoflow, page and snippet to the scripts that create or restate it, in run order (digit runs compared as numbers, so `2-core/` runs before `11-fixes/`; `done-` ignored), and reads `-- supersedes:` / `-- superseded-by:` headers. `exec.sh` records every kept write in a machine-local last-writer ledger (`.claude/loop/script-owners/`, self-gitignored, compacted per write); `restore-mpr.sh` marks it on a manual restore. A replay WARNs only when the newer owner's write is live: ledger first, else one DESCRIBE per document, at most 5 per exec (the rest print `CONFLICT (not checked)`). A new owner without a supersedes header gets one `NOTE:` line on its first run, kept out of BUILD-LOG. Nothing is refused and the exit codes do not change. With no co-owner, `exec.sh` runs one grep over `mdlsource/` and no mxcli call. `gate-check.sh` prints the co-owned count when it is above 0. Part of the rebuilt draft #311 (with the #261 PR). **Behaviour change to check in your project:** `exec.sh` output can carry `WARN: restatement: …` and `NOTE: new owner of …` lines; the BUILD-LOG detail cell can carry `WARN reverts: `. Set `MXTK_RESTATE_GUARD=off` to skip the check. — MendixMau (#269) - fix(exec): **re-running an older `mdlsource/` script over a newer owner's live work now WARNs instead of passing silently** (#269). Three field apps had 99/579, 130/357 and 38/159 documents created by more than one script, and a replay of an older one reverted the newer with exit 0 and every gate green. New `project-bin/script-owners.sh` maps each microflow, nanoflow, page and snippet to the scripts that create or restate it, in run order (digit runs compared as numbers, so `2-core/` runs before `11-fixes/`; `done-` ignored), and reads `-- supersedes:` / `-- superseded-by:` headers. `exec.sh` records every kept write in a machine-local last-writer ledger (`.claude/loop/script-owners/`, self-gitignored, compacted per write); `restore-mpr.sh` marks it on a manual restore. A script counts as run when it has a ledger row, a `done-` name, or an applied `docs/BUILD-LOG.md` row, so a fresh clone or cloud session (empty ledger) still warns on a replay over committed newer work; a script nobody ran yet (draft-ahead) stays quiet. The ledger cannot see Studio Pro edits or a `git pull`/`checkout` of the `.mpr`. A replay WARNs only when the newer owner's write is live: ledger first, else one DESCRIBE per document, at most 5 per exec (the rest print `CONFLICT (not checked)`). A new owner without a supersedes header gets one `NOTE:` line on its first run, kept out of BUILD-LOG. Nothing is refused and the exit codes do not change. With no co-owner, `exec.sh` runs one grep over `mdlsource/` and no mxcli call. `gate-check.sh` prints the co-owned count when it is above 0. Part of the rebuilt draft #311 (with the #261 PR). **Behaviour change to check in your project:** `exec.sh` output can carry `WARN: restatement: …` and `NOTE: new owner of …` lines; the BUILD-LOG detail cell can carry `WARN reverts: `. Set `MXTK_RESTATE_GUARD=off` to skip the check. — MendixMau (#269) -- fix(exec): **re-running an older `mdlsource/` script over a newer owner's live work now WARNs instead of passing silently** (#269). Three field apps had 99/579, 130/357 and 38/159 documents created by more than one script, and a replay of an older one reverted the newer with exit 0 and every gate green. New `project-bin/script-owners.sh` maps each microflow, nanoflow, page and snippet to the scripts that create or restate it, in run order (digit runs compared as numbers, so `2-core/` runs before `11-fixes/`; `done-` ignored), and reads `-- supersedes:` / `-- superseded-by:` headers. `exec.sh` records every kept write in a machine-local last-writer ledger (`.claude/loop/script-owners/`, self-gitignored, compacted per write); `restore-mpr.sh` marks it on a manual restore. A script counts as run when it has a ledger row, a `done-` name, or an applied `docs/BUILD-LOG.md` row, so a fresh clone or cloud session (empty ledger) still warns on a replay over committed newer work; a script nobody ran yet (draft-ahead) stays quiet. The ledger cannot see Studio Pro edits or a `git pull`/`checkout` of the `.mpr`. A replay WARNs only when the newer owner's write is live: ledger first, else one DESCRIBE per document, at most 5 per exec (the rest print `CONFLICT (not checked)`). A new owner without a supersedes header gets one `NOTE:` line on its first run, kept out of BUILD-LOG. Nothing is refused and the exit codes do not change. With no co-owner, `exec.sh` runs one grep over `mdlsource/` and no mxcli call. `gate-check.sh` prints the co-owned count when it is above 0. Part of the rebuilt draft #311 (with the #261 PR). **Behaviour change to check in your project:** `exec.sh` output can carry `WARN: restatement: …` and `NOTE: new owner of …` lines; the BUILD-LOG detail cell can carry `WARN reverts: `. Set `MXTK_RESTATE_GUARD=off` to skip the check. — MendixMau (#269) - fix(root): **the installed and toolkit copies of `_common.sh` now resolve the same project root from inside an app folder** (#489). With the cwd in `app/`, `app/javasource` or `apps//`, the installed copy (`/bin/_common.sh`, tier 2) answered the project while the toolkit copy (tier 3) stopped at the folder holding the `.mpr`, so the same script wrote its look ledger, metrics ledger and BUILD-LOG to two roots depending on where it was run from. `mxtk_walk_root` now returns the project folder when the folder it stops at is `

/app` or `

/apps/` and `

` is a marked project folder (an installed `bin/_common.sh`, or `.claude/` beside `PROJECT.md`); an unmarked app folder is still returned as itself, and so is an app folder that is a project of its own (its own install, or `.claude/` beside `PROJECT.md`); the folder names match in any letter case, and single-tree resolution is unchanged. Builtin tests only, no extra process. `find_mpr` now always prints an absolute path (a cwd-relative `MPR_FILE` broke once a caller did `cd "$PROJECT_ROOT"`), and with several apps and no `MPR_FILE`, sourcing `_common.sh` from inside one app's folder exports `MPR_FILE` as that app's model, so the pick survives exec.sh's `cd "$PROJECT_ROOT"` and reaches `snapshot-mpr.sh` (outside any one app nothing is picked and `find_mpr` still refuses; an explicit `MPR_FILE` or a root/`app/` model always wins; the pick is marked `MXTK_MPR_PICKED`, so a child script whose cwd is inside a different `apps//` re-derives instead of inheriting it, and `model-stamp.sh` says "(picked from your cwd)" when it refuses one). `lint-gate.sh` now honours `MPR_FILE` when `MPR` is unset and it resolves under the project root, so exec.sh run from inside one app no longer reaches its "several apps" refusal; an `MPR_FILE` that does not resolve there (cwd-relative, or absolute through a symlinked path) falls back to discovery as before, never to exit 2. An inherited pick from another project is dropped. **Behaviour change to check in your project:** a toolkit-copy script run from inside `app/` or `apps//` of a wired project now reads and writes state at the project root, as the installed copy already did; if `app/` (or `apps//`) holds `.mpr-snapshots/`, `docs/BUILD-LOG.md` or `.claude/loop/` from a toolkit-copy run, merge them into the project root. `find_mpr` with `MPR_FILE` set now prints an absolute path where it echoed a relative one. A project whose `bin/_common.sh` is hand-edited needs `sync-project.sh --only-bin _common.sh --upgrade-bin _common.sh`, as for #420. `test-common-root-multiapp.sh` R8a, R9c-f, R13g, R17a-j and R18a-h (cwd in `app/`, `app/javasource`, `apps/A`, `apps/A/javasource`, a nested or wired app folder, `App/`, both copies; the cwd's app surviving a `cd` to the root and a child `snapshot-mpr.sh`; a child in another app re-deriving, R18h a child in another project): 67/67 with the fix; `test-lint-gate-accept.sh` T-MA (lint-gate with `MPR_FILE` in a multi-app folder) and T-MF (an unresolvable `MPR_FILE` falls back to discovery). — MendixMau (review 2026-10-20) - fix(portability): **`bug-lookup.sh --index` and `test-routing-tier.sh` no longer depend on sed expanding `\t`, and `check-portability.sh` flags `\t` in a sed pattern or replacement outside a bracket** (#488). BSD sed (macOS) reads `\t` there as the letter `t`, so `sed -E 's/^([0-9]+):## /\1\t/'` fed `awk -F'\t'` no tab and `--index` printed 222 empty lines; the line-count check passed anyway. The index is now one awk (same 222 lines, byte-identical on GNU), the fixture builds its sed script with a real tab (`TAB=$'\t'`), and `test-bug-lookup.sh` runs `--index` under a sed shim that reads `\t` as `t` (fails on master: 0 non-empty lines). The lint also had a blind spot: its comment filter dropped any line holding `:#`, so the bug-lookup line was never checked; it is now anchored on the line number. New rule over the current tree: the two sites on master, 0 elsewhere. `test-sed-tab-bracket.sh` T4. Simulated on Linux only; one macOS run is still owed. — MendixMau (review 2026-10-20) - fix(look hook, verify-module, coverage): **the look hook records from any cwd, and `verify-module.sh` loses four instrument faults** (#446). (a) The PostToolUse(Read) hook was `bash bin/look-ledger.sh seen || true`: from any cwd but the project root (a subagent in a private copy, a shell in `app/javasource`) it exited 127 and `|| true` hid it (field: 30 pages owed, 1 seen). `install-claude-permissions.sh` now writes it to `cd "$CLAUDE_PROJECT_DIR"` first; a missing script or failed run goes to stderr, and is logged once to `.claude/loop/look/hook.log` when that can be written. The installer (and `sync-project.sh`'s hook repair) rewrites the old form in place and removes duplicate look hooks, so one remains; `doctor.sh` flags any look hook not in the root-resolving form. (b) A module with 0 pages in the page scope reports journeys N/A instead of a FAULT (`page-scope.sh` now runs before the runtime rungs; same mxcli calls) — jopterhorst (comment on #446); a missing journey JSON names any `tests/*.test.sh` that exist in another shape; missing e2e dependencies (playwright, axe-core, probed from `JOURNEY_RUNNER`/`MONKEY_JS`/`DESIGN_AUDIT_JS`'s folders) are one FAULT row with the one fix (`npm install playwright axe-core && npx playwright install chromium`), journeys and monkey SKIPPED as dependents of playwright, the design audit static-only; the coverage rung's budget is 1200 s (was 300); `summary.tsv` ends with a `verdict` row (CLEAN / FINDINGS / INCOMPLETE), absent when a run was cut short (`report-normalize.js` skips it); and `coverage-preflight.sh`'s PHANTOM CLAIM pass gives each BRD only the claims it can resolve (12-BRD bench: 16.1 s to 4.2 s), with identical verdicts except that false PHANTOMs for BRDs in another source folder or named only by `.id` are gone (exit can drop 1→0). Found while proving it: every rung's watchdog `sleep` kept the script's stdout open, so `verify-module.sh | tee` waited up to 30 minutes after the verdict; the watchdogs now drop fds 3/4 and take their sleep with them. **Behaviour change to check in your project:** a page-less module's `summary.tsv` has `journeys SKIPPED (N/A: ...)` and the run can exit 0; missing e2e dependencies add an `e2e harness dependency` FAULT row and SKIPPED journeys/monkey rows; `summary.tsv` gains a last `verdict` row; `coverage-preflight.sh` can exit 0 where it exited 1 (the false PHANTOMs above); `.claude/settings.json`'s look hook command changes. Fixtures: `test-look-hook-cwd.sh`, `test-verify-module-446.sh`, `test-coverage-phantom-scope.sh`. — MendixMau (field 2026-10-17), jopterhorst (page-less modules) diff --git a/bin/doctor.sh b/bin/doctor.sh index 5ee40623..a537ab9a 100755 --- a/bin/doctor.sh +++ b/bin/doctor.sh @@ -492,7 +492,7 @@ if [ -n "$_perf_log" ] && [ -f "$_perf_log" ] && grep -q 'steps: ' "$_perf_log" warn "slow execs — $_pmsg" [ "$_pgate" -gt "${MXTK_PERF_GATE_WARN:-45}" ] && note "The gate (mxbuild) dominates: 'Gate mode: check' in PROJECT.md runs mx check per exec and the full build before push." [ -n "$_spawn_ms" ] && [ "$_spawn_ms" -gt "${MXTK_SPAWN_WARN_MS:-30}" ] && note "Process starts are slow too (above): that is the machine, not the model — see the advice there." - note "Per-command profile of one exec: MXTK_PROFILE=1 ./bin/exec.sh