Skip to content

exec: ~60% fewer process starts per exec (Windows overhead, #584/#585) - #590

Merged
MendixMau merged 15 commits into
masterfrom
perf/exec-overhead
Oct 29, 2026
Merged

exec: ~60% fewer process starts per exec (Windows overhead, #584/#585)#590
MendixMau merged 15 commits into
masterfrom
perf/exec-overhead

Conversation

@MendixMau

@MendixMau MendixMau commented Oct 29, 2026

Copy link
Copy Markdown
Owner

This PR is stacked on #587: its base is perf/windows-exec-speed, and it will be retargeted to master once #587 merges.

On Windows every process start costs about 100 ms. Before this change, one exec started about 730 forks and 360 programs, which is 60–70 s of overhead on a slow laptop (#584, #585, and a colleague's field run).

What changes

  • exec.sh and _common.sh resolve once per run and hand the answers down as exported memos: the model path, platform, Python, model folder and model version.
  • Built-ins replace spawns. Timestamps, dirname and basename no longer start processes. PROJECT.md and CLAUDE.local.md fields are read in-shell. Lint output is parsed by one awk instead of about ten tools.
  • An unbounded gate uses wait instead of one sleep 1 process per second.
  • model-stamp asks git once instead of six times. lint-gate skips its rule-receipt Python while no rule file is newer than the last clean check.
  • The same fork trimming was applied in snapshot-mpr, look-ledger, script-owners, metrics-ledger, row-groups, build-plan-status, exec-approval and mdl-scan.
  • MXTK_PROFILE=2 traces the scripts exec.sh starts as well.

Evidence

Linux field run on a copy of the purchase-requests app, 3 runs each, counted with strace -f:

Measure Before After
Forks per exec 730–742 334–338
Programs started per exec 350–375 135–136
Exec time (Linux) 18.2–18.5 s 16.6–17.1 s
  • On a 100 ms-per-start Windows laptop that is an estimated 40–60 s less per exec. This has not been measured on Windows yet.
  • Console output, exit code, the BUILD-LOG row and the metrics-ledger row are identical before and after. The model fingerprint matches the old script's.
  • Fixtures: 59 scoped fixtures pass, covering every test-exec-*, model-stamp, look, lint, metrics-ledger, row-groups, mdl-scan, common, describe-baseline, build-plan, progress-report and conformance. They were re-run after merging the latest Exec speed: mx check gate by default, doctor performance check, Windows overhead fixes (#584, #585) #587: gate-kind 32/32, step-times 7/7, gate-log 24/24, lint-diff 9/9, gate-retry 52/52, crash-rollback 13/13, model-stamp 35/35, module-close 7/7.

Risk

  • Many scripts are touched. The changes are mechanical, but there are many of them.
  • The lint-receipt skip relies on file mtimes.
  • The memos assume one model per run.
  • PROJECT_ROOT is now exported to the scripts exec.sh starts.
  • Not run on Windows or macOS.

Generated by Claude Code

claude added 14 commits October 28, 2026 20:51
@claude
perf(exec): opt-in mx-check gate, Windows exec overhead fixes (#584, #…
d3e42a8 
…585)

- Gate mode: check (PROJECT.md) runs `mx check` per exec instead of the full deploy
  build; the full build still runs on Java/theme changes, on drops/renames when the project
  has Java actions, every 5th exec, on --patch and before the first full build.
- mxbuild path and model version are resolved once per run; the build-plan page refreshes in
  the background; a cold-start notice; MXTK_PROFILE=1 per-command trace; doctor measures
  process-start cost; a clear message when no lint baseline exists.

Field run: purchase-requests app (Linux, Mendix 11.13.0). The gate took 20 s with a full
build and 9-10 s with mx check; the 5th exec ran a full build. A dropped attribute was
caught as 6 x CE1613 by a real mx check on the broken model. Fixture:
tests/wave2/test-exec-gate-kind.sh (16/16), on captured mx check output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
fix(exec): drop … if exists (re-create idiom) no longer forces a full…
ed66333 
… build in check mode

E1 replay (purchase-requests app, 23 scripts): the drop/rename trigger fired on 3 ordinary
re-runnable scripts. Fixture K6k.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
perf(exec): full build at batch boundaries instead of every 5th exec …
c25f1f0 
…(Gate mode: check)

Follows mxcli's own advice (mx check after every exec, mxbuild when packaging) and CI
practice (one full build per push):
- model-stamp records kind: check|full; check --full refuses a check-only stamp
- new pre-push hook runs bin/verify-model.sh when the stamp is check-only, refuses on errors
- module-close.sh runs the full build under Gate mode: check
- verify-model.sh resets the check-only count; backstop MXTK_CHECK_MAX (10) replaces every 5th

Field run (purchase-requests app, Linux, Mendix 11.13.0): an exec under Gate mode: check
wrote kind: check; the following git push ran the full build (20 s) and re-stamped it full.
Fixtures: test-exec-gate-kind.sh 28/28 (K7 stamp kind, K8 pre-push), test-model-stamp 35/35,
test-module-close 7/7, test-exec-step-times 7/7, test-exec-gate-log 24/24.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
feat(doctor, init): measure exec performance; Gate mode: check for ne…
48eb365 
…w projects

Windows field run (ExpenseTracker, Windows 11, Git Bash, 103 ms per process start):
check-only execs 90-103 s vs 138-139 s with the full build; gate 60 s -> 12-14 s.
Doctor smoke: a BUILD-LOG with a 165 s gate row reports WARN with the gate lever; the gk
project (Linux) reports ok, median 19 s. Fixtures: run-tests 33/33, test-doctor-stale-hooks 7/7,
test-test-strategy-docs 15/15, test-affected-tests 138/138.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
fix(exec): review of #587: no gate state outside check mode, refresh …
3a2d20a 
…frees the pipe, profile needs bash 4.1

- gate-kind.state is written only under Gate mode: check, so the pre-push hook and
  module-close full build never fire for projects that did not opt in
- module-close builds only when the stamp says kind: check
- background build-plan refresh sends stdout and stderr to .claude/loop/build-plan-refresh.warn:
  an inherited stderr kept `exec.sh 2>&1 | ...` open until the refresh finished
- MXTK_PROFILE refuses on bash < 4.1 (no BASH_XTRACEFD) with one line
- the per-run mxbuild memo is MXTK_MXBUILD_MEMO, not MXBUILD_PATH (explicit-override
  semantics no longer leak into child scripts)

Fixtures: test-exec-gate-kind 28/28, test-module-close 7/7, test-exec-step-times 7/7,
test-mxbuild-version-match. Field re-run on the purchase-requests app blocked by memory
(mxbuild killed, exit 137, by orphaned deno helpers); to be repeated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
Merge origin/master into perf/windows-exec-speed; credit the CHANGELO…
f164b46 
…G entries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
test(gate-kind): no email literal in fixture git config (leak guard)
a7cb3cf 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
feat(exec): mx check gate is the default; Gate mode: full opts out
71ff2c3 
Maintainer decision 2026-10-29: nobody adds the PROJECT.md line, so the opt-in never
reached the slow machines. The full build still runs before every push (pre-push hook),
at module close, on Java/theme changes, on real drops/renames with Java actions, on the
first exec in a clone and after 10 check-only passes. Fixture K6a/K6a2 (29/29).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
fix(hooks): sync installs the pre-push full-build hook on existing pr…
d3e4127 
…ojects; exit 2 only warns

Review of #587: --check reported a missing pre-push as informational, and sync-project.sh
installs hooks only when --check fails, so existing projects never got the full build
before push. Missing pre-push now fails --check. The hook refuses a push only when the
full build finds errors (verify-model.sh exit 1); exit 2 (mxbuild could not run) warns.
Fixtures: test-exec-gate-kind 32/32 (K8d2, K8g, K8h), test-model-stamp 35/35,
test-restatement-git-hooks 28/28, test-mx-metadata-note 40/40, test-doctor-stale-hooks 7/7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
perf(exec): ~55% fewer forks, ~60% fewer process starts per exec (#584/…
f0aaf6e 
…#585 follow-up)

Field run: the "gk" PurchaseApp project (Purchasing + Approval modules, Mendix
11.13, Gate mode: check), copied to ab-581/ovh, `./bin/exec.sh
mdlsource/9-gk/900-doc.mdl`, Linux, 3 runs before and 3 after. Counted with
`strace -f -e trace=execve,clone,clone3,fork,vfork`:

  forks  733 / 742 / 730  ->  334 / 335 / 338
  execs  350 / 375 / 350  ->  136 / 135 / 135
  MXTK_PROFILE=1 traced commands (exec.sh's own process only)
         1000 / 999 / 1000 -> 914 / 914 / 914
  wall   18.2 / 18.3 / 18.5 s -> 16.6 / 16.6 / 17.1 s (9-10 s of it is mx check,
         3 s mxcli lint, 3 s mxcli report: fixed costs, not process starts)

Output, exit code (4, lint rose) and the BUILD-LOG row are byte-identical
before and after; the model fingerprint is identical to the old script's.

The 40% target holds for process starts, not for the exec.sh-only trace: that
trace counts every statement exec.sh itself runs (conditionals, function bodies,
loops), and a started script is ONE line in it however much it does. Process
starts are what cost ~100 ms on Windows, so those were cut:

- exec.sh: model path, platform, Python, model folder and version resolved once
  and exported (MXTK_MPR_MEMO, MXTK_MODEL_DIR_MEMO, MXTK_PLATFORM, MXTK_PY_MEMO,
  PROJECT_ROOT); EPOCHSECONDS / printf %(..)T / ${x##*/} instead of date, basename,
  dirname; PROJECT.md, CLAUDE.local.md and the gate-kind state read in-shell;
  lint-gate output parsed by one awk (was ~10 grep/sed/tr/sort/paste);
  script-owners greps skipped when it printed nothing.
- _common.sh: unbounded gate waits with `wait`, not a `sleep 1` process per second;
  model version read with one Python start (was two); Studio Pro folder matching
  without basename/cut/sed/grep; find_mpr / find_model_dir / resolve_py honour the
  run memos; sha256 without cut.
- model-stamp.sh: one git rev-parse (was 6), file list in an array (no mktemp,
  paste, rm), stamp read once, `clear` answers before any set-up.
- snapshot-mpr, look-ledger, script-owners, metrics-ledger, row-groups,
  build-plan-status, lint-gate, exec-approval, mdl-scan: dirname / basename / date /
  ls|wc|tr / find|wc forks removed; script-owners record is one awk; lint-gate skips
  its rule-receipt Python while nothing it reads is newer than the last clean check.
- MXTK_PROFILE=2 traces the scripts exec.sh starts too; profile lines name the file.

Fixtures run (scoped, 59, all rc 0): every test-exec-*, test-model-stamp,
test-restatement-guard/-git-hooks, test-look-*, test-lint-*, test-sec014-notice,
test-metrics-ledger, test-row-groups, test-mdl-scan, test-common-*,
test-find-java-version, test-mxbuild-version-match, test-shared-lock-takeover,
test-describe-baseline, test-dryrun-writes-nothing, test-wrong-verdicts,
test-build-plan-*, test-progress-report*, test-module-close, test-conformance-*,
test-change-log, test-doctor-gate-selftest, test-stack-up-local and others.
check-scripts.sh and check-portability.sh clean.

Reported by mkr87.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
Merge remote-tracking branch 'origin/perf/windows-exec-speed' into pe…
7f36ebe 
…rf/exec-overhead

# Conflicts:
#	CHANGELOG.md
#	project-bin/exec.sh
@claude
docs(CHANGELOG): drop the pre-default duplicates the merge kept; cred…
d89de4d 
…it the overhead entry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
fix(lint-gate): the receipt-check skip keys on content, not mtimes (#590
362b4d2 
 review)

An overwrite that keeps an old mtime (cp -p, rsync -t, tar/unzip) let a re-seeded rule skip
the receipt check. The skip marker now holds a cksum signature of the receipt, disabled and
stock lists and every rule file. Fixture T1b in test-lint-receipt-revert.sh (82/82; the
old script fails it, 81/1). test-lint-gate-accept 24/24, -new-findings 15/15,
test-exec-lint-diff green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@claude
test(lint-receipt): T1b rewrites in place and keeps the folder mtime,…
2e99ddf 
… so it fails on the old skip

Re-review of #590: strip_header renamed the file, which bumped the folder mtime and let the
old mtime check pass T1b. Now 82/82 on the new script, T1b FAILs on 362b4d2^.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@MendixMau
MendixMau changed the base branch from perf/windows-exec-speed to master October 29, 2026 12:17
@claude
Merge remote-tracking branch 'origin/master' into perf/exec-overhead
16d6b1f 
# Conflicts:
#	CHANGELOG.md
#	bin/doctor.sh
#	project-bin/_common.sh
#	project-bin/exec.sh
#	project-bin/model-stamp.sh
@MendixMau
MendixMau merged commit f676f60 into master Oct 29, 2026
2 checks passed
This was referenced Oct 29, 2026
Open
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MendixMau @claude