From 9b2d7d0b2be83f60851f38d06136aef1df8da5ff Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:10:49 +0000 Subject: [PATCH 01/61] fix(test): a later passing @expect no longer clears an earlier failure (ako/mxcli#1753) Symptom, from the issue: "On the after-startup test runner, a test with more than one `@expect` is reported **PASS** when an earlier assertion fails and a later one passes. `mxcli test` exits 0." Affected: every Docker run and `--local --legacy-runner`; the endpoint runner is not. Cause: writeExpectAssertion logs one verdict line per assertion, MXTEST:PASS: or MXTEST:FAIL::..., and ParseLogResults let every line overwrite the test's status. The last line won, and the PASS case also cleared the failure message. The generated flow returned $AllPassed = false, but nothing reads it. Fix: the parser folds a test's verdict lines with a sticky failure (applyVerdictLine). Once a test has a FAIL or an ERROR, no later PASS or SKIP line clears it. The first failure's message is reported, with "(and N more failure(s) in this test)" when more followed. This matches the endpoint runner, which evaluates an assertion only while $Verdict is PASS and whose skipped-assertions guard never replaces a failure. The skipped-assertions FAIL (#1448), logged after the PASS lines of the assertions that ran, still wins over them: a sticky failure, not "first line wins". A RUN line resets the fold, so a failure cannot outlive its run. The fix is in the parser, so it also covers logs written by runners generated before it. The generator is unchanged: logging one verdict per test would change a flow whose verdicts were measured on 11.13.0, and it would cover nothing more, because parser and generator ship together. Measurement (unit level, the symptom's layer: the parser reads a log). sticky_failure_test.go, run before the fix existed and again with results.go reverted: --- FAIL: TestLaterPassDoesNotClearEarlierFailure FAIL then PASS: status = PASS, want FAIL FAIL then PASS: message = "", want "Expected $r = 1, actual: 2" FAIL then PASS: the suite passed, so `mxcli test` exits 0 --- FAIL: TestFirstFailureMessageIsReported message = "Expected $r = 4, actual: 2", want "Expected $r = 1, actual: 2 (and 2 more failures in this test)" --- FAIL: TestLaterPassDoesNotClearEarlierError ERROR then PASS:test_1: got PASS "", want ERROR "Setup failed: Shop.Seed" TestSkippedAssertionsFailureStillWins and the PASS-then-FAIL and PASS-then-PASS cases pass both before and after; they are the controls. With the RUN reset stubbed out, TestRunLineStartsVerdictAfresh fails. mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl is the runtime reproduction. It was not run against an app for this change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...later-expect-pass-clears-earlier-fail.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/testrunner/results.go | 125 ++++++++-------- cmd/mxcli/testrunner/sticky_failure_test.go | 135 ++++++++++++++++++ docs-site/src/tools/test-annotations.md | 4 + ...-runner-1753-earlier-expect-fails.test.mdl | 55 +++++++ 6 files changed, 258 insertions(+), 63 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json create mode 100644 cmd/mxcli/testrunner/sticky_failure_test.go create mode 100644 mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json new file mode 100644 index 000000000..42148292a --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "On the after-startup runner (every Docker run, and `--local --legacy-runner`) a test with more than one `@expect` is reported PASS when an earlier assertion fails and a later one passes, and `mxcli test` exits 0. Measured at unit level: log FAIL:test_1:Expected $r = 1, actual: 2 then PASS:test_1 gave status=PASS message=\"\" AllPassed=true. PASS then FAIL reported correctly; the endpoint runner (`--local`, `--attach`, `--watch`) was never affected", "cause": "`writeExpectAssertion` logs a verdict line per assertion (MXTEST:PASS or MXTEST:FAIL), and `ParseLogResults` treated every line as the test's verdict: each case overwrote Status, and the PASS case also cleared Message, so the last line won. The generated flow did compute `$AllPassed = false` and return it — nothing reads that value", "file": "`cmd/mxcli/testrunner/results.go` (`ParseLogResults`, new `applyVerdictLine`)", "insight": "A log protocol that emits several partial verdicts per test needs a monotonic fold, not last-write-wins: FAIL/ERROR is sticky, the first failure's message is reported (the endpoint runner's semantics — its assertions run only while $Verdict is PASS and its skipped-assertions guard never replaces a failure), and further failures are counted onto the message. The plausible wrong turn is 'first verdict line wins': the skipped-assertions guard (#1448) logs its FAIL AFTER the PASS lines of the assertions that did run, so first-wins turns that false green back on — sticky failure, not first line, is the rule. Fixed in the parser rather than the generator: it also corrects logs from runners generated before the fix, and a one-verdict-per-test generator would change a flow shape whose verdicts were measured on 11.13.0 for no additional coverage, since parser and generator ship in one binary. A RUN line resets the fold so a failure cannot outlive the run it was logged in. Second time this parser took a verdict from the wrong line (after #1448's RUN-without-verdict read as the zero status, PASS): when touching `ParseLogResults`, enumerate the line ORDERS the generator can produce per test, not just the line kinds", "refs": ["ako/mxcli#1753", "ako/mxcli#1448"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 0808e4c63..03eb4926e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. +- **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/cmd/mxcli/testrunner/results.go b/cmd/mxcli/testrunner/results.go index 905b232d9..593574229 100644 --- a/cmd/mxcli/testrunner/results.go +++ b/cmd/mxcli/testrunner/results.go @@ -189,6 +189,8 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo // Track which tests we've seen results for resultMap := make(map[string]*TestResult) runTimes := make(map[string]time.Time) + // failures counts each test's FAIL and ERROR lines; see applyVerdictLine. + failures := make(map[string]int) scanner := bufio.NewScanner(logReader) for scanner.Scan() { @@ -233,6 +235,9 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo // verdict did not finish: an error left the runner flow, which // stops it here — a call whose own handler re-raises does exactly // that (ako/mxcli#1448). Its PASS or FAIL line replaces this. + // A RUN line starts the test's verdict afresh, so a failure that + // sticks (applyVerdictLine) cannot outlive the run it belongs to. + delete(failures, id) resultMap[id] = &TestResult{ ID: id, Name: name, @@ -240,77 +245,25 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo Message: "started and reported no verdict: an error left the test runner here, which stopped it (the runtime log has the error)", } - case "PASS": - if r, ok := resultMap[id]; ok { - r.Status = StatusPass - r.Message = "" - if t, ok := runTimes[id]; ok { - r.Duration = time.Since(t) - } - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusPass, - } - } - - case "FAIL": + case "PASS", "FAIL", "ERROR", "SKIP": + // ERROR is a setup that threw: the test never ran, so it is neither a + // pass nor a failure. Only the monolithic runner reports this way — + // the endpoint carries the same outcome back as a SETUP verdict. msg := "" if len(parts) >= 4 { msg = parts[3] } - if r, ok := resultMap[id]; ok { - r.Status = StatusFail - r.Message = msg + r, ok := resultMap[id] + if !ok { + r = &TestResult{ID: id, Name: id} + resultMap[id] = r + } + if action == "PASS" || action == "FAIL" { if t, ok := runTimes[id]; ok { r.Duration = time.Since(t) } - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusFail, - Message: msg, - } - } - - case "ERROR": - // A setup that threw: the test never ran, so it is neither a pass nor - // a failure. Only the monolithic runner reports this way — the - // endpoint carries the same outcome back as a SETUP verdict. - msg := "" - if len(parts) >= 4 { - msg = parts[3] - } - if r, ok := resultMap[id]; ok { - r.Status = StatusError - r.Message = msg - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusError, - Message: msg, - } - } - - case "SKIP": - msg := "" - if len(parts) >= 4 { - msg = parts[3] - } - if r, ok := resultMap[id]; ok { - r.Status = StatusSkip - r.Message = msg - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusSkip, - Message: msg, - } } + applyVerdictLine(r, verdictStatus[action], msg, failures) case "END": result.Duration = time.Since(result.Started) @@ -337,6 +290,13 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo } res.Status = r.Status res.Message = r.Message + if more := failures[tc.ID] - 1; more > 0 { + noun := "failures" + if more == 1 { + noun = "failure" + } + res.Message += fmt.Sprintf(" (and %d more %s in this test)", more, noun) + } res.Duration = r.Duration result.Tests = append(result.Tests, res) } else { @@ -351,6 +311,45 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo return result } +// verdictStatus maps a log protocol verdict action to the status it reports. +var verdictStatus = map[string]TestStatus{ + "PASS": StatusPass, + "FAIL": StatusFail, + "ERROR": StatusError, + "SKIP": StatusSkip, +} + +// applyVerdictLine records one verdict line of the after-startup runner's log +// against a test's result. failures counts each test's FAIL and ERROR lines. +// +// A test can log more than one verdict line: writeExpectAssertion logs PASS or +// FAIL for every @expect on its own, and the skipped-assertions guard logs a +// FAIL after them. A failure is therefore sticky — once a test has a FAIL or an +// ERROR, no later line clears it, and the first failure's message is the one +// reported. When every line simply replaced the last, a test whose first +// assertion failed and whose second held reported PASS with the failure message +// erased, and `mxcli test` exited 0 (ako/mxcli#1753). The rule lives here rather +// than in the generated flow so it also holds for a log written by a runner +// generated before it. +// +// This is the endpoint runner's semantics: there an assertion is evaluated only +// while $Verdict is still PASS, and the skipped-assertions guard does not +// replace an existing failure, so its first failure is what it returns. +// +// The ERROR a RUN line leaves as a placeholder is not a failure line; the +// test's own verdict replaces it. +func applyVerdictLine(r *TestResult, status TestStatus, msg string, failures map[string]int) { + failed := status == StatusFail || status == StatusError + if failed { + failures[r.ID]++ + } + if failures[r.ID] > 1 || (!failed && failures[r.ID] > 0) { + return + } + r.Status = status + r.Message = msg +} + // PrintResults writes a human-readable summary to the writer. func PrintResults(w io.Writer, result *SuiteResult, color bool) { fmt.Fprintf(w, "\n") diff --git a/cmd/mxcli/testrunner/sticky_failure_test.go b/cmd/mxcli/testrunner/sticky_failure_test.go new file mode 100644 index 000000000..993c94a1f --- /dev/null +++ b/cmd/mxcli/testrunner/sticky_failure_test.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" +) + +// twoExpectSuite is one test with two @expect lines, the shape the after-startup +// runner reports as two verdict lines: writeExpectAssertion logs PASS or FAIL +// for each assertion on its own. +func twoExpectSuite(t *testing.T) *TestSuite { + t.Helper() + e1, err := ParseExpect("$r = 1") + if err != nil { + t.Fatalf("ParseExpect: %v", err) + } + e2, err := ParseExpect("$r = 2") + if err != nil { + t.Fatalf("ParseExpect: %v", err) + } + tc := TestCase{ID: "test_1", Name: "two expects", MDL: "$r = 2;", Expects: []Expect{e1, e2}} + return &TestSuite{Name: "S", Tests: []TestCase{tc}} +} + +// verdictLog is a runner log for test_1 with the given verdict lines between +// its RUN line and the suite's END. +func verdictLog(lines ...string) string { + var b strings.Builder + b.WriteString("MXTEST: MXTEST:START:S\n") + b.WriteString("MXTEST: MXTEST:RUN:test_1:two expects\n") + for _, l := range lines { + b.WriteString("MXTEST: MXTEST:" + l + "\n") + } + b.WriteString("MXTEST: MXTEST:END:S\n") + return b.String() +} + +// TestLaterPassDoesNotClearEarlierFailure is ako/mxcli#1753: on the after-startup +// runner a test with more than one @expect reported PASS when an earlier +// assertion failed and a later one held, and `mxcli test` exited 0. Each +// assertion logs its own verdict line, and the parser let every line overwrite +// the test's status — the last one won, and a PASS also cleared the failure +// message. The endpoint runner evaluates an assertion only while the verdict is +// still PASS, so there the first failure sticks; the log parser now does the +// same. +func TestLaterPassDoesNotClearEarlierFailure(t *testing.T) { + const want = "Expected $r = 1, actual: 2" + + got := ParseLogResults(strings.NewReader(verdictLog("FAIL:test_1:"+want, "PASS:test_1")), + twoExpectSuite(t), false) + r := got.Tests[0] + if r.Status != StatusFail { + t.Errorf("FAIL then PASS: status = %v, want FAIL", r.Status) + } + if r.Message != want { + t.Errorf("FAIL then PASS: message = %q, want %q", r.Message, want) + } + if got.AllPassed() { + t.Error("FAIL then PASS: the suite passed, so `mxcli test` exits 0") + } + + // Control: the order the old parser already got right. + got = ParseLogResults(strings.NewReader(verdictLog("PASS:test_1", "FAIL:test_1:"+want)), + twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusFail || r.Message != want { + t.Errorf("control, PASS then FAIL: got %v %q, want FAIL %q", r.Status, r.Message, want) + } + + // Control: every assertion held. + got = ParseLogResults(strings.NewReader(verdictLog("PASS:test_1", "PASS:test_1")), + twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusPass || r.Message != "" { + t.Errorf("control, PASS then PASS: got %v %q, want PASS", r.Status, r.Message) + } +} + +// TestFirstFailureMessageIsReported pins which message a test that failed more +// than once reports: the first, as the endpoint runner's $Verdict does, with a +// count of the rest so the reader knows the log holds more than the summary. +func TestFirstFailureMessageIsReported(t *testing.T) { + got := ParseLogResults(strings.NewReader(verdictLog( + "FAIL:test_1:Expected $r = 1, actual: 2", + "FAIL:test_1:Expected $r = 3, actual: 2", + "PASS:test_1", + "FAIL:test_1:Expected $r = 4, actual: 2", + )), twoExpectSuite(t), false) + r := got.Tests[0] + if r.Status != StatusFail { + t.Fatalf("status = %v, want FAIL", r.Status) + } + const want = "Expected $r = 1, actual: 2 (and 2 more failures in this test)" + if r.Message != want { + t.Errorf("message = %q, want %q", r.Message, want) + } +} + +// TestSkippedAssertionsFailureStillWins pins the guard writeTestBlock emits after +// the assertions: when a call's own handler resumed the flow past some of them, +// the assertions that did run log PASS and the guard logs a FAIL after them +// (ako/mxcli#1448). That FAIL is the first failure, so it is the verdict. +func TestSkippedAssertionsFailureStillWins(t *testing.T) { + got := ParseLogResults(strings.NewReader(verdictLog( + "PASS:test_1", + "FAIL:test_1:"+assertionsSkippedReason, + )), twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusFail || r.Message != assertionsSkippedReason { + t.Errorf("got %v %q, want FAIL with the skipped-assertions reason", r.Status, r.Message) + } +} + +// TestLaterPassDoesNotClearEarlierError: a setup that threw logs ERROR, and no +// verdict line after it may turn the test into a pass or a skip either. +func TestLaterPassDoesNotClearEarlierError(t *testing.T) { + const msg = "Setup failed: Shop.Seed" + for _, later := range []string{"PASS:test_1", "SKIP:test_1:skipped"} { + got := ParseLogResults(strings.NewReader(verdictLog("ERROR:test_1:"+msg, later)), + twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusError || r.Message != msg { + t.Errorf("ERROR then %s: got %v %q, want ERROR %q", later, r.Status, r.Message, msg) + } + } +} + +// TestRunLineStartsVerdictAfresh: a failure sticks for the run it was logged in, +// not for the rest of the log. A RUN line begins a test's verdict, so a log that +// holds the same test twice reports the later run. +func TestRunLineStartsVerdictAfresh(t *testing.T) { + log := verdictLog("FAIL:test_1:Expected $r = 1, actual: 2", "RUN:test_1:two expects", "PASS:test_1", "PASS:test_1") + got := ParseLogResults(strings.NewReader(log), twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusPass { + t.Errorf("got %v %q, want the second run's PASS", r.Status, r.Message) + } +} diff --git a/docs-site/src/tools/test-annotations.md b/docs-site/src/tools/test-annotations.md index 4cd744b27..65f045448 100644 --- a/docs-site/src/tools/test-annotations.md +++ b/docs-site/src/tools/test-annotations.md @@ -77,6 +77,10 @@ FAIL the board is 81 squares expected length($result) = 81, actual: 27 ``` +A test with several `@expect` lines passes only when every one of them holds, and +it reports the first that does not. The Docker / `--legacy-runner` path evaluates +them all, and adds how many more failed — `(and 1 more failure in this test)`. + `@expect` cannot be combined with `@throws`: a body expected to fail produces no result to assert on, so the combination is refused rather than ignored. diff --git a/mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl b/mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl new file mode 100644 index 000000000..02f7ab684 --- /dev/null +++ b/mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl @@ -0,0 +1,55 @@ +-- ako/mxcli#1753: on the after-startup runner (every Docker run, and --local +-- --legacy-runner) a test with more than one @expect was reported PASS when an +-- earlier assertion failed and a later one held, and `mxcli test` exited 0. +-- Each assertion logs its own PASS or FAIL line, and the log parser let the +-- last line be the test's verdict. A failure now sticks, and the first one is +-- the message reported. +-- +-- Setup: test-runner-1514-setup.mdl (Probe.SUB_Make returns 'a;b'). Then: +-- +-- mxcli test mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl -p app.mpr --local --legacy-runner +-- +-- Expected: test 1 FAIL "Expected $a = 'zzz', actual: a;b" (before the fix: +-- PASS), test 2 FAIL with the same message for $b (the control — the order the +-- parser always got right), test 3 FAIL "Expected $c = 'zzz', actual: a;b (and +-- 1 more failure in this test)", test 4 PASS. The endpoint runner (--local +-- alone) evaluates an assertion only while the test is still passing, so it +-- reported test 1 correctly before the fix too. +-- +-- The log parser's verdicts are pinned by cmd/mxcli/testrunner/ +-- sticky_failure_test.go; this file is the runtime reproduction and was not run +-- against an app for the fix. Each test uses its own variable so the file does +-- not depend on the runner renaming a shared one. +mdl 1; +/** + * @test 1 earlier assertion fails, later one holds + * @expect $a = 'zzz' + * @expect $a = 'a;b' + */ +$a = call microflow Probe.SUB_Make(); +/ + +/** + * @test 2 earlier assertion holds, later one fails + * @expect $b = 'a;b' + * @expect $b = 'zzz' + */ +$b = call microflow Probe.SUB_Make(); +/ + +/** + * @test 3 two assertions fail, one holds + * @expect $c = 'zzz' + * @expect $c = 'a;b' + * @expect $c = 'yyy' + */ +$c = call microflow Probe.SUB_Make(); +/ + +/** + * @test 4 every assertion holds + * @expect $d = 'a;b' + * @expect length($d) = 3 + */ +$d = call microflow Probe.SUB_Make(); +/ From be9dc02389364c7a6969bb3ba361ce1b2d2e9fac Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:15:45 +0000 Subject: [PATCH 02/61] fix(test): rename a lower-case `call`/`create` variable per test on the after-startup runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom, from ako/mxcli#1753's sibling note: "on this runner, `assignPattern` matches only upper-case `CALL|CREATE`. So mdl 1's `$r = call …` is not renamed per test, and two tests that both use `$r` fail injection (MDL063 / CE0111, measured)." Affected: every Docker run and `--local --legacy-runner`. The endpoint runner gives each test its own flow and does not rename. Cause: the legacy runner compiles every test into one microflow, so it gives each test's variables a `_N` suffix. assignPattern decides which variables are renamed, and it matched only `CALL|CREATE` in upper case. MDL keywords are case-insensitive, and mdl 1 and `fmt` write them in lower case, so `$r = call microflow …` stayed `$r` in every test. Fix: assignPattern matches `call|create` in any case, followed by a word boundary so `$r = callback` does not count. No other pattern in cmd/mxcli/testrunner makes the upper-case assumption: the rest lower-case their input or use EqualFold. TestThrowsCatchesEveryCallKind and TestVerdictCatchesEveryCallKind matched the legacy output against the unrenamed lower-case statement, so they passed only because of the defect. Their legacy assertions now expect the renamed statement (legacyStmt). The 1448 bug-test's note that the legacy runner "needs a distinct variable name per test" is updated. Measurement. rename_keyword_case_test.go with the old pattern (the upper-case control subtest passes): --- FAIL: TestLegacyRunnerRenamesLowerCaseAssignments/call_microflow,_mdl_1 test 1: no "$r_1 = call microflow" — the assignment was not renamed an assertion still reads the unrenamed $r MDL063: '$r' is created twice in this microflow — first by call microflow, then by call microflow. ... mxbuild rejects this with CE0111 "Duplicate variable name" (and the same for mdl 0, call java action, create object, create list, mixed case) --- FAIL: TestAssignPatternNeedsAWholeKeyword control: $r = call microflow was not picked up: map[] I also swept every non-.fail `.test.mdl` in mdl-examples/ (16 files) through GenerateTestRunner and executor.ValidateProgram. Before the fix, 10 files drew MDL063, among them the 1448, 1514 and 923 bug-tests and doctype-tests/microflow-spec (86). After it, 2 do. Those 2 come from `retrieve $Brands` and from `retrieve $Request` / `$Request = head(…)`. They are a separate gap: assignPattern has never matched `retrieve`, `declare`, list operations, `send rest request` or `execute database query` in any case. This change does not address it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-lower-case-call-variable-not-renamed.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/testrunner/generator.go | 10 +- .../testrunner/on_error_coverage_test.go | 13 ++- .../testrunner/rename_keyword_case_test.go | 94 +++++++++++++++++++ ...est-runner-1448-own-error-handler.test.mdl | 7 +- 6 files changed, 120 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json create mode 100644 cmd/mxcli/testrunner/rename_keyword_case_test.go diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json new file mode 100644 index 000000000..cd1b37502 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "On the after-startup runner (Docker, `--local --legacy-runner`) two tests in one file that both assign the same variable with a lower-case keyword — `$r = call microflow …`, mdl 1's and `fmt`'s spelling — fail injection with MDL063 (CE0111 \"Duplicate variable name\" from mxbuild); the same file runs under the endpoint runner, which gives each test its own flow", "cause": "The legacy runner compiles every test into one microflow and renames each test's variables with a `_N` suffix, but which variables to rename was decided by `assignPattern`, which matched only upper-case `CALL|CREATE` — the spelling of the runner's own early examples. A lower-case assignment was left unrenamed and declared once per test", "file": "`cmd/mxcli/testrunner/generator.go` (`assignPattern`, now `(?mi)…(?:call|create)\\b`)", "insight": "The cheapest detector was a sweep, not a reading: every `.test.mdl` in the repo through `GenerateTestRunner` and `executor.ValidateProgram` — 10 of the 16 in mdl-examples/ drew MDL063 before the fix, 2 after. Those 2 (`retrieve $Brands`, `retrieve $Request` + `$Request = head(…)`) are the remaining, separate gap: `assignPattern` never matched `retrieve $x`, `declare $x`, list operations (`$x = filter(…)`), `send rest request` or `execute database query` in ANY case, so a variable those define in two tests still collides; the durable fix is to take the declared names from the parsed body (`visitor.FlowStatements`, as `attachOnError` already does), not a regex. Two existing tests (`TestThrowsCatchesEveryCallKind`, `TestVerdictCatchesEveryCallKind`) matched the legacy output against the unrenamed lower-case statement and passed only because of the defect — a test that asserts a generator's text verbatim can encode the bug it should catch. The `\\b` matters once the match is case-insensitive (`$r = callback`). Found alongside ako/mxcli#1753, whose bug-test therefore uses a distinct variable per test", "refs": ["ako/mxcli#1753"], "rules": ["MDL063"], "ce": ["CE0111"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 03eb4926e..093bf72ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. +- **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/cmd/mxcli/testrunner/generator.go b/cmd/mxcli/testrunner/generator.go index 674b81b3d..0e347a7f9 100644 --- a/cmd/mxcli/testrunner/generator.go +++ b/cmd/mxcli/testrunner/generator.go @@ -273,9 +273,15 @@ func writeExpectAssertion(b *strings.Builder, testID string, exp Expect, evaluat // varPattern matches $VariableName in MDL ($ followed by word characters). var varPattern = regexp.MustCompile(`\$([A-Za-z_][A-Za-z0-9_]*)`) -// assignPattern matches "$var = CALL" or "$var = CREATE" at the start of any +// assignPattern matches "$var = call …" or "$var = create …" at the start of any // statement line inside the test body. -var assignPattern = regexp.MustCompile(`(?m)^\s*\$([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(?:CALL|CREATE)`) +// +// The keywords match in any case. MDL keywords are case-insensitive, and mdl 1 +// and `fmt` write them in lower case; matching only `CALL|CREATE` left +// `$r = call microflow …` unrenamed, so two tests that both assigned `$r` declared +// it twice in the one runner flow and injection failed with MDL063 (CE0111). +// The `\b` keeps a value that merely starts with those letters from matching. +var assignPattern = regexp.MustCompile(`(?mi)^\s*\$([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(?:call|create)\b`) // extractVariableNames finds all user-defined variable names in the MDL body. // Returns the set of variable names (without $ prefix). diff --git a/cmd/mxcli/testrunner/on_error_coverage_test.go b/cmd/mxcli/testrunner/on_error_coverage_test.go index d190814ac..669a89858 100644 --- a/cmd/mxcli/testrunner/on_error_coverage_test.go +++ b/cmd/mxcli/testrunner/on_error_coverage_test.go @@ -36,7 +36,7 @@ func TestThrowsCatchesEveryCallKind(t *testing.T) { t.Run(c.kind, func(t *testing.T) { tc := parseOne(t, "/**\n * @test it raises\n * @throws 'Only http and https csv urls are supported'\n */\n"+c.stmt+"\n/\n") assertHandled(t, "endpoint runner", GenerateTestFlows(&TestSuite{Name: "s", Tests: []TestCase{tc}}), c.stmt) - assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), c.stmt) + assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), legacyStmt(c.stmt)) }) } } @@ -53,7 +53,7 @@ func TestVerdictCatchesEveryCallKind(t *testing.T) { if !strings.Contains(mdl, verdictFailPrefix+"exception during execution: ") { t.Errorf("no FAIL verdict in the handler:\n%s", mdl) } - assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), c.stmt) + assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), legacyStmt(c.stmt)) }) } } @@ -78,3 +78,12 @@ func assertHandled(t *testing.T, runner, mdl, stmt string) { t.Errorf("%s: generated MDL does not parse: %v\n%s", runner, errs[0], mdl) } } + +// legacyStmt is stmt as the legacy runner writes it for the suite's first test: +// that runner compiles every test into one flow, so it renames the variable a +// call assigns with the test's suffix. These statements used to be matched +// unrenamed, which held only because a lower-case `call` was not renamed +// (TestLegacyRunnerRenamesLowerCaseAssignments). +func legacyStmt(stmt string) string { + return renameVariables(stmt, extractVariableNames(stmt), "_1") +} diff --git a/cmd/mxcli/testrunner/rename_keyword_case_test.go b/cmd/mxcli/testrunner/rename_keyword_case_test.go new file mode 100644 index 000000000..ef6384724 --- /dev/null +++ b/cmd/mxcli/testrunner/rename_keyword_case_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// TestLegacyRunnerRenamesLowerCaseAssignments: the after-startup runner compiles +// every test into one microflow, so a variable two tests both assign has to be +// renamed per test. Which variables are renamed was decided by a pattern that +// matched only upper-case `CALL` and `CREATE` — the spelling the runner's own +// examples used — so `$r = call microflow …`, how mdl 1 and `fmt` spell it, was +// not renamed. Two tests that both assign `$r` then declare it twice, and +// injection fails with MDL063 (CE0111 "Duplicate variable name" from mxbuild). +// Found alongside ako/mxcli#1753. +func TestLegacyRunnerRenamesLowerCaseAssignments(t *testing.T) { + cases := []struct { + name, header, body, renamed string + }{ + {"call microflow, mdl 1", "mdl 1;\n", + "$r = call microflow Probe.SUB_Make();", "$r_%d = call microflow"}, + {"call microflow, mdl 0", "", + "$r = call microflow Probe.SUB_Make();", "$r_%d = call microflow"}, + {"call java action", "mdl 1;\n", + "$r = call java action Probe.JA_Upper(Text = 'a');", "$r_%d = call java action"}, + {"create object", "mdl 1;\n", + "$r = create Probe.Item (Name = 'a');", "$r_%d = create Probe.Item"}, + {"create list", "mdl 1;\n", + "$r = create list of Probe.Item;", "$r_%d = create list of"}, + {"mixed case", "mdl 1;\n", + "$r = Call Microflow Probe.SUB_Make();", "$r_%d = Call Microflow"}, + // Control: the spelling the pattern always matched. + {"upper case (control)", "", + "$r = CALL MICROFLOW Probe.SUB_Make();", "$r_%d = CALL MICROFLOW"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + block := func(n string) string { + return "/**\n * @test " + n + "\n * @expect $r != empty\n */\n" + c.body + "\n/\n" + } + tests, err := parseMDLTests(c.header+block("one")+"\n"+block("two"), "rename.test.mdl") + if err != nil { + t.Fatalf("parse: %v", err) + } + src := GenerateTestRunner(&TestSuite{Name: "s", Tests: tests}) + + for i, want := range []string{ + strings.Replace(c.renamed, "%d", "1", 1), + strings.Replace(c.renamed, "%d", "2", 1), + } { + if !strings.Contains(src, want) { + t.Errorf("test %d: no %q — the assignment was not renamed", i+1, want) + } + } + if strings.Contains(src, "($r != empty)") { + t.Errorf("an assertion still reads the unrenamed $r") + } + + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("generated runner does not parse: %v\n%s", errs[0], src) + } + for _, v := range executor.ValidateProgram(prog, "") { + if v.RuleID == "MDL063" { + t.Errorf("%s: %s", v.RuleID, v.Message) + } + } + if t.Failed() { + t.Logf("generated runner:\n%s", src) + } + }) + } +} + +// TestAssignPatternNeedsAWholeKeyword: matching case-insensitively must not turn +// an assignment whose value merely starts with the letters of a keyword into one. +func TestAssignPatternNeedsAWholeKeyword(t *testing.T) { + for _, body := range []string{ + "$r = callback;", + "$r = createdDate;", + } { + if names := extractVariableNames(body); len(names) != 0 { + t.Errorf("%q: renamed %v, want nothing", body, names) + } + } + if names := extractVariableNames("$r = call microflow M.F();"); !names["r"] { + t.Errorf("control: $r = call microflow was not picked up: %v", names) + } +} diff --git a/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl b/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl index 7bf7545c7..1987183ff 100644 --- a/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl +++ b/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl @@ -31,8 +31,11 @@ -- -- Under --legacy-runner a test whose error leaves the runner (05, 06, 11) ends -- the whole run; 05 was reported PASS there (a RUN line with no verdict read as --- the zero status) and is now an ERROR. That runner also needs a distinct --- variable name per test. +-- the zero status) and is now an ERROR. That runner also needed a distinct +-- variable name per test: it renamed a variable per test only after an +-- upper-case `CALL`, so this file's `$r` and `$Result` drew 14 MDL063 +-- duplicate-variable errors there. It renames one after a lower-case `call` too +-- since the sibling fix to ako/mxcli#1753. mdl 1; /** * @test 01 plain call that throws From 2f9b6260c78b3ab6b1bdadcefd4d422308dbf62a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:50:32 +0000 Subject: [PATCH 03/61] fix(security): keep a module role's description when create or modify omits it Symptom (ako/mxcli#1218): a bare `create or modify module role M.R;` reported "Modified module role: M.R" and erased the role's stored description. `describe` -> `exec` was unaffected because describe prints the description; grants were untouched. Cause: CreateModuleRoleStmt.Description was a plain string, so "no description clause" and `description ''` were the same value, and the OR MODIFY branch of execCreateModuleRole passed it to AddModuleRole, which overwrites the existing role in place. Fix: Description is a *string, nil when the statement has no clause, as CreateUserRoleStmt's properties have been since ako/mxcli#707. OR MODIFY writes the stored description when it is nil, so the write is elided and the run reports "Unchanged"; `description ''` clears it. Adopting an auto-provisioned role still drops its marker description. Docs (skill, docs-site reference, `mxcli syntax`) state the rule. Measurement: - Unit test (mdl/executor/issue1218_module_role_description_test.go) fails before the fix, and with the fix forced back, with the reported symptom: description written = "", want "Read the published OData services. Nothing else." / Modified module role: Shop.ApiUser. - Real app, Mendix 11.13.0: role created with a description; bare statement with the fixed build -> "Unchanged module role", describe keeps the description, describe -> exec "Unchanged"; with the pre-fix build -> "Modified module role" and the description is empty. `description ''` with the fixed build clears it. mx check: 0 errors in every state, the same as the blank app, fault included. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...e-create-or-modify-erases-description.json | 1 + .../skills/mendix/manage-security/SKILL.md | 12 ++- CHANGELOG.md | 1 + cmd/mxcli/syntax/features_security.go | 2 +- .../reference/security/create-module-role.md | 12 ++- .../1218-module-role-description-kept.mdl | 34 +++++++ mdl/ast/ast_security.go | 15 ++- mdl/executor/cmd_security_write.go | 16 +++- .../issue1218_module_role_description_test.go | 94 +++++++++++++++++++ .../issue707_describe_roundtrip_test.go | 4 +- .../module_role_create_statement_test.go | 2 +- mdl/visitor/visitor_security.go | 5 +- .../visitor_security_or_modify_test.go | 4 +- mdl/visitor/visitor_security_test.go | 25 ++++- 14 files changed, 203 insertions(+), 24 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json create mode 100644 mdl-examples/bug-tests/1218-module-role-description-kept.mdl create mode 100644 mdl/executor/issue1218_module_role_description_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json new file mode 100644 index 000000000..cae865287 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-28","symptom":"A bare `create or modify module role M.R;` (no description clause) reported `Modified module role: M.R` and erased the role's stored Description; `describe` → `exec` was unaffected because describe prints the description, and grants (on the entity, keyed by role name) were untouched.","cause":"CreateModuleRoleStmt.Description was a plain string, so 'no clause' and `description ''` were the same value, and execCreateModuleRole's OR MODIFY branch passed it to Backend.AddModuleRole, which overwrites the existing role in place.","file":"mdl/ast/ast_security.go (Description *string), mdl/visitor/visitor_security.go, mdl/executor/cmd_security_write.go (execCreateModuleRole)","fix":"Description is a *string, nil when unstated, as CreateUserRoleStmt's properties have been since ako/mxcli#707; OR MODIFY writes the stored description when it is nil, so the write is elided and the run reports Unchanged. `description ''` clears it. Adopting an auto-provisioned role still drops its marker description.","insight":"A neighbouring statement already had the right shape: the user-role code even said 'the rest keep their stored values, as a module role's missing description does', which was false — a comment asserting a sibling's behaviour is a claim to test, not evidence. The tell for this class is an optional clause parsed into a value type: grep the AST for optional clauses stored as string/bool without a *T or a ...Set flag, then follow each to an OR MODIFY branch. mx check is 0 errors with the fault (measured, 11.13.0), and describe → exec cannot show it because describe restates the value; only a hand-written minimal statement does.","refs":["ako/mxcli#1218"],"test":"mdl/executor/issue1218_module_role_description_test.go, mdl/visitor/visitor_security_test.go"} diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index 9f294a433..e75335f11 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -13,11 +13,11 @@ description: "Configure Mendix security in MDL — module and user roles, access | 27-35 | When to Use This Skill | | 36-43 | Security Concepts | | 44-120 | The System-module ceiling — decide the role model around this FIRST | -| 121-558 | Syntax Reference | -| 559-573 | Starlark Lint Rule APIs | -| 574-626 | Common Workflow: Setting Up Module Security | -| 627-636 | Common Mistakes | -| 637-646 | Validation | +| 121-560 | Syntax Reference | +| 561-575 | Starlark Lint Rule APIs | +| 576-628 | Common Workflow: Setting Up Module Security | +| 629-638 | Common Mistakes | +| 639-648 | Validation | # Security Management Skill @@ -205,7 +205,9 @@ create module role MyModule.Viewer description 'Read-only access'; -- `or modify` updates an existing role's description instead of failing, so the -- whole security script stays re-runnable rather than needing a run-once file. +-- Without `description` it keeps the stored one; `description ''` clears it. create or modify module role MyModule.ApiUser description 'API consumer'; +create or modify module role MyModule.Admin; -- ensures it exists, keeps its description -- Remove a module role (`if exists` makes it a no-op when the role is gone) drop module role MyModule.Viewer; diff --git a/CHANGELOG.md b/CHANGELOG.md index 093bf72ec..a947aaac4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **`check -p --references` reports CE1406 for a retrieve, create or create list of a Hidden entity of a Protected module** (ako/mxcli#1699, mendixlabs/mxcli#1652) — a flow in an app module that ran `retrieve $l from AddOn.Internal`, `$o = create AddOn.Internal` or `$l = create list of AddOn.Internal` on a Hidden entity of a Protected (add-on) module passed `check` and `exec`, and mxbuild refused it: `[CE1406] "Retrieve object(s) activity 'Retrieve list of Internal from database' is not allowed to use entity 'AddOn.Internal' because app modules cannot reference hidden elements in add-on modules."` (likewise "Create object activity", "Create list activity"). The check judged an entity only as a parameter or return type; it now judges the entity these three activities name, in microflows and nanoflows, and `exec` refuses the script before writing it. Measured on Mendix 11.13.0: each shape refused against a Hidden entity, 0 errors against a Usable one and inside the add-on; an association retrieve that reaches a Hidden entity over a Usable association names only the association and is not refused. - **A pluggable widget's text-template parameter across an association builds, and describes so it re-executes** (mendixlabs/mxcli#1494) — `fieldset fs (legend: '{1}', legendParams: ({1} = M.Parent_Child/Label))` passed `check` and `exec` and failed the build with `[CE1365] "Attribute 'M.Child.Label' is not an attribute of entity 'M.Parent'."`: the parameter was written with its attribute but without the association hop, so mxbuild read the attribute against the widget's own entity. The same writer serves an object-list item's template (an Accordion group's `headerText`) and a Data grid 2 column's dynamic text, and all three failed; on Mendix 11.12.4 they now build with 0 errors. `describe page` printed such a parameter as the bare attribute (`{1} = Label`), which re-executed into CE1613 "The selected attribute 'M.Parent.Label' no longer exists"; it now prints `M.Parent_Child/Label`. The companion report, an expression parameter (`{1} = if 1 = 1 then 'A' else 'B'`) stored as an attribute name (mendixlabs/mxcli#1495), was fixed in v0.25.0 (#823) and is now pinned by a test. - **`alter entity … add event handler` adds a second handler for the same event** (mendixlabs/mxcli#1540) — `add event handler on before commit call M.Second($currentObject)` failed with "event handler already exists for Before Commit" when the entity already had one, though Studio Pro allows several handlers for one event and `create or modify entity` with two `on before commit` clauses stores both (measured on 11.12.4: two Before Commit handlers are 0 errors, a void-returning one is CE7247, so the check does look at them). A handler is now identified by its moment, event and microflow: another microflow adds a second handler, the same microflow again is still a duplicate. The same key had made `add event handler if not exists … call M.New` skip — reporting "already exists" — whenever any handler for the event existed, and `drop event handler on before commit` remove whichever handler was stored first. `drop event handler` now takes `call M.Mf` to say which one; without it, a drop that matches several handlers is refused and names them. +- **A bare `create or modify module role M.R;` keeps the role's description** (ako/mxcli#1218) — the "make sure this role exists" statement reported `Modified module role: M.R` and erased the stored description: a statement without `description` was written as an empty one. An omitted `description` now keeps what is stored, as an omitted user role property already did, and the run reports `Unchanged`; `description ''` clears it. Grants were never affected, and `describe` → `exec` was not, since `describe` prints the description. Measured on Mendix 11.13.0: `mx check` is 0 errors with and without the fault, so only the stored description showed it. ## [0.28.0] - 2026-10-23 diff --git a/cmd/mxcli/syntax/features_security.go b/cmd/mxcli/syntax/features_security.go index 83372b349..698f5a7d4 100644 --- a/cmd/mxcli/syntax/features_security.go +++ b/cmd/mxcli/syntax/features_security.go @@ -22,7 +22,7 @@ func init() { "module role", "create role", "drop role", }, Syntax: "CREATE [OR MODIFY] MODULE ROLE . [DESCRIPTION ''];\nDROP MODULE ROLE .;", - Example: "mdl 1;\nCREATE MODULE ROLE Shop.Admin DESCRIPTION 'Full access';\n-- OR MODIFY makes a security script re-runnable:\nCREATE OR MODIFY MODULE ROLE Shop.User DESCRIPTION 'Read-only access';", + Example: "mdl 1;\nCREATE MODULE ROLE Shop.Admin DESCRIPTION 'Full access';\n-- OR MODIFY makes a security script re-runnable:\nCREATE OR MODIFY MODULE ROLE Shop.User DESCRIPTION 'Read-only access';\n-- Without DESCRIPTION it keeps the stored one; DESCRIPTION '' clears it:\nCREATE OR MODIFY MODULE ROLE Shop.Admin;", SeeAlso: []string{"security.user-role", "security.entity-access"}, }) diff --git a/docs-site/src/reference/security/create-module-role.md b/docs-site/src/reference/security/create-module-role.md index 13ed8b972..6a073d25c 100644 --- a/docs-site/src/reference/security/create-module-role.md +++ b/docs-site/src/reference/security/create-module-role.md @@ -3,7 +3,7 @@ ## Synopsis ```sql -CREATE MODULE ROLE module.RoleName [ DESCRIPTION 'text' ] +CREATE [ OR MODIFY ] MODULE ROLE module.RoleName [ DESCRIPTION 'text' ] ``` ## Description @@ -12,7 +12,7 @@ Creates a new module role within a module. Module roles are the foundation of Me Each module role belongs to exactly one module. The role name must be unique within the module. By convention, common role names include `User`, `Admin`, `Viewer`, and `Manager`. -If the module does not exist, the statement returns an error. +If the module does not exist, the statement returns an error. If the role exists, `CREATE MODULE ROLE` returns an error; `CREATE OR MODIFY` updates it instead, so a security script can be re-run. Without `DESCRIPTION`, `CREATE OR MODIFY` keeps the role's stored description; `DESCRIPTION ''` clears it. ## Parameters @@ -20,7 +20,7 @@ If the module does not exist, the statement returns an error. : Qualified name consisting of the module name and the new role name, separated by a dot. The module must already exist. `DESCRIPTION 'text'` -: Optional human-readable description of the role's purpose. +: Optional human-readable description of the role's purpose. Left out, a new role has none and an existing role keeps its own. ## Examples @@ -45,6 +45,12 @@ CREATE MODULE ROLE HR.Employee DESCRIPTION 'Self-service access'; CREATE MODULE ROLE HR.Viewer DESCRIPTION 'Read-only access'; ``` +Make sure a role exists, keeping whatever description it has: + +```sql +CREATE OR MODIFY MODULE ROLE HR.Manager; +``` + ## See Also [CREATE USER ROLE](create-user-role.md), [GRANT](grant.md), [REVOKE](revoke.md) diff --git a/mdl-examples/bug-tests/1218-module-role-description-kept.mdl b/mdl-examples/bug-tests/1218-module-role-description-kept.mdl new file mode 100644 index 000000000..417838328 --- /dev/null +++ b/mdl-examples/bug-tests/1218-module-role-description-kept.mdl @@ -0,0 +1,34 @@ +mdl 1; +-- ako/mxcli#1218: a bare `create or modify module role M.R;` silently erased +-- the role's description. +-- +-- Reported: +-- $ mxcli exec probe.mdl -p Formula1Backend.mpr +-- Modified module role: Formula1Backend.ApiUser +-- and `show module roles` had the role's Description empty afterwards. The +-- executor handed the statement's empty description to the backend, which +-- overwrites; an omitted DESCRIPTION clause was read as "set it to ''". +-- +-- An omitted clause now keeps what is stored, as an omitted user role property +-- already did (ako/mxcli#707). `description ''` is the spelling that clears it. +-- +-- mxcli exec 1218-module-role-description-kept.mdl -p app.mpr +-- mxcli -p app.mpr -c "show module roles in Zz1218" +-- +-- After the run Zz1218.ApiUser has the description from statement 1, and +-- Zz1218.Cleared has none. Re-running the script reports `Unchanged` for the +-- bare statement and `Modified` only where it states a different description. +-- Measured on Mendix 11.13.0: `mx check` is 0 errors with the fault and +-- without it, so only the stored description shows the difference. +-- The unit test is mdl/executor/issue1218_module_role_description_test.go. + +create or modify module Zz1218; + +create or modify module role Zz1218.ApiUser description 'Read the published OData services. Nothing else.'; + +-- The "make sure this role exists" idiom: keeps the description above. +create or modify module role Zz1218.ApiUser; + +-- Stating an empty description is how a script clears one. +create or modify module role Zz1218.Cleared description 'Temporary'; +create or modify module role Zz1218.Cleared description ''; diff --git a/mdl/ast/ast_security.go b/mdl/ast/ast_security.go index 18c3d38fc..80e6f0328 100644 --- a/mdl/ast/ast_security.go +++ b/mdl/ast/ast_security.go @@ -10,13 +10,26 @@ package ast type CreateModuleRoleStmt struct { CreateGuard // `create … if not exists` (ako/mxcli#731) Name QualifiedName - Description string + // Description is nil when the statement has no DESCRIPTION clause: a new + // role then has none, and `create or modify` keeps the stored one + // (ako/mxcli#1218), as CreateUserRoleStmt's properties do. `description ''` + // is the stated empty description, which clears it. + Description *string // CreateOrModify makes the statement idempotent: an existing role has its // description updated instead of the statement failing, so a security script // can be re-run. CreateOrModify bool } +// DescriptionText is the description the statement states, or "" when it +// states none. +func (s *CreateModuleRoleStmt) DescriptionText() string { + if s.Description == nil { + return "" + } + return *s.Description +} + func (s *CreateModuleRoleStmt) isStatement() {} // DropModuleRoleStmt represents: DROP MODULE ROLE [IF EXISTS] Module.RoleName diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 808f4354a..3b8eccd8a 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -44,7 +44,9 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { if mr.Description == autoDocumentRoleDescription { oldQualified := s.Name.Module + "." + mr.Name newQualified := s.Name.Module + "." + s.Name.Name - if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.Description); err != nil { + // Adopting the role takes it over: the marker description is not + // the author's documentation, so an unstated description clears it. + if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.DescriptionText()); err != nil { return mdlerrors.NewBackend("create module role", err) } // If the casing actually changed, propagate the rename across every @@ -64,8 +66,14 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { if s.CreateOrModify { // Re-running a security script must not fail on a role that is // already there. AddModuleRole overwrites, so this also adopts a new - // description and the caller's casing. - if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.Description); err != nil { + // description and the caller's casing. A statement without a + // description clause keeps the stored one: passing "" erased it, + // reported only as "Modified" (ako/mxcli#1218). + description := mr.Description + if s.Description != nil { + description = *s.Description + } + if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, description); err != nil { return mdlerrors.NewBackend("modify module role", err) } if !ctx.Quiet { @@ -76,7 +84,7 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { return mdlerrors.NewAlreadyExists("module role", s.Name.Module+"."+s.Name.Name) } - if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.Description); err != nil { + if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.DescriptionText()); err != nil { return mdlerrors.NewBackend("create module role", err) } diff --git a/mdl/executor/issue1218_module_role_description_test.go b/mdl/executor/issue1218_module_role_description_test.go new file mode 100644 index 000000000..767bb64d0 --- /dev/null +++ b/mdl/executor/issue1218_module_role_description_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// ako/mxcli#1218: "A bare `create or modify module role M.R;` silently erases +// the role's description". The statement reported "Modified module role" and +// the role's Description was gone: the executor handed the statement's empty +// description to AddModuleRole, which overwrites. An omitted clause keeps what +// is stored, as an omitted user role property already did (ako/mxcli#707); a +// description clause with an empty string is how a script clears it. +// +// The statements are parsed with the real visitor, so the test covers the +// whole chain from "the statement does not say" to the value written. +func TestIssue1218_CreateOrModifyModuleRole_KeepsUnstatedDescription(t *testing.T) { + const stored = "Read the published OData services. Nothing else." + cases := []struct { + name string + src string + stored *security.ModuleRole // nil: the role does not exist yet + want string // the description AddModuleRole writes + }{ + {"bare statement keeps the stored description", + "create or modify module role Shop.ApiUser;", + &security.ModuleRole{Name: "ApiUser", Description: stored}, stored}, + {"a stated description replaces it", + "create or modify module role Shop.ApiUser description 'Calls the API';", + &security.ModuleRole{Name: "ApiUser", Description: stored}, "Calls the API"}, + {"an empty description clears it", + "create or modify module role Shop.ApiUser description '';", + &security.ModuleRole{Name: "ApiUser", Description: stored}, ""}, + {"a casing change keeps the stored description", + "create or modify module role Shop.apiUser;", + &security.ModuleRole{Name: "ApiUser", Description: stored}, stored}, + {"or replace reads as or modify and keeps it too", + "mdl 1;\ncreate or replace module role Shop.ApiUser;", + &security.ModuleRole{Name: "ApiUser", Description: stored}, stored}, + // Adopting a role mxcli auto-provisioned takes it over: its marker + // description is not the author's documentation, so it is not kept. + {"adopting an auto-provisioned role drops its marker", + "create module role Shop.ApiUser;", + &security.ModuleRole{Name: "ApiUser", Description: autoDocumentRoleDescription}, ""}, + {"a new role without a description has none", + "create or modify module role Shop.ApiUser;", + nil, ""}, + {"a new role takes the stated description", + "create module role Shop.ApiUser description 'Calls the API';", + nil, "Calls the API"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + mod := &model.Module{BaseElement: model.BaseElement{ID: "mod-shop"}, Name: "Shop"} + ms := &security.ModuleSecurity{BaseElement: model.BaseElement{ID: "ms-shop"}, ContainerID: mod.ID} + if tc.stored != nil { + ms.ModuleRoles = []*security.ModuleRole{tc.stored} + } + var written []string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleSecurityFunc: func(model.ID) (*security.ModuleSecurity, error) { return ms, nil }, + AddModuleRoleFunc: func(_ model.ID, _ string, description string) error { + written = append(written, description) + return nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + prog, errs := visitor.Build(tc.src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + stmt := findStmt[*ast.CreateModuleRoleStmt](t, prog, tc.src) + if err := execCreateModuleRole(ctx, stmt); err != nil { + t.Fatalf("exec: %v\n%s", err, buf.String()) + } + if len(written) != 1 { + t.Fatalf("AddModuleRole called %d time(s), want 1\n%s", len(written), buf.String()) + } + if written[0] != tc.want { + t.Errorf("description written = %q, want %q\n%s", written[0], tc.want, buf.String()) + } + }) + } +} diff --git a/mdl/executor/issue707_describe_roundtrip_test.go b/mdl/executor/issue707_describe_roundtrip_test.go index 923c7868a..296dcd9fb 100644 --- a/mdl/executor/issue707_describe_roundtrip_test.go +++ b/mdl/executor/issue707_describe_roundtrip_test.go @@ -114,8 +114,8 @@ func TestIssue707_ModuleRoleDescriptionReparses(t *testing.T) { out := buf.String() stmt := findStmt[*ast.CreateModuleRoleStmt](t, reparse(t, out), out) - if stmt.Description != "Can't delete orders" { - t.Errorf("description: stored %q, re-exec stores %q\n%s", "Can't delete orders", stmt.Description, out) + if stmt.DescriptionText() != "Can't delete orders" { + t.Errorf("description: stored %q, re-exec stores %q\n%s", "Can't delete orders", stmt.DescriptionText(), out) } } diff --git a/mdl/visitor/module_role_create_statement_test.go b/mdl/visitor/module_role_create_statement_test.go index abdc78eab..6aaaf4296 100644 --- a/mdl/visitor/module_role_create_statement_test.go +++ b/mdl/visitor/module_role_create_statement_test.go @@ -35,7 +35,7 @@ func TestCreateModuleRole_IsACreateStatementKind(t *testing.T) { if !ok { t.Fatalf("%s: got %T", tc.src, prog.Statements[0]) } - if s.Name.String() != "M.Admin" || s.CreateOrModify != tc.orModify || s.Description != tc.desc { + if s.Name.String() != "M.Admin" || s.CreateOrModify != tc.orModify || s.DescriptionText() != tc.desc { t.Errorf("%s: got %+v", tc.src, s) } if got := len(prog.Deprecations) > 0; got != tc.deprecated { diff --git a/mdl/visitor/visitor_security.go b/mdl/visitor/visitor_security.go index 0defad6da..3f00edd35 100644 --- a/mdl/visitor/visitor_security.go +++ b/mdl/visitor/visitor_security.go @@ -27,7 +27,10 @@ func (b *Builder) ExitCreateModuleRoleStatement(ctx *parser.CreateModuleRoleStat } if ctx.DESCRIPTION() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Description = unquoteStringLit(sl) + // A pointer, so `description ''` (clear it) and no clause at all + // (keep what is stored) stay different statements (ako/mxcli#1218). + d := unquoteStringLit(sl) + stmt.Description = &d } } b.statements = append(b.statements, stmt) diff --git a/mdl/visitor/visitor_security_or_modify_test.go b/mdl/visitor/visitor_security_or_modify_test.go index 22521e404..f38cbed14 100644 --- a/mdl/visitor/visitor_security_or_modify_test.go +++ b/mdl/visitor/visitor_security_or_modify_test.go @@ -42,8 +42,8 @@ func TestCreateModuleRole_OrModify(t *testing.T) { if got.Name.Name != tc.wantName { t.Errorf("Name = %q, want %q", got.Name.Name, tc.wantName) } - if got.Description != tc.desc { - t.Errorf("Description = %q, want %q", got.Description, tc.desc) + if got.DescriptionText() != tc.desc { + t.Errorf("Description = %q, want %q", got.DescriptionText(), tc.desc) } }) } diff --git a/mdl/visitor/visitor_security_test.go b/mdl/visitor/visitor_security_test.go index 440717b87..00e783c1a 100644 --- a/mdl/visitor/visitor_security_test.go +++ b/mdl/visitor/visitor_security_test.go @@ -28,8 +28,25 @@ func TestCreateModuleRole(t *testing.T) { if stmt.Name.Module != "MyModule" || stmt.Name.Name != "Editor" { t.Errorf("Expected MyModule.Editor, got %s.%s", stmt.Name.Module, stmt.Name.Name) } - if stmt.Description != "Can edit content" { - t.Errorf("Expected description 'Can edit content', got %q", stmt.Description) + if stmt.DescriptionText() != "Can edit content" { + t.Errorf("Expected description 'Can edit content', got %q", stmt.DescriptionText()) + } +} + +// A description clause with an empty string states an empty description, which +// clears a stored one under `create or modify`; no clause states none, which +// keeps it (ako/mxcli#1218). +func TestCreateModuleRole_EmptyDescriptionIsStated(t *testing.T) { + prog, errs := Build(`create or modify module role MyModule.Viewer description '';`) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateModuleRoleStmt) + if !ok { + t.Fatalf("Expected CreateModuleRoleStmt, got %T", prog.Statements[0]) + } + if stmt.Description == nil || *stmt.Description != "" { + t.Errorf("Description = %v, want a stated empty string", stmt.Description) } } @@ -46,8 +63,8 @@ func TestCreateModuleRole_NoDescription(t *testing.T) { if !ok { t.Fatalf("Expected CreateModuleRoleStmt, got %T", prog.Statements[0]) } - if stmt.Description != "" { - t.Errorf("Expected empty description, got %q", stmt.Description) + if stmt.Description != nil { + t.Errorf("Expected no description clause, got %q", *stmt.Description) } } From 7f0d71e39c6e5bfd3b3a09dc529412e872805c8a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:28:53 +0000 Subject: [PATCH 04/61] fix(modelsdk): declare AggregateFunction on Mendix 9, floor Reduce's fold at 10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom (ako/mxcli#1163, 02-microflow-examples on 9.24.30): every count/sum/average/minimum/maximum in a microflow was refused with "this Mendix 9.24.30 project cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". Cause: not an older storage shape. The vendored generated version data gave aggregateFunction the row of reduceReturnDataType ({Introduced 10.0.0, Required}) and had no row for reduceReturnDataType, so the property floor refused the key every aggregate carries and let the fold's return type through unrefused. Evidence: mendixmodelsdk 4.117.0 has aggregateFunction with no version (its default even changes at 9.24.0) and reduceReturnDataType introduced 10.0.0. Studio Pro 9.24.0 stores AggregateFunction on 10 of 10 aggregates in Workflow Commons 2.6.0. A floor-off build's aggregate survives `mx convert` 9.24.30 with every key. Fix: correct the Microflows$AggregateAction row by hand (gen cannot be regenerated here) and regenerate the copy in modelsdk/version. Measured: bug-test 1163-aggregate-on-mendix9.mdl on a fresh 9.24.30 app: exec writes all five aggregates, mx check 0 errors, mx convert strips nothing. Before, the first was refused. Control: the new tests fail on the unfixed table with the reported refusal. 02-microflow-examples on 9.24 now stops at the expression aggregate (`sum … of …`, UseExpression and Expression, Mendix 10.0+), which is correct. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...ggregate-function-floored-at-mendix10.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 5 +- .../bug-tests/1163-aggregate-on-mendix9.mdl | 36 ++++++++ modelsdk/gen/microflows/version.go | 8 +- .../mpr/writer_version_shape_1163_test.go | 83 +++++++++++++++++++ modelsdk/version/metamodel_versions_gen.go | 8 +- 7 files changed, 138 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json create mode 100644 mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl create mode 100644 modelsdk/mpr/writer_version_shape_1163_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json new file mode 100644 index 000000000..bac85dc0a --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x every count/sum/average/minimum/maximum in a microflow was refused: `this Mendix 9.24.30 project cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)` (02-microflow-examples on 9.24)", "cause": "The vendored generated version data (modelsdk/gen/microflows/version.go) gave aggregateFunction the row of reduceReturnDataType ({Introduced 10.0.0, Required}) and had no row for reduceReturnDataType, so the floor refused the one key every aggregate has and let the fold's return type through unrefused", "file": "`modelsdk/gen/microflows/version.go` (Microflows$AggregateAction, fixed by hand), `modelsdk/version/metamodel_versions_gen.go` (go generate ./modelsdk/version)", "insight": "**A refusal can be the floor's data being wrong, not the writer's shape.** The prompt to look for an 'older shape' was a guess; evidence settled it in two steps: mendixmodelsdk 4.117.0 has `aggregateFunction: {}` (no version) and even sets its default differently either side of 9.24.0, and Studio Pro-authored 9.24.0 bytes (Workflow Commons 2.6.0 from the Marketplace, `mxcli marketplace download 117066 --version 2.6.0`, project.mpr inside the .mpk) store `AggregateFunction: \"Count\"` on 10 of 10 aggregates. A one-off script comparing every SDK StructureVersionInfo `introduced` with modelsdk/gen/*/version.go found only 4 mismatches in the whole table (this one; CustomWidgets$WidgetReturnType.type 9.20 vs none; Forms$GroupBox.caption 8.10 vs none; Workflows$CallMicroflowTask.boundaryEvents), so the generator's misattribution is rare, not systematic. **The cheapest 'which keys does version X declare' oracle is a floor-off build + `mx convert -p` on that version**: build mxcli with StripUndeclaredProperties stubbed, exec, dump units, convert, dump again, diff key sets per $Type — 9.24's convert kept all of AggregateAction's keys. Measured: 5 aggregates on a fresh 9.24.30 app → exec ok, mx check 0 errors, mx convert strips nothing (before: refused). Tests: TestNormalize_AggregateFunctionIsDeclaredOnMendix9, TestNormalize_ReduceFoldIsRefusedBelowMendix10; mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index a947aaac4..d0905b71c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 67099ac20..05ee85520 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -186,11 +186,12 @@ Some page properties kept their meaning and changed how they are stored. mxcli b | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | -What an older shape cannot hold is refused, naming the widget, rather than dropped: +What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; - conditional data view editability below 10.7; - a placeholder with parameters below 10.11; -- a data view fed by another widget below 10.0. +- a data view fed by another widget below 10.0; +- an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl b/mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl new file mode 100644 index 000000000..d13e79e3a --- /dev/null +++ b/mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl @@ -0,0 +1,36 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x every count/sum/average/minimum/maximum in a +-- microflow was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+) +-- +-- The property floor came from the vendored generated version data, which had +-- given AggregateFunction the row of ReduceReturnDataType (the fold Reduce +-- added in 10.0) and left that one out. Studio Pro 9.24.0 stores +-- AggregateFunction on every aggregate (Workflow Commons 2.6.0), `mx convert` +-- 9.24.30 keeps it, and mendixmodelsdk gives it no version. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec +-- writes all five, and `mxcli docker check` reports the blank app's error +-- count. An aggregate over an expression (`sum $L of …`) and `reduce`, `all` +-- and `any` are Mendix 10.0 and are still refused below it. + +create module Probe1163Agg; + +create persistent entity Probe1163Agg.Line ( + Amount: Decimal +); + +create microflow Probe1163Agg.Totals ( + $Lines: list of Probe1163Agg.Line +) +returns decimal as $Sum +begin + $Count = count $Lines; + $Sum = sum $Lines by Amount; + $Average = average $Lines by Amount; + $Min = minimum $Lines by Amount; + $Max = maximum $Lines by Amount; + return $Sum; +end; diff --git a/modelsdk/gen/microflows/version.go b/modelsdk/gen/microflows/version.go index 01eea8f03..dcd1c8162 100644 --- a/modelsdk/gen/microflows/version.go +++ b/modelsdk/gen/microflows/version.go @@ -45,9 +45,15 @@ var VersionInfos = map[string]version.TypeVersionInfo{ }, "Microflows$AggregateAction": { Properties: map[string]version.PropertyVersionInfo{ - "aggregateFunction": {Introduced: "10.0.0", Required: true}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave + // aggregateFunction the row of reduceReturnDataType, which it then + // left out. mendixmodelsdk 4.117.0 has aggregateFunction with no + // version and reduceReturnDataType introduced 10.0.0 and required; + // Studio Pro 9.24 stores AggregateFunction (Workflow Commons 2.6.0). + "aggregateFunction": {}, "expression": {Introduced: "10.0.0"}, "reduceInitialValueExpression": {Introduced: "10.0.0"}, + "reduceReturnDataType": {Introduced: "10.0.0", Required: true}, "useExpression": {Introduced: "10.0.0"}, }, }, diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go new file mode 100644 index 000000000..d62ab14de --- /dev/null +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// ako/mxcli#1163: what the 9.24 doctype run found mxcli writing that Mendix 9 +// stores differently, or not at all. Each case is the document as mxcli builds +// it, normalized for the project's version the way every unit write is. + +// aggregateMicroflow is a microflow holding one Aggregate list activity as the +// writer builds it: Count/Sum/… carry no fold; Reduce, All and Any carry +// ReduceInitialValueExpression and ReduceReturnDataType. +func aggregateMicroflow(function string) bson.D { + action := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$AggregateAction"}, + {Key: "AggregateFunction", Value: function}, + {Key: "AggregateVariableName", Value: "Items"}, + {Key: "Attribute", Value: "M.Item.Price"}, + {Key: "ErrorHandlingType", Value: "Rollback"}} + switch function { + case "Reduce", "All", "Any": + initial := "" + if function == "Reduce" { + initial = "0" + } + action = append(action, + bson.E{Key: "ReduceInitialValueExpression", Value: initial}, + bson.E{Key: "ReduceReturnDataType", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "DataTypes$BooleanType"}}}) + } + action = append(action, bson.E{Key: "VariableName", Value: "Total"}) + activity := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$ActionActivity"}, {Key: "Action", Value: action}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$Microflow"}, {Key: "Name", Value: "Agg"}, + {Key: "ObjectCollection", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), activity}}}}} +} + +// Microflows$AggregateAction.AggregateFunction is declared on every version +// mxcli writes. The generated version data floored it at 10.0.0, so every +// count/sum/average/minimum/maximum on 9.x was refused ("cannot store +// Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)"). Studio Pro +// 9.24.0 stores it (Workflow Commons 2.6.0: 10 of 10 aggregates, `Count`), `mx +// convert` 9.24.30 keeps it, and mendixmodelsdk gives it no version at all; the +// 10.0.0 floor belongs to ReduceReturnDataType, the fold Reduce added. +func TestNormalize_AggregateFunctionIsDeclaredOnMendix9(t *testing.T) { + for _, v := range []*version.ProjectVersion{pv(9, 0, 5), pv(9, 24, 30), pv(10, 0, 0), pv(11, 13, 0)} { + d, err := normalizeShape(t, v, aggregateMicroflow("Sum")) + if err != nil { + t.Errorf("%d.%d: a sum aggregate is refused: %v", v.MajorVersion, v.MinorVersion, err) + continue + } + if f, _ := get(find(d, "Microflows$AggregateAction"), "AggregateFunction"); f != "Sum" { + t.Errorf("%d.%d: AggregateFunction = %v, want Sum", v.MajorVersion, v.MinorVersion, f) + } + } +} + +// The fold is what Mendix 10.0 added (mendixmodelsdk: ReduceReturnDataType +// introduced 10.0.0). `all` carries an empty initial value, so the fold's +// return type is the one key that says it: on 9.24 it is refused, naming it, +// rather than written into a document Studio Pro 9 cannot resolve. Before, the +// generated data had no row for it and it passed through. +func TestNormalize_ReduceFoldIsRefusedBelowMendix10(t *testing.T) { + for _, fn := range []string{"Reduce", "All"} { + if _, err := normalizeShape(t, pv(9, 24, 30), aggregateMicroflow(fn)); err == nil || + !strings.Contains(err.Error(), "Microflows$AggregateAction.ReduceReturnDataType (Mendix 10.0.0+)") { + t.Errorf("%s on 9.24 was not refused naming its fold's return type: %v", fn, err) + } + d, err := normalizeShape(t, pv(10, 0, 0), aggregateMicroflow(fn)) + if err != nil { + t.Fatalf("%s on 10.0 is refused: %v", fn, err) + } + if _, ok := get(find(d, "Microflows$AggregateAction"), "ReduceReturnDataType"); !ok { + t.Errorf("%s on 10.0 lost ReduceReturnDataType", fn) + } + } +} diff --git a/modelsdk/version/metamodel_versions_gen.go b/modelsdk/version/metamodel_versions_gen.go index d5e02fa57..37591339c 100644 --- a/modelsdk/version/metamodel_versions_gen.go +++ b/modelsdk/version/metamodel_versions_gen.go @@ -1241,9 +1241,15 @@ var metamodelVersions = []map[string]TypeVersionInfo{ }, "Microflows$AggregateAction": { Properties: map[string]PropertyVersionInfo{ - "aggregateFunction": {Introduced: "10.0.0", Required: true}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave + // aggregateFunction the row of reduceReturnDataType, which it then + // left out. mendixmodelsdk 4.117.0 has aggregateFunction with no + // version and reduceReturnDataType introduced 10.0.0 and required; + // Studio Pro 9.24 stores AggregateFunction (Workflow Commons 2.6.0). + "aggregateFunction": {}, "expression": {Introduced: "10.0.0"}, "reduceInitialValueExpression": {Introduced: "10.0.0"}, + "reduceReturnDataType": {Introduced: "10.0.0", Required: true}, "useExpression": {Introduced: "10.0.0"}, }, }, From 57cb3179ade693af0a1c4ea78b68ab56ba6d38b0 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:41:27 +0000 Subject: [PATCH 05/61] fix(canon): write no workflow PersistentId below Mendix 10.21 Symptom (ako/mxcli#1163, 25-rename on 9.24.30): every workflow was refused with "this Mendix 9.24.30 project cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". The same held on 10.0-10.20. Cause: the workflow writer, and the ALTER WORKFLOW mutator for the activities it inserts, mint a fresh PersistentId on the workflow and on every activity, outcome and boundary event. mendixmodelsdk introduces PersistentId in 10.21.0 on all four (Workflow, and the abstract WorkflowActivity, Outcome and BoundaryEvent). The floor refused the workflow's own. It never saw the others, which are declared on supertypes it does not look up. `mx convert` 9.24.30 strips the PersistentId from all 6 element types a floor-off build wrote. Fix: canon.LowerToProjectShape drops PersistentId from every Workflows$ element below 10.21. It runs on every unit write before the floor, so CREATE, CREATE OR MODIFY and wfmutator's raw inserts are all covered. The ID is dropped rather than refused because the writer mints it; no script states it, and Studio Pro 9.24 stores none. Measured: bug-test 1163-workflow-on-mendix9.mdl on a fresh 9.24.30 app: exec ok, mx check 0 errors, `mx convert -p` strips no key. 25-rename on 9.24: FAIL (the refusal) -> PASS. Controls: 25-rename and 24b-workflow-mendix10 PASS on 10.24.24 and 11.13.0. The unit test fails on the unfixed code with the reported refusal. Not fixed: a parallel split writes Workflows$EndOfParallelSplitPathActivity, a 10.21 type, and 9.24 then does not load the project. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-28-workflow-persistentid-below-10-21.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 5 +- .../bug-tests/1163-workflow-on-mendix9.mdl | 55 +++++++++++++ modelsdk/canon/versionshape.go | 27 +++++- .../mpr/writer_version_shape_1163_test.go | 82 +++++++++++++++++++ 6 files changed, 167 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json create mode 100644 mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json new file mode 100644 index 000000000..52ce2c428 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.20 every workflow was refused: `this Mendix 9.24.30 project cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)` (25-rename on 9.24)", "cause": "The workflow writer (and the ALTER WORKFLOW mutator, for the activities it inserts) mints a fresh PersistentId on the workflow and on every activity, outcome and boundary event. PersistentId is 10.21.0 on all four (Workflow, and the abstract WorkflowActivity, Outcome, BoundaryEvent). The floor refused the workflow's own and never saw the rest, because it does not look up abstract supertypes", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: dropPersistentID on every Workflows$ element below 10.21)", "insight": "**An identity the writer mints is not a value the statement said**, so below its version it is dropped, not refused (unlike ForceFullObjects, which a script can set). Lower it in canon.LowerToProjectShape rather than encodeWorkflow: wfmutator's raw inserts (mutator.go: PersistentId on inserted activities, outcomes, boundary events) never pass encodeWorkflow, but every write passes normalizePropertySetsFor. **Test with a non-zero GUID**: a zero GUID is an empty value the floor silently drops, so the first draft of the test passed on the unfixed code. **The floor-off build + `mx convert -p` 9.24 key diff** showed the whole picture at once: PersistentId on 6 element types, plus the empty keys the floor already handles. The same run found the next gap: a `parallel split` writes Workflows$EndOfParallelSplitPathActivity (a 10.21 type) and 9.24 does not load the project (TypeCacheUnknownTypeException) — not fixed here. Measured: bug-test on a fresh 9.24.30 app → exec ok, mx check 0 errors, mx convert strips no key (adds WorkflowOnStateChangeEvent/UsertaskOnStateChangeEvent null). 25-rename 9.24 FAIL→PASS; 25-rename and 24b on 10.24.24 and 11.13 PASS. Tests: TestNormalize_WorkflowPersistentIDsBelowMendix10_21; mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index d0905b71c..5a03ede06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 05ee85520..3f773a76f 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -174,9 +174,9 @@ What *is* version-specific is the optional/default-value half: `PageParameter.Is Atlas UI v3 (Mendix 11.0+) introduced new design properties like "Card style" and "Disable row wrap" for containers. These don't exist in the Atlas v2 theme bundled with 10.x, causing CE6083. -### Page Properties Stored in an Older Shape (below 10.2, 10.7, 10.11) +### Properties Stored in an Older Shape -Some page properties kept their meaning and changed how they are stored. mxcli builds the current shape and, on every write, rewrites it into the shape the project's version stores. `describe` reads both shapes ([ako/mxcli#1209](https://github.com/ako/mxcli/issues/1209)). The older shapes were measured on Studio Pro-authored 9.24.30 pages: +Some properties kept their meaning and changed how they are stored. mxcli builds the current shape and, on every write, rewrites it into the shape the project's version stores. `describe` reads both shapes ([ako/mxcli#1209](https://github.com/ako/mxcli/issues/1209), [ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). The older shapes were measured on Studio Pro-authored 9.24 documents and with `mx convert` 9.24.30: | Property | Below | Stored as | |---|---|---| @@ -185,6 +185,7 @@ Some page properties kept their meaning and changed how they are stored. mxcli b | A data view's editability | 10.7 | `Editable: true\|false`, instead of `Editability` | | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | +| The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; diff --git a/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl new file mode 100644 index 000000000..ecea673ad --- /dev/null +++ b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl @@ -0,0 +1,55 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.20 every workflow was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Workflows$Workflow.PersistentId (Mendix 10.21.0+) +-- +-- Workflows exist from 9.0; PersistentId arrived in 10.21.0, on the workflow +-- and on every activity, outcome and boundary event (mendixmodelsdk). The +-- activities' and outcomes' were declared on abstract supertypes the property +-- floor does not look up, so they were never refused, and `mx convert` 9.24.30 +-- strips every one. A PersistentId is minted by the writer, never stated by a +-- script, so below 10.21 none is written: for CREATE, CREATE OR MODIFY and +-- the activities ALTER WORKFLOW inserts on its own. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec, +-- then `mxcli docker check` reports the blank app's error count, and `mx +-- convert -p` strips no PersistentId. +-- +-- Not covered: a `parallel split`. Its paths end in a +-- Workflows$EndOfParallelSplitPathActivity, a type Mendix introduced in 10.21, +-- so below 10.21 the project does not load (TypeCacheUnknownTypeException). + +create module Probe1163Wf; + +create persistent entity Probe1163Wf.Request ( + Title: String(100), + Amount: Integer +); + +create microflow Probe1163Wf.ACT_Validate ($WorkflowContext: Probe1163Wf.Request) +begin + log info node 'Probe1163Wf' 'validate'; +end; + +create microflow Probe1163Wf.ACT_Notify ($WorkflowContext: Probe1163Wf.Request) +begin + log info node 'Probe1163Wf' 'notify'; +end; + +create workflow Probe1163Wf.WF_Request + parameter $WorkflowContext: Probe1163Wf.Request + display 'Request' +begin + call microflow Probe1163Wf.ACT_Validate; + decision $WorkflowContext/Amount > 1000 + outcomes + true -> { call microflow Probe1163Wf.ACT_Notify; } + false -> { end workflow; } + ; +end workflow; + +-- ALTER serializes the inserted activity on its own, outside CREATE's encoder. +alter workflow Probe1163Wf.WF_Request { + insert after ACT_Validate { call microflow Probe1163Wf.ACT_Notify; } +}; diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index fb1efebde..31be06cae 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -4,6 +4,7 @@ package canon import ( "fmt" + "strings" "go.mongodb.org/mongo-driver/v2/bson" @@ -37,6 +38,10 @@ import ( // CE6050 "The selected entity must be configured with a page parameter". // - ForceFullObjects, declared on the abstract Forms$DataSource from 10.8.0: // every data source carried it, and the floor never saw it. +// - Workflows$ PersistentId, from 10.21.0 on the workflow and on the abstract +// WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows +// exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's +// own was refused; the others slipped past the floor. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -53,8 +58,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { textPlaceholder: !PropertyDeclared("Forms$AttributeWidgetWithPlaceholder", "PlaceholderTemplate", *pv), namedParameter: !PropertyDeclared("Forms$EntityPathSource", "SourceVariable", *pv), noFullObjects: !PropertyDeclared("Forms$DataSource", "ForceFullObjects", *pv), + noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), } - if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects { + if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && + !l.noPersistentID { return contents, nil } var doc bson.D @@ -74,7 +81,7 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool - namedParameter, noFullObjects bool + namedParameter, noFullObjects, noPersistentID bool changed bool } @@ -112,6 +119,9 @@ func (l *shapeLowering) walk(v any) (any, error) { return nil, err } } + if l.noPersistentID && strings.HasPrefix(typ, "Workflows$") { + x = l.dropPersistentID(x) + } for i := range x { if x[i].Value, err = l.walk(x[i].Value); err != nil { return nil, err @@ -261,6 +271,19 @@ func (l *shapeLowering) dropForceFullObjects(d bson.D, typ string) (bson.D, erro return append(append(bson.D{}, d[:i]...), d[i+1:]...), nil } +// dropPersistentID removes a workflow element's PersistentId below 10.21, where +// none declares it. Whatever it holds is dropped, not refused: a PersistentId is +// minted by the writer (a fresh GUID on every build), never stated by a script, +// and a workflow Studio Pro 9.24 stores has none. +func (l *shapeLowering) dropPersistentID(d bson.D) bson.D { + i := shapeIndex(d, "PersistentId") + if i < 0 { + return d + } + l.changed = true + return append(append(bson.D{}, d[:i]...), d[i+1:]...) +} + // listItems returns the elements of a stored list, without its type marker. func listItems(v any) bson.A { a, _ := v.(bson.A) diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index d62ab14de..f3ae39ac5 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -81,3 +81,85 @@ func TestNormalize_ReduceFoldIsRefusedBelowMendix10(t *testing.T) { } } } + +// pid is a fresh PersistentId: a non-zero GUID, as the writer mints one (a zero +// GUID is an empty value, which the floor would drop rather than refuse). +func pid() bson.Binary { + pidSeq++ + return bson.Binary{Subtype: 0, Data: []byte{pidSeq, 0x11, 2, 3, 4, 5, 0x46, 7, 0x88, 9, 10, 11, 12, 13, 14, 15}} +} + +var pidSeq byte + +// workflowWithPersistentIDs is a workflow as mxcli builds it: a fresh +// PersistentId on the workflow, on every activity and on every outcome. +func workflowWithPersistentIDs() bson.D { + outcome := func(value bool) bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$BooleanConditionOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: bson.A{int32(3)}}}}, + {Key: "PersistentId", Value: pid()}, {Key: "Value", Value: value}} + } + split := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$ExclusiveSplitActivity"}, + {Key: "Caption", Value: "Active?"}, {Key: "Expression", Value: "$WorkflowContext/Active"}, {Key: "Name", Value: "split"}, + {Key: "Outcomes", Value: bson.A{int32(3), outcome(true), outcome(false)}}, + {Key: "PersistentId", Value: pid()}} + call := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$CallMicroflowTask"}, + {Key: "BoundaryEvents", Value: bson.A{int32(2)}}, {Key: "Caption", Value: "Validate"}, + {Key: "Microflow", Value: "M.Validate"}, {Key: "Name", Value: "call"}, {Key: "PersistentId", Value: pid()}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), call, split}}}}, + {Key: "Name", Value: "WF"}, + {Key: "PersistentId", Value: pid()}, + {Key: "Title", Value: "WF"}} +} + +// persistentIDHolders returns the $Type of every element in v carrying a +// PersistentId. +func persistentIDHolders(v any) []string { + var out []string + switch x := v.(type) { + case bson.D: + typ, _ := get(x, "$Type") + for _, e := range x { + if e.Key == "PersistentId" { + out = append(out, typ.(string)) + } + out = append(out, persistentIDHolders(e.Value)...) + } + case bson.A: + for _, el := range x { + out = append(out, persistentIDHolders(el)...) + } + } + return out +} + +// Workflows exist from 9.0, PersistentId from 10.21.0 — on the workflow and on +// WorkflowActivity, Outcome and BoundaryEvent (mendixmodelsdk). Below that a +// workflow was refused ("cannot store Workflows$Workflow.PersistentId (Mendix +// 10.21.0+)"), and the activities' and outcomes' were never seen by the floor, +// which does not look up the abstract supertypes that declare them: `mx +// convert` 9.24.30 strips all of them. A PersistentId is minted by the writer, +// never stated by a script, so below 10.21 it is not written. +func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want int + }{ + {pv(9, 24, 30), 0}, + {pv(10, 20, 0), 0}, + {pv(10, 21, 0), 5}, + {pv(10, 24, 24), 5}, + {pv(11, 13, 0), 5}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithPersistentIDs()) + if err != nil { + t.Errorf("%d.%d: a workflow is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + if got := persistentIDHolders(d); len(got) != tt.want { + t.Errorf("%d.%d: PersistentId on %d elements %v, want %d", tt.v.MajorVersion, tt.v.MinorVersion, len(got), got, tt.want) + } + } +} From 2c7876b6b9a461a1c773675a5abef512148bb355 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:43:08 +0000 Subject: [PATCH 06/61] fix(canon): complete a workflow's state-change handlers on Mendix 9.24 Symptom (found while measuring ako/mxcli#1163): with PersistentId lowered, `mx convert -p` 9.24.30 strips nothing from an mxcli-written workflow but adds WorkflowOnStateChangeEvent: null and UsertaskOnStateChangeEvent: null. Studio Pro's next save then differs in property names, which Mendix's merge engine refuses to compare (#1373). Cause: the two defaults (mendixmodelsdk: introduced 9.12.0, removed 11.0.0) were measured on 10.24.24 only, and an entry is added only from the lowest version it was measured on. Fix: lower their measured floor to 9.24.30, where `mx convert` adds both with the same null value. Measured: bug-test 1163-workflow-on-mendix9.mdl on a fresh 9.24.30 app: mx check 0 errors, and `mx convert -p` now neither strips nor adds a key on the workflow. Before, it added these two. The test fails on the previous floor (present false on 9.24). The 11.13 case stays absent as the control. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- CHANGELOG.md | 1 + .../canon/studiopro_property_defaults.json | 4 +-- .../mpr/writer_version_shape_1163_test.go | 26 +++++++++++++++++++ 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a03ede06..bcd5486ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. diff --git a/modelsdk/canon/studiopro_property_defaults.json b/modelsdk/canon/studiopro_property_defaults.json index 5ceeade35..f05785e61 100644 --- a/modelsdk/canon/studiopro_property_defaults.json +++ b/modelsdk/canon/studiopro_property_defaults.json @@ -109,6 +109,6 @@ {"type":"Workflows$InterruptingTimerBoundaryEvent","key":"Annotation","since":"11.6.8","seen":"11.15.0","value":{"v":null}}, {"type":"Workflows$NonInterruptingTimerBoundaryEvent","key":"Annotation","since":"11.6.8","seen":"11.15.0","value":{"v":null}}, {"type":"Workflows$NonInterruptingTimerEventSubProcessStartActivity","key":"Recurrence","since":"11.15.0","seen":"11.15.0","value":{"v":null}}, - {"type":"Workflows$Workflow","key":"UsertaskOnStateChangeEvent","since":"10.24.24","until":"10.25.0","seen":"10.24.24","value":{"v":null}}, - {"type":"Workflows$Workflow","key":"WorkflowOnStateChangeEvent","since":"10.24.24","until":"10.25.0","seen":"10.24.24","value":{"v":null}} + {"type":"Workflows$Workflow","key":"UsertaskOnStateChangeEvent","since":"9.24.30","until":"10.25.0","seen":"10.24.24","value":{"v":null}}, + {"type":"Workflows$Workflow","key":"WorkflowOnStateChangeEvent","since":"9.24.30","until":"10.25.0","seen":"10.24.24","value":{"v":null}} ] diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index f3ae39ac5..a916b70ea 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -163,3 +163,29 @@ func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { } } } + +// Studio Pro's property set for a workflow on 9.24 includes the two state-change +// event handlers Mendix removed in 11.0: `mx convert` 9.24.30 adds both as null +// to a workflow mxcli wrote, as 10.24.24 does — a property-name difference +// Mendix's merge engine refuses to compare (#1373). +func TestNormalize_WorkflowStateChangeEventsOnMendix9(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 24, 24), true}, + {pv(11, 13, 0), false}, // removed in 11.0 + } { + d, err := normalizeShape(t, tt.v, workflowWithPersistentIDs()) + if err != nil { + t.Fatal(err) + } + wf := find(d, "Workflows$Workflow") + for _, k := range []string{"WorkflowOnStateChangeEvent", "UsertaskOnStateChangeEvent"} { + if v, ok := get(wf, k); ok != tt.want || v != nil { + t.Errorf("%d.%d: %s present %v (%v), want present %v and null", tt.v.MajorVersion, tt.v.MinorVersion, k, ok, v, tt.want) + } + } + } +} From 4df889ee9ce46ef1b3826ce535e3c456049c0781 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:51:21 +0000 Subject: [PATCH 07/61] fix(canon): write a constant override flat below Mendix 10.9 Symptom (ako/mxcli#1163, 14-project-settings on 9.24.30): `alter settings constant @M.C value '...' in configuration 'Default'` was refused with "this Mendix 9.24.30 project cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". The same held on 10.0-10.8. Cause: the settings overlay writes a new override as a Settings$SharedValue nested under SharedOrPrivateValue. From 10.9 that is the only shape Mendix reads. Below 10.9 an override is {$ID, $Type, ConstantId, Value} with a flat string (mendixmodelsdk: Value deleted and SharedOrPrivateValue introduced in 10.9.0). With the floor off, `mx convert` 9.24.30 turns the nested form into Value: "" and the override silently becomes empty. Fix: canon.LowerToProjectShape replaces SharedOrPrivateValue with the flat Value in its place, or fills the empty flat sibling the overlay leaves. A Settings$PrivateValue has no flat form, so it is refused, naming the constant. The reader already preferred a flat Value; a backend test now pins reading and updating a 9.x flat override. Measured: bug-test 1163-constant-override-on-mendix9.mdl on a fresh 9.24.30 app: describe settings shows 'broker:9092', mx check 0 errors, and mx convert keeps the value and changes no key in the settings unit. 14-project-settings on 9.24: FAIL (the refusal) -> PASS. Controls: 10.24.24 and 11.13.0 PASS. The unit tests fail on the unfixed code with the reported refusal. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-28-constant-override-flat-below-10-9.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-constant-override-on-mendix9.mdl | 30 ++++++ .../settings_mendix9_constant_value_test.go | 101 ++++++++++++++++++ mdl/settingsoverlay/settingsoverlay.go | 8 +- modelsdk/canon/versionshape.go | 38 ++++++- .../mpr/writer_version_shape_1163_test.go | 71 ++++++++++++ 8 files changed, 248 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json create mode 100644 mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl create mode 100644 mdl/backend/modelsdk/settings_mendix9_constant_value_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json new file mode 100644 index 000000000..27ca2e65a --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.8 a new constant override was refused: `alter settings constant @M.C value '…' in configuration 'Default'` → `this Mendix 9.24.30 project cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)` (14-project-settings on 9.24)", "cause": "settingsoverlay writes a NEW override in the nested shape (Settings$SharedValue under SharedOrPrivateValue), which is the only shape 10.9+ reads; below 10.9 the override is `{$ID, $Type, ConstantId, Value}` with a flat string. The overlay already kept an existing flat override flat, so only a new one failed", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: lowerConstantValue), `mdl/settingsoverlay/settingsoverlay.go` (comment)", "insight": "**`mx convert` on the older version is a silent-data-loss detector, not just a key oracle**: given the nested form, 9.24.30 kept the override but emptied it (`Value: \"\"`) and still reported success — an unlowered write that got past the floor would have shipped an empty override with mx check at 0 errors. Measure the VALUE after convert, not only the key set. A private value (Settings$PrivateValue, workstation-local) has no flat form and is refused naming the constant. The reader (settings_read.go constantValueOf) already preferred the flat Value, so no reader change was needed — TestProjectSettings_Mendix9FlatConstantValue pins it. Measured: bug-test on a fresh 9.24.30 app → describe settings prints 'broker:9092', mx check 0 errors, mx convert keeps the value and changes no key in the settings unit. 14-project-settings 9.24 FAIL→PASS; 10.24.24 and 11.13 PASS. Tests: TestNormalize_ConstantValueIsFlatBelowMendix10_9, TestNormalize_RefusesAPrivateConstantValueBelowMendix10_9; mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl", "refs": ["#1163", "#801"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index bcd5486ec..5437a9653 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 3f773a76f..6256f085e 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -186,12 +186,14 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | +| A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; - conditional data view editability below 10.7; - a placeholder with parameters below 10.11; - a data view fed by another widget below 10.0; +- a private constant value in a configuration below 10.9; - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl b/mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl new file mode 100644 index 000000000..a0d8db719 --- /dev/null +++ b/mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl @@ -0,0 +1,30 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.8 a new constant override in a +-- configuration was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+) +-- +-- Before 10.9 an override's value is a flat Value string on the +-- Settings$ConstantValue; from 10.9 it is a Settings$SharedValue nested under +-- SharedOrPrivateValue (mendixmodelsdk). `mx convert` 9.24.30 turns the nested +-- form into `{$ID, $Type, ConstantId, Value: ""}`: it does not know it, and the +-- override silently becomes empty. mxcli builds the nested form and lowers it +-- on the write. A private value (stored on the developer's workstation) has no +-- flat form and is refused below 10.9. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec, +-- then `describe settings` shows the override as 'broker:9092', and `mxcli +-- docker check` reports the blank app's error count. + +create module Probe1163Const; + +create constant Probe1163Const.ServerUrl ( Type: string, DefaultValue: 'localhost:9092' ); + +-- A new override: built nested, stored flat. +alter settings constant @Probe1163Const.ServerUrl value 'kafka:9092' + in configuration 'Default'; + +-- An existing flat override is updated in place. +alter settings constant @Probe1163Const.ServerUrl value 'broker:9092' + in configuration 'Default'; diff --git a/mdl/backend/modelsdk/settings_mendix9_constant_value_test.go b/mdl/backend/modelsdk/settings_mendix9_constant_value_test.go new file mode 100644 index 000000000..b574c4221 --- /dev/null +++ b/mdl/backend/modelsdk/settings_mendix9_constant_value_test.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/bsonutil" + "github.com/mendixlabs/mxcli/mdl/settingsoverlay" + "github.com/mendixlabs/mxcli/modelsdk/bsoncompat" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// seedFlatConstantValue gives the fixture's Default configuration one constant +// override in the shape Mendix stores below 10.9 (ako/mxcli#1163; `mx convert` +// 9.24.30): `{$ID, $Type, ConstantId, Value}`, the value a flat string. +func seedFlatConstantValue(t *testing.T, proj, constantID, value string) { + t.Helper() + r, err := mmpr.OpenWithOptions(proj, mmpr.OpenOptions{ReadOnly: false}) + if err != nil { + t.Fatalf("open fixture: %v", err) + } + defer r.Close() + refs, err := r.ListUnitsByType("Settings$ProjectSettings") + if err != nil || len(refs) != 1 { + t.Fatalf("ListUnitsByType(Settings$ProjectSettings) = %v, %v", refs, err) + } + raw, err := r.GetRawUnitBytes(refs[0].ID) + if err != nil { + t.Fatalf("GetRawUnitBytes: %v", err) + } + var doc bson.M + if err := bsoncompat.UnmarshalDocM(raw, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + seeded := false + for _, part := range settingsoverlay.ArrayElements(doc["Settings"]) { + if part["$Type"] != "Settings$ConfigurationSettings" { + continue + } + cfgs := settingsoverlay.ArrayElements(part["Configurations"]) + if len(cfgs) == 0 { + t.Fatalf("fixture has no configurations to seed") + } + cfgs[0]["ConstantValues"] = bson.A{int32(3), bson.D{ + {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, + {Key: "$Type", Value: "Settings$ConstantValue"}, + {Key: "ConstantId", Value: constantID}, + {Key: "Value", Value: value}, + }} + seeded = true + } + if !seeded { + t.Fatalf("fixture has no Settings$ConfigurationSettings part") + } + contents, err := bson.Marshal(bsonutil.OrderStorageValue(doc)) + if err != nil { + t.Fatalf("marshal: %v", err) + } + if err := mmpr.NewWriterWithReader(r).UpdateRawUnit(refs[0].ID, contents); err != nil { + t.Fatalf("UpdateRawUnit: %v", err) + } +} + +// A Mendix 9 constant override is read from its flat Value, and updating it +// keeps that shape rather than introducing the nested one 9.x does not have. +func TestProjectSettings_Mendix9FlatConstantValue(t *testing.T) { + proj := copyFixture(t) + const id = "MyFirstModule.ServerUrl" + seedFlatConstantValue(t, proj, id, "kafka:9092") + + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + ps, err := b.GetProjectSettings() + if err != nil { + t.Fatalf("GetProjectSettings: %v", err) + } + cvs := ps.Configuration.Configurations[0].ConstantValues + if len(cvs) != 1 || cvs[0].ConstantId != id || cvs[0].Value != "kafka:9092" { + t.Fatalf("flat override read as %+v, want %s = kafka:9092", cvs, id) + } + + cvs[0].Value = "broker:9092" + if err := b.UpdateProjectSettings(ps); err != nil { + t.Fatalf("UpdateProjectSettings: %v", err) + } + if err := b.Disconnect(); err != nil { + t.Fatalf("disconnect: %v", err) + } + after := settingsoverlay.ArrayElements(readConfiguration(t, proj)["ConstantValues"]) + if len(after) != 1 || after[0]["Value"] != "broker:9092" { + t.Fatalf("flat override after update = %#v, want Value broker:9092", after) + } + if _, nested := after[0]["SharedOrPrivateValue"]; nested { + t.Errorf("updating a flat override added the nested shape: %#v", after[0]) + } +} diff --git a/mdl/settingsoverlay/settingsoverlay.go b/mdl/settingsoverlay/settingsoverlay.go index a11bfa375..1434bf917 100644 --- a/mdl/settingsoverlay/settingsoverlay.go +++ b/mdl/settingsoverlay/settingsoverlay.go @@ -343,9 +343,11 @@ func SetWebUISettings(ws *model.WebUISettings, raw map[string]any) map[string]an // ConstantValues rebuilds a configuration's ConstantValues list, updating each // override in the slot it is already stored in so its value shape survives. -// Studio Pro and mxbuild only read the nested SharedOrPrivateValue; a flat "Value" -// is a legacy shape mxcli's reader tolerates but the platform ignores, so writing -// every override flat made all constant overrides look empty in Studio Pro. +// From Mendix 10.9 Studio Pro and mxbuild only read the nested +// SharedOrPrivateValue, so writing every override flat made all constant +// overrides look empty in Studio Pro. Below 10.9 the flat "Value" is the only +// shape: a new override is still built nested here, and the write lowers it +// (canon.LowerToProjectShape, ako/mxcli#1163). func ConstantValues(cvs []*model.ConstantValue, raw any) bson.A { rawValues := ArrayElements(raw) byConstant := make(map[string]map[string]any, len(rawValues)) diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 31be06cae..399afe049 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -38,6 +38,10 @@ import ( // CE6050 "The selected entity must be configured with a page parameter". // - ForceFullObjects, declared on the abstract Forms$DataSource from 10.8.0: // every data source carried it, and the floor never saw it. +// - Settings$ConstantValue: a Settings$SharedValue under SharedOrPrivateValue +// from 10.9.0; before, the value as a flat Value string, `{$ID, $Type, +// ConstantId, Value}` (`mx convert` 9.24.30, which empties the nested form +// it does not know). // - Workflows$ PersistentId, from 10.21.0 on the workflow and on the abstract // WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows // exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's @@ -59,9 +63,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { namedParameter: !PropertyDeclared("Forms$EntityPathSource", "SourceVariable", *pv), noFullObjects: !PropertyDeclared("Forms$DataSource", "ForceFullObjects", *pv), noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), + flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID { + !l.noPersistentID && !l.flatConstant { return contents, nil } var doc bson.D @@ -82,6 +87,7 @@ type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool + flatConstant bool changed bool } @@ -113,6 +119,10 @@ func (l *shapeLowering) walk(v any) (any, error) { if err = l.lowerPlaceholder(x, typ); err != nil { return nil, err } + case l.flatConstant && typ == "Settings$ConstantValue": + if x, err = l.lowerConstantValue(x); err != nil { + return nil, err + } } if l.noFullObjects { if x, err = l.dropForceFullObjects(x, typ); err != nil { @@ -271,6 +281,32 @@ func (l *shapeLowering) dropForceFullObjects(d bson.D, typ string) (bson.D, erro return append(append(bson.D{}, d[:i]...), d[i+1:]...), nil } +// lowerConstantValue replaces a constant override's nested SharedOrPrivateValue +// with the flat Value older versions store, in its place. A flat Value already +// beside it (the overlay clears one to "" when it writes the nested form) takes +// the nested value. A private value — kept on the developer's workstation, which +// the flat form cannot say — is refused. +func (l *shapeLowering) lowerConstantValue(d bson.D) (bson.D, error) { + i := shapeIndex(d, "SharedOrPrivateValue") + if i < 0 { + return d, nil + } + inner, _ := d[i].Value.(bson.D) + if t, _ := shapeString(inner, "$Type"); t == "Settings$PrivateValue" { + constant, _ := shapeString(d, "ConstantId") + return nil, fmt.Errorf("constant %q has a private value in this configuration, which Mendix introduced in 10.9 "+ + "(this project is %s): a constant value there is shared, stored in the model", constant, l.pv) + } + value, _ := shapeString(inner, "Value") + l.changed = true + if j := shapeIndex(d, "Value"); j >= 0 { + d[j].Value = value + return append(append(bson.D{}, d[:i]...), d[i+1:]...), nil + } + d[i] = bson.E{Key: "Value", Value: value} + return d, nil +} + // dropPersistentID removes a workflow element's PersistentId below 10.21, where // none declares it. Whatever it holds is dropped, not refused: a PersistentId is // minted by the writer (a fresh GUID on every build), never stated by a script, diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index a916b70ea..eb00c2189 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -164,6 +164,77 @@ func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { } } +// settingsWithConstantValue is the project settings unit as `alter settings +// constant … in configuration` writes a new override: the value nested in a +// Settings$SharedValue (or a Settings$PrivateValue, which holds none). flat is +// the empty flat Value the overlay leaves beside a nested one, "-" for none. +func settingsWithConstantValue(valueType, value, flat string) bson.D { + inner := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: valueType}} + if valueType == "Settings$SharedValue" { + inner = append(inner, bson.E{Key: "Value", Value: value}) + } + cv := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ConstantValue"}, + {Key: "ConstantId", Value: "M.ServerUrl"}, {Key: "SharedOrPrivateValue", Value: inner}} + if flat != "-" { + cv = append(cv, bson.E{Key: "Value", Value: flat}) + } + config := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ServerConfiguration"}, + {Key: "ConstantValues", Value: bson.A{int32(3), cv}}, {Key: "Name", Value: "Default"}} + part := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ConfigurationSettings"}, + {Key: "Configurations", Value: bson.A{int32(3), config}}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ProjectSettings"}, + {Key: "Settings", Value: bson.A{int32(2), part}}} +} + +// A configuration's constant override is a flat Value string below 10.9 and a +// Settings$SharedValue under SharedOrPrivateValue from it (mendixmodelsdk: +// Value deleted, SharedOrPrivateValue introduced 10.9.0). `mx convert` 9.24.30 +// turns the nested form into `{$ID, $Type, ConstantId, Value: ""}` — it does +// not know the nested value, so the override silently became empty, and mxcli +// refused it ("cannot store Settings$ConstantValue.SharedOrPrivateValue +// (Mendix 10.9.0+)"). +func TestNormalize_ConstantValueIsFlatBelowMendix10_9(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + flat bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 8, 0), true}, + {pv(10, 9, 0), false}, + {pv(11, 13, 0), false}, // control + } { + for _, flat := range []string{"-", ""} { + d, err := normalizeShape(t, tt.v, settingsWithConstantValue("Settings$SharedValue", "kafka:9092", flat)) + if err != nil { + t.Errorf("%d.%d: a constant override is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + cv := find(d, "Settings$ConstantValue") + if !tt.flat { + if _, ok := get(cv, "SharedOrPrivateValue"); !ok { + t.Errorf("%d.%d: SharedOrPrivateValue was lowered: %v", tt.v.MajorVersion, tt.v.MinorVersion, keysOf(cv)) + } + continue + } + if got := strings.Join(keysOf(cv), ","); got != "$ID,$Type,ConstantId,Value" { + t.Errorf("%d.%d: constant value keys = %s, want $ID,$Type,ConstantId,Value", tt.v.MajorVersion, tt.v.MinorVersion, got) + } + if v, _ := get(cv, "Value"); v != "kafka:9092" { + t.Errorf("%d.%d: Value = %v, want kafka:9092", tt.v.MajorVersion, tt.v.MinorVersion, v) + } + } + } +} + +// A private value lives on the developer's workstation, which a flat Value +// cannot say: refused, naming the constant, not written as an empty override. +func TestNormalize_RefusesAPrivateConstantValueBelowMendix10_9(t *testing.T) { + _, err := normalizeShape(t, pv(9, 24, 30), settingsWithConstantValue("Settings$PrivateValue", "", "-")) + if err == nil || !strings.Contains(err.Error(), "M.ServerUrl") { + t.Errorf("a private constant value on 9.24 was not refused naming the constant: %v", err) + } +} + // Studio Pro's property set for a workflow on 9.24 includes the two state-change // event handlers Mendix removed in 11.0: `mx convert` 9.24.30 adds both as null // to a workflow mxcli wrote, as 10.24.24 does — a property-name difference From de548c61da5d2eee0d94864b18b162f016ff6240 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:58:53 +0000 Subject: [PATCH 08/61] fix(canon): leave a $currentObject flow argument out below Mendix 10.10 Symptom (ako/mxcli#1163, 29-datagrid DG12 on 9.24.30): a page passing `$currentObject` to a microflow or nanoflow was refused with "this Mendix 9.24.30 project cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". The same held on 10.0-10.9. Cause: bindParameterMappingValue writes such an argument as Expression text with Variable null. Expression is 10.10.0 on both mapping types. Before 10.10 a mapping holds only Variable, a Forms$PageVariable that is required there. Measured with the floor off: 9.24 does not load the page at all ("An error occurred when trying to set the 'Variable' property of a Microflow parameter mapping ... Value cannot be null"). Studio Pro 9 stores no mapping for an argument that is the enclosing object; it infers it. The 9.24.30 starter app and Workflow Commons 2.6.0 (9.24.0) have 82 flow calls taking 1-2 parameters and no mapping on any of them. Fix: below 10.10, canon.LowerToProjectShape leaves a $currentObject (or empty) argument out of every ParameterMappings list. An argument bound through Variable is kept; the floor drops its empty Expression. Any other expression has no form there, so it is refused, naming the parameter. Measured: bug-test 1163-flow-arguments-on-mendix9.mdl on a fresh 9.24.30 app: exec ok, mx check 0 errors, mx convert changes no key on the page, and describe -> exec is Unchanged. 29-datagrid on 9.24 now stops at DG15 (page variables, 10.17, a correct refusal); the DG2 NullReferenceException in mx check is the separate CE0463 gap. Controls: 29-datagrid PASS on 10.24.24 and 11.13.0. The unit tests fail on the unfixed code with the reported refusal. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-flow-argument-expression-below-10-10.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-flow-arguments-on-mendix9.mdl | 50 +++++++ modelsdk/canon/versionshape.go | 62 ++++++++- .../mpr/writer_version_shape_1163_test.go | 127 ++++++++++++++++++ 6 files changed, 241 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json create mode 100644 mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json new file mode 100644 index 000000000..78c6b027a --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.9 a page passing `$currentObject` to a microflow or nanoflow was refused: `this Mendix 9.24.30 project cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)` (29-datagrid DG12 on 9.24)", "cause": "bindParameterMappingValue writes an argument that is not a page/snippet parameter or widget as Expression text with Variable null. Expression is 10.10.0; before it a mapping holds only Variable (a Forms$PageVariable, required before 10.10), and Studio Pro 9 does not store an argument that is the enclosing object at all — it infers it", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: lowerFlowArguments on any ParameterMappings list)", "insight": "**The floor's refusal hid a worse failure**: with the floor off, 9.24 does not load the project at all — `An error occurred when trying to set the 'Variable' property of a Microflow parameter mapping … ArgumentNullException` — so dropping only Expression would have shipped an unloadable page. **'Older shape' can be 'nothing stored'**: the 9.24 starter app and Workflow Commons 2.6.0 (9.24.0, from the Marketplace) have 82 flow calls taking 1–2 parameters and NOT ONE mapping (count calls against the called flow's parameters; an empty list alone proves nothing). So below 10.10 `$currentObject` (or an empty argument) is left out, a Variable-bound one keeps its Variable (the floor drops the empty Expression), and any other expression is refused naming the parameter. Readers needed nothing: describe already prints a call with no mappings as `call microflow M.F`, and re-executing that on 9.24 is Unchanged. Seen in passing: mxcli writes a populated Forms$CallNanoflowClientAction.ParameterMappings list with marker 3 (no RegisterListMarker for NanoflowParameterMapping), Studio Pro 2. Measured: bug-test on a fresh 9.24.30 app → exec ok, mx check 0 errors, mx convert changes no key; describe → exec Unchanged. 29-datagrid on 9.24 now stops at DG15 page variables (10.17, correct); 10.24.24 and 11.13 PASS. Tests: TestNormalize_FlowArgumentsBelowMendix10_10, TestNormalize_RefusesAnExpressionArgumentBelowMendix10_10; mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl", "refs": ["#1163", "#1140"], "ce": ["CE1571"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5437a9653..2c79bb216 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 6256f085e..006e0596c 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -187,6 +187,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | +| A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; @@ -194,6 +195,7 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a placeholder with parameters below 10.11; - a data view fed by another widget below 10.0; - a private constant value in a configuration below 10.9; +- a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl b/mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl new file mode 100644 index 000000000..8934125f3 --- /dev/null +++ b/mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl @@ -0,0 +1,50 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.9 a page passing +-- `$currentObject` to a microflow or nanoflow was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+) +-- +-- An argument stored as Expression text arrived in 10.10. Before it, a +-- mapping holds only a Variable (a Forms$PageVariable, required: a null one +-- makes 9.24 fail to load the page). An argument that is the enclosing object +-- is not stored at all, because Studio Pro 9.24 infers it: Workflow Commons +-- 2.6.0 (9.24.0) has 77 flow calls taking parameters and no mapping on any. +-- Below 10.10 mxcli therefore leaves `$currentObject` out and keeps a page +-- parameter as its Variable. Any other expression (a literal, an attribute +-- path) has no form there and is refused, naming the parameter. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec, +-- then `mxcli docker check` reports the blank app's error count, and `mx +-- convert -p` changes no key on the page. + +create module Probe1163Args; + +create persistent entity Probe1163Args.Item ( + Name: String(100) +); + +create microflow Probe1163Args.ACT_Item ( $Item: Probe1163Args.Item ) +begin + log info node 'Probe1163Args' 'item'; +end; + +create nanoflow Probe1163Args.NF_Item ( $Item: Probe1163Args.Item ) +begin + log info node 'Probe1163Args' 'item'; +end; + +create page Probe1163Args.Item_Edit ( + Title: 'Item', + Layout: Atlas_Core.Atlas_Default, + Params: ( $Item: Probe1163Args.Item ) +) { + dataview dv (DataSource: $Item) { + actionbutton btnCurrent (Caption: 'Current', + Action: call microflow Probe1163Args.ACT_Item(Item = $currentObject)) + actionbutton btnNanoflow (Caption: 'Nanoflow', + Action: call nanoflow Probe1163Args.NF_Item(Item = $currentObject)) + } + actionbutton btnParam (Caption: 'Param', + Action: call microflow Probe1163Args.ACT_Item(Item = $Item)) +}; diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 399afe049..0ee201248 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -42,6 +42,13 @@ import ( // from 10.9.0; before, the value as a flat Value string, `{$ID, $Type, // ConstantId, Value}` (`mx convert` 9.24.30, which empties the nested form // it does not know). +// - Forms$MicroflowParameterMapping / NanoflowParameterMapping: an argument +// as Expression text from 10.10.0; before, only a Variable (a required +// Forms$PageVariable), and an argument that is the enclosing object is not +// stored at all — Studio Pro 9.24 infers it (Workflow Commons 2.6.0: 77 +// flow calls taking parameters, no mapping on any). mxcli writes that +// argument as Expression `$currentObject` with Variable null, which 9.24 +// cannot even load. // - Workflows$ PersistentId, from 10.21.0 on the workflow and on the abstract // WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows // exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's @@ -64,9 +71,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { noFullObjects: !PropertyDeclared("Forms$DataSource", "ForceFullObjects", *pv), noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), + variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant { + !l.noPersistentID && !l.flatConstant && !l.variableArgs { return contents, nil } var doc bson.D @@ -87,7 +95,7 @@ type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool - flatConstant bool + flatConstant, variableArgs bool changed bool } @@ -132,6 +140,11 @@ func (l *shapeLowering) walk(v any) (any, error) { if l.noPersistentID && strings.HasPrefix(typ, "Workflows$") { x = l.dropPersistentID(x) } + if l.variableArgs { + if err = l.lowerFlowArguments(x); err != nil { + return nil, err + } + } for i := range x { if x[i].Value, err = l.walk(x[i].Value); err != nil { return nil, err @@ -307,6 +320,51 @@ func (l *shapeLowering) lowerConstantValue(d bson.D) (bson.D, error) { return d, nil } +var flowParameterMappings = map[string]bool{ + "Forms$MicroflowParameterMapping": true, "Forms$NanoflowParameterMapping": true, +} + +// lowerFlowArguments rewrites the flow call arguments in d's ParameterMappings +// list, if it has one, into what a version before 10.10 stores: an argument +// bound through Variable keeps it (its empty Expression is the floor's to +// drop); an argument that is the enclosing object (`$currentObject`) — or one +// that binds nothing — is left out, which is how Studio Pro 9.24 stores it. +// Any other expression has no Variable form and is refused, naming the +// parameter. +func (l *shapeLowering) lowerFlowArguments(d bson.D) error { + i := shapeIndex(d, "ParameterMappings") + if i < 0 { + return nil + } + list, _ := d[i].Value.(bson.A) + out := bson.A{} + dropped := false + for _, el := range list { + m, ok := el.(bson.D) + typ, _ := shapeString(m, "$Type") + if !ok || !flowParameterMappings[typ] { + out = append(out, el) + continue + } + if v, _ := shapeValue(m, "Variable"); v != nil { + out = append(out, el) + continue + } + expr, _ := shapeString(m, "Expression") + if e := strings.TrimSpace(expr); e != "" && !strings.EqualFold(e, "$currentObject") { + param, _ := shapeString(m, "Parameter") + return fmt.Errorf("the argument for %s is the expression %s, which Mendix introduced in 10.10 (this project is %s): "+ + "an argument there is the enclosing object, or a page parameter, snippet parameter or widget", param, expr, l.pv) + } + dropped = true + } + if dropped { + d[i].Value = out + l.changed = true + } + return nil +} + // dropPersistentID removes a workflow element's PersistentId below 10.21, where // none declares it. Whatever it holds is dropped, not refused: a PersistentId is // minted by the writer (a fresh GUID on every build), never stated by a script, diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index eb00c2189..3d33cfcb7 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -235,6 +235,133 @@ func TestNormalize_RefusesAPrivateConstantValueBelowMendix10_9(t *testing.T) { } } +// flowMapping is one argument of a page's flow call as mxcli builds it +// (bindParameterMappingValue): a page-variable reference under Variable with an +// empty Expression, anything else as Expression text with Variable null. +func flowMapping(mappingType, param, expression, pageParam string) bson.D { + var variable any + if pageParam != "" { + variable = bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$PageVariable"}, + {Key: "PageParameter", Value: pageParam}, {Key: "SnippetParameter", Value: ""}, + {Key: "UseAllPages", Value: false}, {Key: "Widget", Value: ""}} + } + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: mappingType}, + {Key: "Expression", Value: expression}, {Key: "Parameter", Value: param}, {Key: "Variable", Value: variable}} +} + +// pageWithFlowCalls is a page with a microflow button and a nanoflow button, +// each passing the given mappings. +func pageWithFlowCalls(microflowMappings, nanoflowMappings []bson.D) bson.D { + list := func(ms []bson.D) bson.A { + a := bson.A{int32(2)} + for _, m := range ms { + a = append(a, m) + } + return a + } + mfButton := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$ActionButton"}, {Key: "Name", Value: "btnMf"}, + {Key: "Action", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$MicroflowAction"}, + {Key: "MicroflowSettings", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$MicroflowSettings"}, + {Key: "Microflow", Value: "M.ACT"}, {Key: "ParameterMappings", Value: list(microflowMappings)}}}}}} + nfButton := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$ActionButton"}, {Key: "Name", Value: "btnNf"}, + {Key: "Action", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$CallNanoflowClientAction"}, + {Key: "Nanoflow", Value: "M.NF"}, {Key: "ParameterMappings", Value: list(nanoflowMappings)}}}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$Page"}, {Key: "Name", Value: "P"}, + {Key: "Widgets", Value: bson.A{int32(2), mfButton, nfButton}}} +} + +// mappingsOf returns the mappings stored under the first $Type holder in d. +func mappingsOf(t *testing.T, d bson.D, holder string) []bson.D { + t.Helper() + list, _ := get(find(d, holder), "ParameterMappings") + var out []bson.D + for _, el := range listItems(list) { + out = append(out, el.(bson.D)) + } + return out +} + +func listItems(v any) bson.A { + a, _ := v.(bson.A) + if len(a) > 0 { + if _, marker := a[0].(int32); marker { + return a[1:] + } + } + return a +} + +// A flow argument on a page is a Variable (a Forms$PageVariable, required) below +// 10.10, and Expression text only from 10.10 (mendixmodelsdk). mxcli writes +// `$currentObject` as Expression with Variable null: the floor refused it on +// 9.24 ("cannot store Forms$MicroflowParameterMapping.Expression (Mendix +// 10.10.0+)"), and without the floor 9.24 does not load the page at all ("An +// error occurred when trying to set the 'Variable' property of a Microflow +// parameter mapping … Value cannot be null"). Studio Pro 9.24 stores no mapping +// for an argument that is the enclosing object — it infers it: Workflow +// Commons 2.6.0 (9.24.0), 75 microflow and 2 nanoflow calls taking parameters, +// none with a mapping; the 9.24.30 starter app's ShowPasswordForm(Account) +// button likewise. So below 10.10 that mapping is not written; a page +// parameter keeps its Variable and loses only the empty Expression. +func TestNormalize_FlowArgumentsBelowMendix10_10(t *testing.T) { + build := func() bson.D { + return pageWithFlowCalls( + []bson.D{flowMapping("Forms$MicroflowParameterMapping", "M.ACT.Item", "$currentObject", ""), + flowMapping("Forms$MicroflowParameterMapping", "M.ACT.Order", "", "Order")}, + []bson.D{flowMapping("Forms$NanoflowParameterMapping", "M.NF.Item", "$currentObject", "")}) + } + for _, tt := range []struct { + v *version.ProjectVersion + lowered bool + mfCount int + nfCount int + mfParams string + }{ + {pv(9, 24, 30), true, 1, 0, "M.ACT.Order"}, + {pv(10, 9, 0), true, 1, 0, "M.ACT.Order"}, + {pv(10, 10, 0), false, 2, 1, "M.ACT.Item,M.ACT.Order"}, + {pv(11, 13, 0), false, 2, 1, "M.ACT.Item,M.ACT.Order"}, // control + } { + d, err := normalizeShape(t, tt.v, build()) + if err != nil { + t.Errorf("%d.%d: the page is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + mf := mappingsOf(t, d, "Forms$MicroflowSettings") + nf := mappingsOf(t, d, "Forms$CallNanoflowClientAction") + var params []string + for _, m := range mf { + p, _ := get(m, "Parameter") + params = append(params, p.(string)) + if _, ok := get(m, "Expression"); ok == tt.lowered { + t.Errorf("%d.%d: mapping %v keeps Expression %v", tt.v.MajorVersion, tt.v.MinorVersion, p, ok) + } + if v, _ := get(m, "Variable"); tt.lowered && v == nil { + t.Errorf("%d.%d: mapping %v has no Variable, which 9.x cannot load", tt.v.MajorVersion, tt.v.MinorVersion, p) + } + } + if len(mf) != tt.mfCount || len(nf) != tt.nfCount || strings.Join(params, ",") != tt.mfParams { + t.Errorf("%d.%d: %d microflow mappings (%s) and %d nanoflow mappings, want %d (%s) and %d", + tt.v.MajorVersion, tt.v.MinorVersion, len(mf), strings.Join(params, ","), len(nf), tt.mfCount, tt.mfParams, tt.nfCount) + } + } +} + +// Any other expression — a literal, an attribute path — has no Variable form: +// refused below 10.10, naming the parameter, rather than dropped. +func TestNormalize_RefusesAnExpressionArgumentBelowMendix10_10(t *testing.T) { + for _, m := range []bson.D{ + flowMapping("Forms$MicroflowParameterMapping", "M.ACT.Flag", "true", ""), + flowMapping("Forms$NanoflowParameterMapping", "M.NF.Name", "$Order/Name", ""), + } { + _, err := normalizeShape(t, pv(9, 24, 30), pageWithFlowCalls([]bson.D{m}, nil)) + p, _ := get(m, "Parameter") + if err == nil || !strings.Contains(err.Error(), p.(string)) || !strings.Contains(err.Error(), "10.10") { + t.Errorf("an expression argument for %v on 9.24 was not refused naming it: %v", p, err) + } + } +} + // Studio Pro's property set for a workflow on 9.24 includes the two state-change // event handlers Mendix removed in 11.0: `mx convert` 9.24.30 adds both as null // to a workflow mxcli wrote, as 10.24.24 does — a property-name difference From baaeda16f358a5497b953f317c96eaaf7b78010d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 18:07:21 +0000 Subject: [PATCH 09/61] fix(executor): gate create published odata service at Mendix 10.4 Symptom (ako/mxcli#1163, 10-odata and 18-folder on 9.24.30): a published OData service was refused at write time with "this Mendix 9.24.30 project cannot store ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)". By then the script's earlier statements were already written. Cause: not the property. mxcli writes the service as ODataPublish$PublishedODataService2, a document type mendixmodelsdk introduces in 10.4.0. Before 10.4 a published OData service is a Rest$PublishedODataService, which is deleted in 10.4.0: a different document with its own children, and mxcli does not write it. The generated version data has property floors only, so the floor tripped over the first versioned key. Measured with the floor off: mx check 9.24.30 cannot load the project ("TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name ODataPublish$PublishedODataService2"). Lowering the property would have made an unloadable app. Fix: a version gate. `create [or modify] published odata service` needs integration.published_odata_service, 10.4.0. `check -p` and exec's pre-flight report it as MDL-VERSION01 before anything is written, with a hint naming the 9.x document. Writing Rest$PublishedODataService for 9.x is not attempted here. Measured: bug-test 1163-published-odata-below-10-4.mdl with `check -p` on a fresh 9.24.30 app reports MDL-VERSION01. 10-odata and 18-folder on 9.24 now stop at the gate. Controls: both PASS on 10.24.24 and 11.13.0. The new gate cases fail without the gate (0 violations). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...28-published-odata-is-a-10-4-document.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-published-odata-below-10-4.mdl | 42 +++++++++++++++++++ mdl/executor/version_gates.go | 11 ++++- mdl/executor/version_gates_test.go | 11 ++++- sdk/versions/mendix-10.yaml | 11 +++++ sdk/versions/mendix-11.yaml | 4 ++ 8 files changed, 79 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json create mode 100644 mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json new file mode 100644 index 000000000..87ed75f19 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-28", "symptom": "On Mendix 9.x `create published odata service` was refused at write time with `this Mendix 9.24.30 project cannot store ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)` (10-odata and 18-folder on 9.24), after the earlier statements of the script were already written", "cause": "mxcli writes a published OData service as ODataPublish$PublishedODataService2, a document type introduced in 10.4.0; before it the service is Rest$PublishedODataService (deleted 10.4.0), a different document with its own children. The generated version data has no type-level floor, so the property floor could only trip over the first versioned key and named the wrong thing", "file": "`mdl/executor/version_gates.go` (CreateODataServiceStmt gate), `sdk/versions/mendix-10.yaml` / `mendix-11.yaml` (integration.published_odata_service 10.4.0)", "insight": "**Before lowering a refused property, check the type it sits on exists in that version.** Lowering IncludeMetadataByDefault was the obvious fix and would have made it worse: with the floor off, mx check 9.24.30 stops loading the project at all (`TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name ODataPublish$PublishedODataService2`). mendixmodelsdk says it directly: `PublishedODataService2.versionInfo = {introduced: \"10.4.0\"}`, `rest.PublishedODataService` `deleted: \"10.4.0\"`. The generated tables carry property floors only, so a whole-type floor has to be a version gate (check-time, MDL-VERSION01, before anything is written) — the same gap lets Workflows$EndOfParallelSplitPathActivity (10.21) through on 9.x. Measured: 10-odata and 18-folder on 9.24 now fail at the gate (no write); on 10.24.24 and 11.13 both PASS. Tests: TestCheckVersionGates_ReportsEveryHandlerGate (published odata on 9.24.30 and 10.3), TestCheckVersionGates_UngatedFormsPass (10.4 passes); mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl", "refs": ["#1163", "#1434"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c79bb216..8982c988e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **`create published odata service` on Mendix 9 and 10.0–10.3 is refused up front, naming the real reason** (ako/mxcli#1163) — it was refused at write time for "ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)". That property was not the problem: the whole document mxcli writes is a 10.4 type, and Mendix 9.24 cannot load a project that holds one. Before 10.4 a published OData service is a different document that mxcli does not write. `check -p` and `exec` now report MDL-VERSION01 before writing anything. - **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 006e0596c..e6b4cd22a 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -200,6 +200,8 @@ What an older shape cannot hold is refused, naming the element, rather than drop Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. +A published OData service is not an older shape of the same document. Before 10.4 it is a different document (`Rest$PublishedODataService`, not `ODataPublish$PublishedODataService2`), which mxcli does not write. `create published odata service` is refused below 10.4 with MDL-VERSION01, by `check -p` and by `exec` before anything is written. Mendix 9.24 cannot load a project that holds the 10.4 document. + ### Widget Type Definitions Pluggable widget PropertyTypes change between Mendix versions. A widget template extracted from 11.6 may have more or fewer properties than the widget installed in a 10.24 project. The [widget augmentation system](../internals/widget-templates.md) handles this by reconciling templates against the project's `.mpk` files. diff --git a/mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl b/mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl new file mode 100644 index 000000000..d42e11fea --- /dev/null +++ b/mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl @@ -0,0 +1,42 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x a published OData service was refused with +-- a message naming the wrong thing: +-- +-- this Mendix 9.24.30 project cannot store +-- ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+) +-- +-- That property is not the problem. The whole document mxcli writes, +-- ODataPublish$PublishedODataService2, was introduced in 10.4.0 +-- (mendixmodelsdk). Before 10.4 a published OData service is a +-- Rest$PublishedODataService, a different document that mxcli does not +-- write. With the floor off, mx check 9.24.30 cannot load the project +-- ("TypeCacheUnknownTypeException: ... ODataPublish$PublishedODataService2"), +-- so lowering the property would have produced an unloadable app. +-- +-- `mxcli check -p <9.24 app>` now reports MDL-VERSION01 "create +-- published odata service" before anything is written, and exec refuses the +-- same way. On 10.4 and later the service is written as before. + +create module Probe1163OData; + +create persistent entity Probe1163OData.Customer ( + Name: String(100), + Email: String(200) unique +); + +create published odata service Probe1163OData.CustomerAPI ( + Path: 'odata/customers/', + Version: '1.0.0', + ODataVersion: OData4, + Namespace: 'Probe1163OData.Customers', + ServiceName: 'Customer Service' +) +{ + publish entity Probe1163OData.Customer as 'Customers' ( + ReadMode: source + ) + expose ( + Email as 'customerId' (KEY, Filterable, Sortable), + Name + ); +}; diff --git a/mdl/executor/version_gates.go b/mdl/executor/version_gates.go index 8d6fe0ae5..33514acad 100644 --- a/mdl/executor/version_gates.go +++ b/mdl/executor/version_gates.go @@ -53,11 +53,18 @@ func statementVersionGates(stmt ast.Statement) []versionGate { return []versionGate{{"integration", "published_rest_grant_revoke", "grant access on published rest service", "upgrade your project to 10.0+"}} case *ast.CreateODataServiceStmt: + // The document mxcli writes, ODataPublish$PublishedODataService2, is + // 10.4+; before, a published OData service is a Rest$PublishedODataService, + // a different document mxcli does not write (ako/mxcli#1163). + gates := []versionGate{{"integration", "published_odata_service", "create published odata service", + "Before Mendix 10.4 a published OData service is a different document " + + "(Rest$PublishedODataService), which mxcli does not write. Publish it in Studio Pro, or upgrade the project."}} if s.SupportsGraphQL { - return []versionGate{{"integration", "odata_graphql", "SupportsGraphQL on a published OData service", + gates = append(gates, versionGate{"integration", "odata_graphql", "SupportsGraphQL on a published OData service", "Publishing a service as GraphQL as well arrived in Studio Pro 10.14. " + - "Remove SupportsGraphQL to publish OData only."}} + "Remove SupportsGraphQL to publish OData only."}) } + return gates case *ast.CreateMessageDefinitionStmt: return []versionGate{messageDefinitionDocumentGate("create message definition")} case *ast.AlterSettingsStmt: diff --git a/mdl/executor/version_gates_test.go b/mdl/executor/version_gates_test.go index 18b555be7..7b320b2e0 100644 --- a/mdl/executor/version_gates_test.go +++ b/mdl/executor/version_gates_test.go @@ -32,6 +32,10 @@ func TestCheckVersionGates_ReportsEveryHandlerGate(t *testing.T) { {"alter published rest", &ast.AlterPublishedRestServiceStmt{}, [3]int{9, 24, 0}, "alter published rest service"}, {"grant published rest", &ast.GrantPublishedRestServiceAccessStmt{}, [3]int{9, 24, 0}, "grant access on published rest service"}, {"odata graphql", &ast.CreateODataServiceStmt{SupportsGraphQL: true}, [3]int{10, 13, 0}, "SupportsGraphQL"}, + // ako/mxcli#1163: the document mxcli writes is 10.4+; 9.24.30 does not + // load a project holding one (TypeCacheUnknownTypeException). + {"published odata", &ast.CreateODataServiceStmt{}, [3]int{9, 24, 30}, "create published odata service"}, + {"published odata 10.3", &ast.CreateODataServiceStmt{}, [3]int{10, 3, 0}, "create published odata service"}, {"message definition document", &ast.CreateMessageDefinitionStmt{}, [3]int{11, 14, 0}, "create message definition"}, {"workflow group", &ast.AlterSettingsStmt{Section: "WORKFLOWS", AddGroup: true, GroupName: "G"}, [3]int{11, 1, 0}, "add group"}, {"view entity", &ast.CreateViewEntityStmt{}, [3]int{10, 17, 0}, "create view entity"}, @@ -104,14 +108,13 @@ func TestCheckVersionGates_ReportsEveryHandlerGate(t *testing.T) { // Statements with no gate, or a gated statement without the gated clause, are // not reported — `synchronize all` on 9.0, a page without parameters, OData -// without GraphQL. +// without GraphQL from 10.4. func TestCheckVersionGates_UngatedFormsPass(t *testing.T) { prog := &ast.Program{Statements: []ast.Statement{ &ast.CreateModuleStmt{Name: "M"}, &ast.CreateMicroflowStmt{}, &ast.CreateNanoflowStmt{Body: []ast.MicroflowStatement{&ast.SynchronizeStmt{SyncType: "All"}}}, &ast.CreatePageStmtV3{}, - &ast.CreateODataServiceStmt{}, &ast.AlterSettingsStmt{Section: "MODEL"}, // No EXPORT LEVEL clause: the stored level is carried (#1498). &ast.CreateEntityStmt{Attributes: []ast.Attribute{{Name: "A"}}}, @@ -122,6 +125,10 @@ func TestCheckVersionGates_UngatedFormsPass(t *testing.T) { if vs := versionGateViolations(versionCtx(t, 9, 0, 0), prog); len(vs) != 0 { t.Errorf("ungated forms reported on 9.0: %+v", vs) } + odata := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, &ast.CreateODataServiceStmt{}}} + if vs := versionGateViolations(versionCtx(t, 10, 4, 0), odata); len(vs) != 0 { + t.Errorf("a published OData service without GraphQL reported on 10.4: %+v", vs) + } } // One source of truth: a handler that called checkFeature with its own feature diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index adebf0d35..207443cab 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -167,6 +167,17 @@ features: mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." odata_client: min_version: "10.0.0" + published_odata_service: + # mxcli writes a published OData service as ODataPublish$PublishedODataService2, + # introduced in 10.4.0 (mendixmodelsdk); before, it is a + # Rest$PublishedODataService, a different document with its own children, + # which mxcli does not write. Measured: mx check 9.24.30 cannot load a + # project holding a PublishedODataService2 — + # "TypeCacheUnknownTypeException: ... ODataPublish$PublishedODataService2" + # (ako/mxcli#1163). Below this the property floor used to refuse + # IncludeMetadataByDefault instead, which named the wrong thing. + min_version: "10.4.0" + mdl: "CREATE PUBLISHED ODATA SERVICE Module.Name (...) { PUBLISH ENTITY ... }" odata_graphql: # A published OData service can also answer GraphQL — one boolean, same # resources, an extra endpoint. Introduced in Studio Pro 10.14 as an diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index 687a18b3b..02a08812e 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -179,6 +179,10 @@ features: mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." odata_client: min_version: "10.0.0" + published_odata_service: + # ODataPublish$PublishedODataService2 is 10.4+; see mendix-10.yaml. + min_version: "10.4.0" + mdl: "CREATE PUBLISHED ODATA SERVICE Module.Name (...) { PUBLISH ENTITY ... }" odata_graphql: # A published OData service can also answer GraphQL — one boolean, same # resources, an extra endpoint. Introduced in Studio Pro 10.14 as an From c190ea705e6c231cd27ae1316d18320942e35262 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 18:07:44 +0000 Subject: [PATCH 10/61] docs(wiki): a version refusal is not always an older shape The 9.24 doctype run for ako/mxcli#1163 split five refusals three ways: a wrong floor in the generated data (AggregateFunction), a document type the version lacks (PublishedODataService2, where lowering would have produced an unloadable project), and an older shape that stores nothing (a flow argument that is the enclosing object). Record what settled each one: the SDK's version data, Studio Pro-authored bytes from a Marketplace .mpk, and `mx convert -p` on the older version against a floor-off build, comparing values as well as keys. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../bug-patterns/the-platform-ci-does-not-run.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md b/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md index 1b1116a8d..8e4c471e1 100644 --- a/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md +++ b/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md @@ -91,6 +91,18 @@ every unit write passes (`canon.LowerToProjectShape`, before the floor), and to read both shapes. A starter app is the cheapest source of the older shape's real bytes: `mxcli new --version 9.24.30.51827` authored by Studio Pro. +A refusal is not always an older shape, though. In the next 9.24 run (#1163), one +of five refusals was the floor's own data being wrong: an aggregate's function had +been given the row of the fold `reduce` added in 10.0. One was a document type the +version does not have: a published OData service before 10.4. That needs a gate, +because lowering its property would have produced a project that does not load. +For one more, the older shape was "nothing stored": Studio Pro 9 infers a flow +argument that is the enclosing object. Three things settled each one: the SDK's +version data, Studio Pro-authored bytes (a Marketplace `.mpk` holds a whole +`project.mpr` of its Mendix version), and `mx convert -p` on the older version +run against a floor-off build. Convert strips undeclared keys and empties values +it does not understand, so compare values as well as key sets. + **mxcli sometimes generates the environment it then fails in.** The non-root devcontainer that could not provision PostgreSQL was scaffolded by `mxcli new` — the same code emits that base image, installs postgresql, and runs as `vscode`. From 42778ae484e3125a7e8074808f08ffad2b1dc372 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:49:15 +0000 Subject: [PATCH 11/61] fix(test): git doctor tests no longer read the developer's git config `git remote get-url` applies url..insteadOf, and doctorEnv left the system, global and GIT_CONFIG_COUNT/KEY/VALUE configuration in place. In a sandbox that rewrites git@github.com: to https://github.com/ through the environment, the SSH remote the tests add already read back as HTTPS, and TestGitDoctor_SSHRemoteRewrittenByFix and TestGitDoctor_SSHURLFormRewrittenByFix failed `make test` on main (#1670) while CI was green. A ~/.gitconfig rewriting https to git@ fails the same two tests the other way. doctorEnv now sets GIT_CONFIG_GLOBAL to the null device, GIT_CONFIG_NOSYSTEM=1 and GIT_CONFIG_COUNT=0. Control: before the fix both tests fail under each rewrite separately (environment, and HOME with an https->git@ insteadOf); with an empty HOME and no environment rewrite they pass. After it, both rewrites pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...026-10-28-git-doctor-tests-read-ambient-insteadof.json | 1 + cmd/mxcli/gitdoctor_test.go | 8 ++++++++ 2 files changed, 9 insertions(+) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json new file mode 100644 index 000000000..85b1854a1 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json @@ -0,0 +1 @@ +{"area":"cmd/mxcli","date":"2026-10-28","symptom":"`make test` failed on main right after #1670 (git doctor) merged, in a hosted sandbox only: TestGitDoctor_SSHRemoteRewrittenByFix and TestGitDoctor_SSHURLFormRewrittenByFix reported `remote url = \"https://github.com/org/app.git\\n\", want https://github.com/org/app`; CI was green","cause":"`git remote get-url` applies url..insteadOf, and the tests ran git under the developer's own configuration. The sandbox sets GIT_CONFIG_COUNT/KEY/VALUE rewriting git@github.com: and ssh://git@github.com/ to https://github.com/, so the SSH remote the test added already read back as HTTPS and --fix never touched it; a ~/.gitconfig rewriting https://github.com/ to git@github.com: (the usual developer setup) fails the same two tests the other way, `git@github.com:org/app`","file":"`cmd/mxcli/gitdoctor_test.go` (`doctorEnv`: GIT_CONFIG_GLOBAL=os.DevNull, GIT_CONFIG_NOSYSTEM=1, GIT_CONFIG_COUNT=0)","insight":"A test that shells out to git inherits three configuration sources besides the repository's — system, global, and the GIT_CONFIG_COUNT environment — and isolating HOME or the identity variables covers none of the third. Reproduce each source separately, because they fail in opposite directions: env rewrite on this host, then `env -u GIT_CONFIG_* HOME=git@ insteadOf>`, with an empty HOME as the passing control. Both failed before the fix and pass after. Not fixed, and a product question rather than a test one: with a global https->git@ rewrite, `git doctor` reads the effective URL, so it reports SSH and `--fix` sets an HTTPS URL that the rewrite turns straight back into SSH, every run","refs":["ako/mxcli#1670"]} diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index c0e0393f4..fb2cf56c1 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -23,6 +23,14 @@ import ( func doctorEnv(t *testing.T, dir string) { t.Helper() requireGit(t) + // The repository under test is the only git configuration these tests may + // see. `git remote get-url` applies url..insteadOf, so a rewrite in + // ~/.gitconfig (https -> git@, common on developer machines) or in the + // GIT_CONFIG_COUNT/KEY/VALUE environment (git@ -> https, set by hosted + // sandboxes) changed the URL the remote tests read back. + t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) + t.Setenv("GIT_CONFIG_NOSYSTEM", "1") + t.Setenv("GIT_CONFIG_COUNT", "0") t.Setenv("GIT_CEILING_DIRECTORIES", filepath.Dir(dir)) t.Setenv("GIT_AUTHOR_NAME", "t") t.Setenv("GIT_AUTHOR_EMAIL", "t@example.com") From a6f3dfad1255de0a084fa7ce91d60e19c42d891d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:29:12 +0000 Subject: [PATCH 12/61] fix(make): fmt-check uses the toolchain's gofmt, not the one on PATH go.mod's toolchain line makes `go` (and `make fmt`) switch to go1.27.3, but fmt-check ran a bare `gofmt`, which is whatever Go is installed. With go1.24.7 installed, `make lint` failed main on a file CI accepts (mdl/backend/mfmutator/splice_handler_test.go) and passed a file CI then rejected (ako/mxcli#1759). GOFMT defaults to $(go env GOROOT)/bin/gofmt. Control: `make fmt-check GOFMT=/usr/local/go/bin/gofmt` still fails on splice_handler_test.go; the default passes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- Makefile | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 12f80861e..4af1eae41 100644 --- a/Makefile +++ b/Makefile @@ -470,13 +470,21 @@ lint-go: fmt-check vet # Tracked files only: the ANTLR parser under mdl/grammar/parser is generated at # build time and deliberately not committed. The empty-list guard is a positive # control -- a check that inspected nothing must fail loudly rather than pass. +# +# The gofmt is the toolchain's, not the first one on PATH. go.mod's `toolchain` +# line makes `go` (and so `make fmt`) switch to that release, but a bare `gofmt` +# is whatever Go is installed, and gofmt's output changes between releases. With +# Go 1.24.7 installed and the toolchain at 1.27.3, the bare one failed main on a +# file CI accepts (mdl/backend/mfmutator/splice_handler_test.go) and passed a +# file CI then rejected (ako/mxcli#1759). +GOFMT ?= $(shell go env GOROOT)/bin/gofmt fmt-check: @files=$$(git ls-files '*.go' | grep -v '/parser/'); \ if [ -z "$$files" ]; then \ echo "fmt-check: found no Go files to check -- the file list is wrong"; \ exit 1; \ fi; \ - unformatted=$$(gofmt -l $$files); \ + unformatted=$$($(GOFMT) -l $$files); \ if [ -n "$$unformatted" ]; then \ echo "Not gofmt-clean (run 'make fmt'):"; \ echo "$$unformatted"; \ From f4398e6652509db7b981d7e9881b7eed818e2e63 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:29:12 +0000 Subject: [PATCH 13/61] build(make): add ci-local, the build-and-test job run before pushing `make build test lint` covers less than CI's build-and-test job gates on: check-mdl, check-findings, check-skill-index, check-wiki-pages and the rest ran only in CI unless someone remembered them. ci-local runs the job's steps in its order (check-skill-mdl standing for its three skill/docs MDL steps), without govulncheck, which the job installs @latest over the network, and without the integration jobs, which need mxbuild. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- Makefile | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 4af1eae41..f64a174fa 100644 --- a/Makefile +++ b/Makefile @@ -8,6 +8,7 @@ # make test-integration - Run integration tests (requires mx/mxbuild) # make test-mdl - Run MDL integration tests (requires Docker) # make lint - Lint all code (Go + TypeScript) +# make ci-local - Run what CI's build-and-test job runs, before pushing # make lint-go - Lint Go code (fmt + vet) # make lint-ts - Lint TypeScript code (tsc --noEmit) # make grammar - Regenerate ANTLR parser @@ -35,7 +36,7 @@ GO_BUILD_FLAGS = -trimpath # Clean version for VS Code extension (must be valid semver: major.minor.patch) VSCE_VERSION = $(shell echo "$(VERSION)" | sed 's/^v//; s/-.*//' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$$' || echo "0.0.0") -.PHONY: build build-debug size release clean test test-mdl check-mdl check-skill-mdl check-conformance conformance-shrink gen-migration-reference check-migration-reference check-skill-pack-js check-findings check-wiki-pages digest-status check-tunnel-deps check-test-timeouts check-widget-versions test-integration test-integration-executor test-integration-roundtrip test-integration-parity test-integration-upgrade test-integration-other grammar completions sync-skills sync-skill-packs sync-commands sync-mods sync-lint-rules sync-changelog sync-all docs documentation docs-site docs-serve vscode-ext vscode-install source-tree sbom sbom-report lint lint-go lint-ts fmt fmt-check vet skill-index check-skill-index +.PHONY: build build-debug size release clean test ci-local test-mdl check-mdl check-skill-mdl check-conformance conformance-shrink gen-migration-reference check-migration-reference check-skill-pack-js check-findings check-wiki-pages digest-status check-tunnel-deps check-test-timeouts check-widget-versions test-integration test-integration-executor test-integration-roundtrip test-integration-parity test-integration-upgrade test-integration-other grammar completions sync-skills sync-skill-packs sync-commands sync-mods sync-lint-rules sync-changelog sync-all docs documentation docs-site docs-serve vscode-ext vscode-install source-tree sbom sbom-report lint lint-go lint-ts fmt fmt-check vet skill-index check-skill-index # Helper: copy file only if content differs (avoids mtime updates that invalidate go build cache) # Usage: $(call copy-if-changed,src,dst) @@ -453,6 +454,20 @@ check-widget-versions: build # Lint all code (Go + TypeScript) lint: lint-go lint-ts check-conformance check-test-timeouts +# What the build-and-test job in .github/workflows/push-test.yml runs, in its +# order, so a push is proven against the gate it will meet rather than against +# `build test lint`, which covers less of it: check-mdl, check-findings, +# check-skill-index and the rest run only in CI unless someone remembers them. +# check-skill-mdl is a superset of the job's three skill/docs MDL steps. +# +# Left out: govulncheck (the job installs it @latest, over the network) and the +# integration jobs, which need mxbuild -- `make test-integration` runs those. +# A step added to the job belongs here too. +ci-local: build test check-tunnel-deps check-mdl check-skill-pack-js check-findings \ + check-wiki-pages check-skill-index check-test-timeouts check-skill-mdl \ + check-conformance check-migration-reference lint-go + @echo "ci-local passed (build-and-test, without govulncheck)" + # Lint Go code # # Depends on fmt-check, which VERIFIES, not on fmt, which rewrites. `go fmt ./...` From 880a2841c6e6c154baf7f9505a1b0c2427e7eb18 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:49:31 +0000 Subject: [PATCH 14/61] docs(skills): add the steward skill, the loop from PR opened to merged Holds this repository's conventions for driving a PR Claude opened to green: the user reviews before the PR is opened and it is merged on green by default (merge commit, expectedHeadSha, branch restarted from main afterwards); `make ci-local` before every push; the macOS tunnel-seam job that is cancelled before it gets a runner, and so gets one re-run; the conflicts this repo keeps hitting (CHANGELOG, generated indexes and tables); and how to treat the AI PR Review comment's Critical / Moderate / Minor sections. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .claude/skills/steward/SKILL.md | 121 ++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 .claude/skills/steward/SKILL.md diff --git a/.claude/skills/steward/SKILL.md b/.claude/skills/steward/SKILL.md new file mode 100644 index 000000000..76b305efe --- /dev/null +++ b/.claude/skills/steward/SKILL.md @@ -0,0 +1,121 @@ +--- +name: steward +description: Drive an mxcli pull request Claude opened to green and merge it. Covers when to open it, the local gate to pass before every push (make ci-local), which CI failure is a known runner flake, how to resolve the conflicts this repo keeps hitting, how to read the AI PR Review comment, and the merge-on-green default. Read before acting on a CI, review or conflict event on such a PR. +--- + +## Find it fast + +`sed -n ',p' SKILL.md` one section rather than reading the file; +`./mxcli syntax ` answers *how do I write this* for a fraction of either. + +| lines | section | +|---|---| +| 27-54 | Merge on green is the default | +| 55-72 | Before every push: `make ci-local` | +| 73-87 | Reading a red CI job | +| 88-103 | Conflicts this repo keeps hitting | +| 104-121 | The AI PR Review comment | + + +# Steward: Driving a PR to Merged + +This is a **contributor** skill (dev-only); it is not synced to user projects. +It holds this repository's conventions for the loop after a PR is opened. How +to make the change itself is `/mxcli-dev:fix-issue` and the CLAUDE.md working +rules; this file starts where they stop. + +## Merge on green is the default + +The user reviews a change **in the session, before the PR exists**. So: + +- Open a PR only after the user has seen the change summarized in the session + and asked for it. Never open one to get a review. +- Once it is open, merge it without asking again as soon as all of these hold + on the **current head**: + - every job of `Build, Test & Lint` (`.github/workflows/push-test.yml`) is + green, `integration-passed` included. A push to the PR branch runs that + workflow twice on the same commit, for the push and the pull_request + event; a job that passed in one run and was cancelled before it got a + runner in the other counts as green; + - GitHub reports no merge conflict; + - no human review thread or changes-requested review is open; + - no Critical item from the AI PR Review comment is unanswered (see below). +- Merge with a **merge commit** (not squash, not rebase), passing the head SHA + you checked as `expectedHeadSha`, so a push that lands in between is not + merged unseen. +- After the merge, unsubscribe from the PR, cancel its pending check-in, and + restart the working branch from the new main: + `git fetch origin main && git checkout -B origin/main`, then push. + The next PR starts from there; never stack new work on merged history. + +Ask instead of merging when a human reviewer has commented or requested +changes, or when the PR is someone else's. Never merge a red head, whatever +the reason it is red. + +## Before every push: `make ci-local` + +`make build test lint` is a subset of the gate CI applies. `make ci-local` +runs what the `build-and-test` job runs, in its order, minus `govulncheck`. +Run it before the first push and before every push that changes code, and +push only on a clean exit. It takes a while; run it in the background. + +- **Formatting:** `make fmt` is the fixer. `fmt-check` uses the toolchain's + gofmt (`go env GOROOT`), because the gofmt on `PATH` can be an older Go than + the `toolchain` line in `go.mod`, and its verdict differs from CI's. +- **Integration suites** need mxbuild, so `ci-local` does not run them. When + the change touches a package with integration tests, run that suite: + `make test-integration-`. +- **When the job gains a step,** add it to `ci-local` in the same PR. +- **A failure in code the change does not touch:** run the same target on + `origin/main`. If it fails there too, it is main's: fix it in its own PR + (it can go first), and never skip or disable the test to get past it. + +## Reading a red CI job + +The workflow is the source of truth for what runs; these are the jobs whose +failures need reading differently. + +- **`build-and-test`:** reproduce with `make ci-local`. +- **`integration ()`:** reproduce with `make test-integration-`. +- **`tunnel-seam-cross-platform (macos-latest | windows-latest)`:** the one + known runner flake. The hosted macOS queue sometimes cancels the job before + it gets a runner (the job shows runner id 0 and no step ran). That is a job + that died before any test body ran: re-run the failed jobs **once**. A test + failure in this job is real: it runs the tunnel seam's own tests on that + platform (ADR-0009). +- **`AI PR Review`** is always green: it posts a comment, it does not gate. + +## Conflicts this repo keeps hitting + +While the PR is open, merge `origin/main` into its head. Do not rebase or +force-push it. + +| File | Resolution | +|---|---| +| `CHANGELOG.md` `[Unreleased]` | Keep **both** sides: every entry is one line, so a conflict is two fixes landing at once. Afterwards check for a stray blank line or a duplicated entry. | +| A `SKILL.md` section index | Keep either side, then `make skill-index`. | +| `cmd/mxcli/skills/` | Never resolve by hand: `make sync-skills` regenerates it from `.claude/skills/mendix/`. | +| `docs-site/src/language/versions.md` tables | `make gen-migration-reference`. | +| `mdl/conformance/allowlist.txt` | Take the higher count on each conflicting line, then `make conformance-shrink`, which lowers every count to what the gate finds and never raises one. | +| Bug findings | They do not conflict: each finding is its own new file. | + +Run `make ci-local` on the merge result before pushing it. + +## The AI PR Review comment + +`ai-review.yml` posts **one PR comment** headed `## AI Code Review`, not review +threads. It has sections Critical Issues, Moderate Issues, Minor Issues, What +Looks Good and Recommendation. On a large PR the model sees only a selected +part of the diff, so a claim that something is missing may only mean that file +was outside the selection. + +| Section | How to treat it | +|---|---| +| Critical | Blocking. Verify each item against the code: fix and push it, or answer it with the path you traced and why it does not hold. | +| Moderate | A claim to verify, not an order. Fix it when a real caller reaches the failure and the fix is in proportion to it; otherwise say why not. | +| Minor | Optional. Never start a push for one. Carry the plainly correct ones into a push that already touches those files. | + +Answer it with **one** PR comment that gives the outcome for each Critical and +Moderate item. Sometimes nothing is posted at all: the workflow skips posting +when its API key is missing or the API call fails, and still reports success. +A missing comment is neither an approval nor a blocker. From 277bf8e54eff42a0ce3d79af7b9c971d435002ae Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 21:55:19 +0000 Subject: [PATCH 15/61] fix(git): doctor judges notes only once the remote's notes are fetched `git clone` never fetches refs/notes/*, so on a fresh clone or a CI checkout `git doctor` reported every commit as "without a note" and exited 1, while the same report said `[x] origin has refs/notes/mx_metadata`. `--fix` then wrote computed notes over commits the remote already annotates, and the push it recommends was rejected as non-fast-forward. Anyone who forced that push would overwrite the real notes on the remote. The remote check already reads the remote's notes commit with ls-remote; it is now passed on. If the local ref is not, or does not contain, that commit, the commits are not judged. The finding is the unfetched ref, given with the command that fetches it. `--fix` fetches it as a fast-forward only, so a note that exists only locally is never dropped, and stops if that fetch is refused. Test: TestGitDoctor_UnfetchedNotesAreNotMissing (bare remote, pushed note, fresh clone). Before the fix it fails with the reported symptom: "[ ] 9450425668 without a note" next to "[x] origin has refs/notes/mx_metadata", and --fix replaced the remote's note on HEAD. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...judges-notes-against-an-unfetched-ref.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_doctor.go | 11 +++-- cmd/mxcli/gitdoctor.go | 46 ++++++++++++++++++- cmd/mxcli/gitdoctor_test.go | 42 +++++++++++++++++ docs-site/src/tools/outside-studio-pro.md | 11 +++++ 6 files changed, 107 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json new file mode 100644 index 000000000..0e730ec6a --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "`mxcli git doctor` on a fresh clone (or any CI checkout) reports every commit as `[ ] without a note` and exits 1, while the same report says `[x] origin has refs/notes/mx_metadata`; `--fix` then writes computed notes over commits the remote already annotates, and the `git push origin refs/notes/mx_metadata` it tells you to run is rejected as non-fast-forward", "cause": "doctorNotes judged each commit against the LOCAL refs/notes/mx_metadata only, and nothing fetched it: git clone and a plain git fetch never fetch refs/notes/*. The remote check had already run `git ls-remote refs/notes/mx_metadata` and printed the tip, but that result was not passed on, so an empty local ref read as 'no commit has a note'", "file": "`cmd/mxcli/gitdoctor.go` (`doctorRun` passes the remote's notes commit to `doctorNotes`; `hasNotesCommit`: the local ref is, or descends from, that commit); test `TestGitDoctor_UnfetchedNotesAreNotMissing` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**Every existing notes test built its repository with `git init`, never `git clone`**, so the local notes ref was always the only one there was; the defect lives exactly in the gap between a repository that writes notes and one that receives them. A fixture with a bare remote, a pushed note and a fresh clone reproduced it on the first run. **Do not judge, and do not write, against a ref that is behind the remote**: Studio Pro's own fetch force-replaces the local notes ref with the remote's, so the remote's notes are the ones that count; a note written over a remote-annotated commit is either refused on push or thrown away on Studio Pro's next fetch. **Fetch as a fast-forward only** (`git fetch refs/notes/mx_metadata:refs/notes/mx_metadata`, no `+`): it never discards a note that exists only locally, which a force-fetch would; when the local ref has diverged the fetch is refused and the doctor says so instead of writing. Report mode stays read-only: it names the unfetched ref and the command rather than fetching", "refs": ["ako/mxcli#1670", "mendixlabs/mxcli#1669"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 12a35f81c..e97d561d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. Docs: "Working Outside Studio Pro" › *Auditing a repository*. +- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_doctor.go b/cmd/mxcli/cmd_git_doctor.go index 1e17d23f3..486c56323 100644 --- a/cmd/mxcli/cmd_git_doctor.go +++ b/cmd/mxcli/cmd_git_doctor.go @@ -29,16 +29,21 @@ Checked: (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. Other non-HTTPS URLs are reported without being changed. 3. The remote's refs/notes/mx_metadata. Absent or unreachable is a hint: - push the notes ref together with the branch. + push the notes ref together with the branch. When the remote has notes + the local ref does not hold (git clone never fetches refs/notes/*), the + commits are not judged: that is the finding, and --fix fetches the ref, + as a fast-forward only. 4. Every commit's mx_metadata note. Missing notes make Studio Pro show "(unknown)"; a malformed note it cannot read at all. --fix computes real notes from the model (with -p), and falls back to a template note with a --version ModelerVersion where the model at a commit cannot be read. Notes Studio Pro wrote are never touched. -Exits 1 while an issue remains, so the audit can gate CI. --fix does not push: -push the branch and the notes ref together yourself. +Exits 1 while an issue remains, so the audit can gate CI; fetch the notes +first there. --fix does not push: push the branch and the notes ref together +yourself. + git fetch refs/notes/mx_metadata:refs/notes/mx_metadata git push refs/notes/mx_metadata`, Example: ` mxcli git doctor mxcli git doctor -p app.mpr --fix diff --git a/cmd/mxcli/gitdoctor.go b/cmd/mxcli/gitdoctor.go index 48f045728..d01dfbaca 100644 --- a/cmd/mxcli/gitdoctor.go +++ b/cmd/mxcli/gitdoctor.go @@ -76,6 +76,7 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W fmt.Fprintln(out, "Remote") remote := detectDoctorRemote(root, remoteOverride) + remoteNotes := "" // the remote's notes commit, when it has one and could be reached switch { case remote == "": fmt.Fprintln(out, " [i] no remote configured; skipping remote checks") @@ -112,12 +113,15 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W fmt.Fprintf(out, " [i] could not reach %s to check %s\n", remote, mxMetadataRef) case strings.Contains(ls, mxMetadataRef): fmt.Fprintf(out, " [x] %s has %s\n", remote, mxMetadataRef) + if f := strings.Fields(ls); len(f) > 0 { + remoteNotes = f[0] + } default: fmt.Fprintf(out, " [i] %s has no %s yet; push it together with the branch\n", remote, mxMetadataRef) } } - fn, fx, err := doctorNotes(root, mprPath, version, fix, out) + fn, fx, err := doctorNotes(root, mprPath, version, remote, remoteNotes, fix, out) if err != nil { return err } @@ -143,12 +147,35 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W // doctorNotes reports (and with fix, repairs) missing and malformed notes over // the whole current history. It returns how many it found and how many it fixed. -func doctorNotes(root, mprPath, version string, fix bool, out io.Writer) (found, fixed int, err error) { +// +// remoteNotes is the remote's notes commit ("" when it has none or could not +// be reached). Those are the notes Studio Pro reads: its fetch force-replaces +// the local ref with them. A local ref that does not contain that commit — a +// fresh clone, since git clone never fetches refs/notes/* — cannot be judged: +// every commit the remote annotates would look unannotated, and a note written +// over it could not be pushed (non-fast-forward). So the commits are not +// judged until the ref is fetched; --fix fetches it, as a fast-forward only, +// which never drops a note that exists only locally. +func doctorNotes(root, mprPath, version, remote, remoteNotes string, fix bool, out io.Writer) (found, fixed int, err error) { fmt.Fprintln(out, "mx_metadata notes") if _, err := gitOut(root, "rev-parse", "--verify", "--quiet", "HEAD"); err != nil { fmt.Fprintln(out, " [i] no commits yet") return 0, 0, nil } + if remoteNotes != "" && !hasNotesCommit(root, remoteNotes) { + found++ + fmt.Fprintf(out, " [ ] %s from %s is not fetched; the notes cannot be judged without it\n", mxMetadataRef, remote) + if !fix { + fmt.Fprintf(out, " git fetch %s %s:%s\n", remote, mxMetadataRef, mxMetadataRef) + return found, 0, nil + } + if _, err := gitOut(root, "fetch", "--quiet", remote, mxMetadataRef+":"+mxMetadataRef); err != nil { + fmt.Fprintf(out, " could not fetch it as a fast-forward (%v); if the local ref has notes the remote lacks, push or reconcile them first\n", err) + return found, 0, nil + } + fixed++ + fmt.Fprintf(out, " fetched from %s\n", remote) + } revs := strings.Fields(gitOutOr(root, "rev-list", "--reverse", "HEAD")) noted := map[string]bool{} for _, line := range strings.Split(strings.TrimRight(gitOutOr(root, "notes", "--ref="+mxMetadataRef, "list"), "\n"), "\n") { @@ -316,6 +343,21 @@ func sshToHTTPS(url string) (host, p string, ok bool) { return "", "", false } +// hasNotesCommit reports whether the local notes ref is, or contains, the +// notes commit c. False when the ref is absent or c is not in its history +// (including when c's object was never fetched). +func hasNotesCommit(root, c string) bool { + local, err := gitOut(root, "rev-parse", "--verify", "--quiet", mxMetadataRef) + if err != nil { + return false + } + if strings.TrimSpace(local) == c { + return true + } + _, err = gitOut(root, "merge-base", "--is-ancestor", c, strings.TrimSpace(local)) + return err == nil +} + func gitNoteText(root, c string) string { out, err := gitOut(root, "notes", "--ref="+mxMetadataRef, "show", c) if err != nil { diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index fb2cf56c1..be8a948d4 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -218,6 +218,48 @@ func TestGitDoctor_FixKeepsRealNotes(t *testing.T) { } } +// A fresh clone has no refs/notes/* (git clone never fetches them), while the +// remote carries the notes Studio Pro wrote. Judged against the empty local +// ref, every commit looked unannotated, and --fix wrote notes over the +// remote's that it then refused to accept (non-fast-forward). +func TestGitDoctor_UnfetchedNotesAreNotMissing(t *testing.T) { + src, _ := doctorNoteRepo(t) + doctorEnv(t, src) + gitIn(t, src, "notes", "--ref="+mxMetadataRef, "add", "-f", "-m", realStudioProNote, "HEAD") + remote := t.TempDir() + gitIn(t, remote, "init", "-q", "--bare") + gitIn(t, src, "remote", "add", "origin", remote) + gitIn(t, src, "push", "-q", "origin", "feature", mxMetadataRef) + + clone := filepath.Join(t.TempDir(), "clone") + gitIn(t, filepath.Dir(clone), "clone", "-q", "-b", "feature", remote, clone) + for _, kv := range mxConfigKeys { + gitIn(t, clone, "config", "--local", kv[0], kv[1]) + } + mprPath := filepath.Join(clone, "App.mpr") + head := strings.TrimSpace(gitOutOr(clone, "rev-parse", "HEAD")) + + var out bytes.Buffer + err := doctorRun(clone, mprPath, "", "", false, &out) + if strings.Contains(out.String(), shortCommit(head)+" without a note") { + t.Errorf("HEAD has a note on the remote; an unfetched notes ref must not report it missing:\n%s", out.String()) + } + if !errors.Is(err, errDoctorIssuesRemain) || !strings.Contains(out.String(), "git fetch origin "+mxMetadataRef) { + t.Errorf("the unfetched notes ref must be the finding, with the command that fetches it: %v\n%s", err, out.String()) + } + + out.Reset() + if err := doctorRun(clone, mprPath, "", "", true, &out); err != nil { + t.Fatalf("--fix must fetch the notes and annotate the rest: %v\n%s", err, out.String()) + } + if got := strings.TrimSpace(noteOn(t, clone, "HEAD")); got != realStudioProNote { + t.Fatalf("--fix replaced the remote's note on HEAD:\n got %s\nwant %s\n%s", got, realStudioProNote, out.String()) + } + if _, err := gitOut(clone, "push", "origin", "feature", mxMetadataRef); err != nil { + t.Errorf("the notes --fix leaves must push as a fast-forward: %v\n%s", err, out.String()) + } +} + func TestGitDoctor_TemplateWithoutProjectUsesVersionFlag(t *testing.T) { dir := t.TempDir() doctorEnv(t, dir) diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 9cf80d53a..6c38a75b9 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -227,11 +227,22 @@ mxcli git doctor --fix --version 10.21.0 | local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` | sets them | | the remote URL: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged | | the remote's `refs/notes/mx_metadata` | reported only — push the notes ref together with the branch | +| the local `refs/notes/mx_metadata` holds the remote's notes: `git clone` and a plain `git fetch` never fetch them, so a fresh clone or a CI checkout has none | fetches them, as a fast-forward only; refused when the local ref has notes the remote does not | | each commit on the checked-out branch and its mx_metadata note: missing notes make Studio Pro show `(unknown)`; a malformed note it cannot read at all | computes real notes from the model (`-p`), falling back to a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); when no version is known the newest valid note in history supplies it | Notes Studio Pro wrote are never touched, with or without `--fix`. `--fix` does not push. +The commits are judged only once the local notes ref holds the remote's. Until +then the report names the unfetched ref and the command that fetches it, rather +than calling every commit the remote annotates unannotated. In CI, fetch the +notes before the audit: + +```bash +git fetch origin refs/notes/mx_metadata:refs/notes/mx_metadata +mxcli git doctor -p app.mpr +``` + ## Commit metadata: `mxcli git note` Studio Pro attaches a git note under `refs/notes/mx_metadata` to every commit From 61af7ee22785aa61e32bd3e10232ed3b8824a8ed Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 21:57:11 +0000 Subject: [PATCH 16/61] fix(git): doctor judges the configured remote URL, not the insteadOf rewrite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `git doctor` read the remote with `git remote get-url`, which applies url..insteadOf. With a global https -> git@ rule (a common developer setup), an HTTPS remote read as SSH and the doctor exited 1. `--fix` then set the configured URL to HTTPS on every run, which the rule rewrote straight back, and dropped the URL's `.git` suffix along the way. ako/mxcli#1765 isolated the tests from this and left the product question open; this answers it. The check now reads `git config --get remote..url`, the value --fix actually writes. An SSH remote is still found and rewritten when a rule makes it read as HTTPS. When the effective URL differs, the report gives it as an [i] line naming the insteadOf rule, which --fix leaves alone. It is not counted as an issue: the rule applies to every git command on the machine, and changing the remote cannot undo it. Tests: TestGitDoctor_InsteadOfRewriteIsNotAnSSHRemote and TestGitDoctor_ConfiguredSSHRemoteFoundBehindRewrite. Both use a rule in the test repository's own config, on a host nobody rewrites. Before the fix: "origin: git@git.example.com:org/app.git — SSH; Studio Pro needs HTTPS", --fix changed the configured URL to https://git.example.com/org/app, and the configured SSH remote was missed. The two existing SSH tests also pass now in an environment that rewrites github.com URLs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...es-the-insteadof-rewritten-remote-url.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_doctor.go | 8 +-- cmd/mxcli/gitdoctor.go | 9 +++- cmd/mxcli/gitdoctor_test.go | 50 +++++++++++++++++++ docs-site/src/tools/outside-studio-pro.md | 2 +- 6 files changed, 66 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json new file mode 100644 index 000000000..608387c55 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "With a global `url.git@github.com:.insteadOf https://github.com/` rule (a common developer setup), `mxcli git doctor` reports an HTTPS remote as `origin: git@github.com:org/app.git — SSH; Studio Pro needs HTTPS`, exits 1, and `--fix` \"rewrites\" the configured URL to HTTPS on every run (dropping its `.git` suffix) without changing anything git uses", "cause": "The remote check read `git remote get-url`, which applies url..insteadOf, and --fix set `remote..url`, which the rule rewrites again on the next read, so the check could never pass. The same read made the two SSH tests depend on the developer's and the sandbox's git config (ako/mxcli#1765 isolated the tests; the product question stayed open)", "file": "`cmd/mxcli/gitdoctor.go` (`doctorRun`: judge `git config --get remote..url`; report a differing `git remote get-url` as an [i] insteadOf line); tests `TestGitDoctor_InsteadOfRewriteIsNotAnSSHRemote`, `TestGitDoctor_ConfiguredSSHRemoteFoundBehindRewrite` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**A check and its fix must read and write the same thing.** `remote get-url` is the effective URL, `remote set-url` writes the configured one; with a rewrite in between, a fix can never make its own check pass, and a loop like that is the tell. Judge what you can change (the configured URL) and report what you cannot (the rule, which applies to every git command on the machine). **Put the rule in the test repository's own config and use a host nobody rewrites** (git.example.com): the test then shows the behaviour without inheriting the developer's or the CI sandbox's rewrites of github.com, which point in opposite directions. Set GIT_SSH_COMMAND=false when the effective URL is SSH, or ls-remote reaches for the network", "refs": ["ako/mxcli#1670", "ako/mxcli#1765"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index e97d561d1..d13bae0fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. +- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_doctor.go b/cmd/mxcli/cmd_git_doctor.go index 486c56323..06fb78f18 100644 --- a/cmd/mxcli/cmd_git_doctor.go +++ b/cmd/mxcli/cmd_git_doctor.go @@ -25,9 +25,11 @@ Checked: 1. Local git config: core.autocrlf=False (a checkout must not rewrite line endings inside .mxunit files), mendix.commits-since-gc and mendix.lineEndingResetDone (Studio Pro's own bookkeeping). --fix sets them. - 2. The remote URL. Studio Pro can only use HTTPS; an SSH remote - (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. Other - non-HTTPS URLs are reported without being changed. + 2. The remote URL, as configured. Studio Pro can only use HTTPS; an SSH + remote (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. + Other non-HTTPS URLs are reported without being changed. A + url..insteadOf rule in your git config that makes git use another + URL is reported too; --fix leaves it alone. 3. The remote's refs/notes/mx_metadata. Absent or unreachable is a hint: push the notes ref together with the branch. When the remote has notes the local ref does not hold (git clone never fetches refs/notes/*), the diff --git a/cmd/mxcli/gitdoctor.go b/cmd/mxcli/gitdoctor.go index d01dfbaca..0d106e9f6 100644 --- a/cmd/mxcli/gitdoctor.go +++ b/cmd/mxcli/gitdoctor.go @@ -81,7 +81,11 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W case remote == "": fmt.Fprintln(out, " [i] no remote configured; skipping remote checks") default: - urlRaw, urlErr := gitOut(root, "remote", "get-url", remote) + // The configured URL, not `git remote get-url`: that one applies + // url..insteadOf, so a rewrite in the user's git config (https -> + // git@ is a common one) made an HTTPS remote read as SSH, and --fix set + // the configured URL on every run without changing what git uses. + urlRaw, urlErr := gitOut(root, "config", "--get", "remote."+remote+".url") if urlErr != nil { found++ fmt.Fprintf(out, " [ ] remote %s does not exist (git remote -v lists the names)\n", remote) @@ -106,6 +110,9 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W } else { fmt.Fprintf(out, " [i] %s: %s (not HTTPS; Studio Pro can only use HTTPS remotes)\n", remote, url) } + if eff, err := gitOut(root, "remote", "get-url", remote); err == nil && strings.TrimSpace(eff) != url { + fmt.Fprintf(out, " [i] a url..insteadOf rule in your git config makes git use %s instead; --fix leaves the rule alone\n", strings.TrimSpace(eff)) + } // Absent or unreachable notes ref is a hint, not an issue. ls, err := gitOut(root, "ls-remote", remote, mxMetadataRef) switch { diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index be8a948d4..1a3b49e3e 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -143,6 +143,56 @@ func TestGitDoctor_SSHURLFormRewrittenByFix(t *testing.T) { } } +// A url..insteadOf rule rewrites the URL every git command uses, not the +// URL the repository is configured with. The doctor judged the rewritten one: +// with the common https -> git@ rule in ~/.gitconfig an HTTPS remote read as +// SSH, and --fix "repaired" the configured URL on every run. +func TestGitDoctor_InsteadOfRewriteIsNotAnSSHRemote(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + t.Setenv("GIT_SSH_COMMAND", "false") // the rewritten URL is SSH: never reach the network + gitIn(t, dir, "init", "-q", "-b", "main") + for _, kv := range mxConfigKeys { + gitIn(t, dir, "config", "--local", kv[0], kv[1]) + } + gitIn(t, dir, "config", "--local", "url.git@git.example.com:.insteadOf", "https://git.example.com/") + gitIn(t, dir, "remote", "add", "origin", "https://git.example.com/org/app.git") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", false, &out); err != nil { + t.Errorf("a configured HTTPS remote is not an issue, whatever a rewrite makes of it: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "insteadOf") { + t.Errorf("the report must explain the rewrite:\n%s", out.String()) + } + + out.Reset() + if err := doctorRun(dir, "", "", "", true, &out); err != nil { + t.Errorf("--fix: %v\n%s", err, out.String()) + } + if got := configLocal(t, dir, "remote.origin.url"); got != "https://git.example.com/org/app.git" { + t.Errorf("--fix changed the configured URL to %q:\n%s", got, out.String()) + } +} + +// The configured URL is what --fix rewrites, so an SSH remote is still found +// when a rule rewrites it to HTTPS (as hosted sandboxes do). +func TestGitDoctor_ConfiguredSSHRemoteFoundBehindRewrite(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "--local", "url.https://git.example.com/.insteadOf", "git@git.example.com:") + gitIn(t, dir, "remote", "add", "origin", "git@git.example.com:org/app.git") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", true, &out); err != nil { + t.Fatalf("--fix: %v\n%s", err, out.String()) + } + if got := configLocal(t, dir, "remote.origin.url"); got != "https://git.example.com/org/app" { + t.Errorf("configured URL = %q, want https://git.example.com/org/app\n%s", got, out.String()) + } +} + // --- doctor: mx_metadata notes ---------------------------------------------- // doctorNoteRepo makes a two-commit repository holding an MPR v2 project: diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 6c38a75b9..e48b61ff9 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -225,7 +225,7 @@ mxcli git doctor --fix --version 10.21.0 | Check | `--fix` | |-------|---------| | local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` | sets them | -| the remote URL: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged | +| the remote URL as configured: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged. A `url..insteadOf` rule in your git config that makes git use another URL is reported, and left alone | | the remote's `refs/notes/mx_metadata` | reported only — push the notes ref together with the branch | | the local `refs/notes/mx_metadata` holds the remote's notes: `git clone` and a plain `git fetch` never fetch them, so a fresh clone or a CI checkout has none | fetches them, as a fast-forward only; refused when the local ref has notes the remote does not | | each commit on the checked-out branch and its mx_metadata note: missing notes make Studio Pro show `(unknown)`; a malformed note it cannot read at all | computes real notes from the model (`-p`), falling back to a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); when no version is known the newest valid note in history supplies it | From eac6fbfb02b34e14aaaa9fbc894817bedd6da749 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 21:59:07 +0000 Subject: [PATCH 17/61] fix(git): commit --dry-run leaves what the user staged as it was `mxcli git commit --dry-run` staged everything in the real index to report it, then undid that with `git reset -q`. A reset goes back to HEAD, not to the state before the run, so anything the user had staged was unstaged. The docs promise that a dry run writes nothing. A dry run now stages into a temporary copy of the index (GIT_INDEX_FILE), so the real index is never touched: staged changes, stat data and intent-to-add entries all stay as they were. A real commit still uses the real index, as before. `gitOutInEnv` is the git helper with extra environment. Test: TestGitCommit_DryRunKeepsTheUsersStaging. Before the fix: `staged after --dry-run = "", want exactly what the user had staged (staged.txt)`. The existing dry-run test started from an empty index, where the bug and the correct behaviour look the same. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...it-dry-run-unstages-the-users-staging.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_commit.go | 70 ++++++++++++++++--- cmd/mxcli/gitdoctor_test.go | 28 ++++++++ cmd/mxcli/gitnote.go | 8 ++- docs-site/src/tools/outside-studio-pro.md | 3 +- 6 files changed, 99 insertions(+), 13 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json new file mode 100644 index 000000000..80af6632b --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "`mxcli git commit -m … -p app.mpr --dry-run` unstages everything the user had staged: a file added with `git add` before the dry run is no longer staged afterwards (`git diff --cached` is empty), although the command committed nothing and documents that a dry run writes nothing", "cause": "The dry run staged everything in the real index (`git add -A`) to report it, then undid that with a plain `git reset -q`, which resets the index to HEAD rather than to what it was before the run", "file": "`cmd/mxcli/cmd_git_commit.go` (`commitRun`: a dry run stages into a temporary copy of the index through GIT_INDEX_FILE, `indexCopy`); `cmd/mxcli/gitnote.go` (`gitOutInEnv`); test `TestGitCommit_DryRunKeepsTheUsersStaging` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**The existing dry-run test started from an empty index**, so 'nothing staged afterwards' was both the correct result and the bug's result; a test of an undo has to start from a state the undo can get wrong. **Do not undo, isolate**: staging into a GIT_INDEX_FILE copy leaves the real index untouched byte for byte (stat data and intent-to-add entries included), which no reset or write-tree/read-tree round trip quite does. An empty file is not a valid index, so with no index yet the temporary path is left free for git to create", "refs": ["ako/mxcli#1670", "mendixlabs/mxcli#1669"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index d13bae0fb..22b0ac578 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. +- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index cbdb52cbe..3db953617 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -60,7 +60,7 @@ func init() { gitCommitCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") _ = gitCommitCmd.MarkFlagRequired("project") gitCommitCmd.Flags().Bool("push", false, "push the branch and the notes ref to their remote afterwards") - gitCommitCmd.Flags().Bool("dry-run", false, "stage and report, then unstage; commit nothing") + gitCommitCmd.Flags().Bool("dry-run", false, "report what would be committed; commit nothing and leave your staging as it is") gitCmd.AddCommand(gitCommitCmd) } @@ -115,12 +115,25 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error } } - // 2. Stage everything, then unstage what Studio Pro never commits. - if _, err := gitOutIn(root, nil, "add", "-A"); err != nil { + // 2. Stage everything, then unstage what Studio Pro never commits. A dry + // run stages into a copy of the index: resetting the real one afterwards + // unstaged whatever the user had staged before running it. + var env []string + if dryRun { + idx, cleanup, err := indexCopy(root) + if err != nil { + return fmt.Errorf("copy the index for the dry run: %w", err) + } + defer cleanup() + env = []string{"GIT_INDEX_FILE=" + idx} + } + git := func(args ...string) (string, error) { return gitOutInEnv(root, env, nil, args...) } + if _, err := git("add", "-A"); err != nil { return fmt.Errorf("stage changes: %w", err) } // -z: paths come back NUL-separated, so spaces and quotes stay inside one path. - staged := nulFields(gitOutOr(root, "diff", "--cached", "--name-only", "-z")) + cached, _ := git("diff", "--cached", "--name-only", "-z") + staged := nulFields(cached) var junk []string for _, f := range staged { if isJunkPath(f) { @@ -128,8 +141,7 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error } } if len(junk) > 0 { - args := append([]string{"reset", "-q", "--"}, junk...) - if _, err := gitOutIn(root, nil, args...); err != nil { + if _, err := git(append([]string{"reset", "-q", "--"}, junk...)...); err != nil { return fmt.Errorf("unstage files Studio Pro never commits: %w", err) } fmt.Fprintf(out, "Left out %d file(s) Studio Pro never commits (project-settings.user.json, *.mpr.lock, caches).\n", len(junk)) @@ -151,10 +163,7 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error } if dryRun { - if _, err := gitOutIn(root, nil, "reset", "-q"); err != nil { - return fmt.Errorf("unstage: %w", err) - } - fmt.Fprintln(out, "Dry run: staged nothing permanently, committed nothing.") + fmt.Fprintln(out, "Dry run: committed nothing; what you had staged is unchanged.") return nil } @@ -185,6 +194,47 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error return nil } +// indexCopy copies the repository's index to a temporary file, for a +// GIT_INDEX_FILE that a dry run can stage into without touching the real one. +// With no index yet, the path is left free for git to create it. +func indexCopy(root string) (string, func(), error) { + p, err := gitOut(root, "rev-parse", "--git-path", "index") + if err != nil { + return "", nil, err + } + src := strings.TrimSpace(p) + if !filepath.IsAbs(src) { + src = filepath.Join(root, src) + } + f, err := os.CreateTemp("", "mxcli-dry-run-index-*") + if err != nil { + return "", nil, err + } + tmp := f.Name() + cleanup := func() { _ = os.Remove(tmp) } + data, err := os.ReadFile(src) + switch { + case errors.Is(err, os.ErrNotExist): + f.Close() + cleanup() // an empty file is not a valid index + return tmp, cleanup, nil + case err != nil: + f.Close() + cleanup() + return "", nil, err + } + if _, err := f.Write(data); err != nil { + f.Close() + cleanup() + return "", nil, err + } + if err := f.Close(); err != nil { + cleanup() + return "", nil, err + } + return tmp, cleanup, nil +} + // nulFields splits git's -z output (NUL-separated, no quoting) into paths. func nulFields(s string) []string { var out []string diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index 1a3b49e3e..edb2b38ec 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -458,6 +458,34 @@ func TestGitCommit_RefusedWhileStudioProOpen(t *testing.T) { } } +// --dry-run staged everything to report it and then reset the whole index, so +// whatever the user had staged before running it was unstaged. +func TestGitCommit_DryRunKeepsTheUsersStaging(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + for _, name := range []string{"staged.txt", "unstaged.txt"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte(name), 0o644); err != nil { + t.Fatal(err) + } + } + gitIn(t, dir, "add", "staged.txt") + + var out bytes.Buffer + if err := commitRun(mprPath, "dry", false, true, &out); err != nil { + t.Fatalf("dry run failed: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "unstaged.txt") { + t.Errorf("the dry run must still report every file it would commit:\n%s", out.String()) + } + staged, err := gitOut(dir, "diff", "--cached", "--name-only") + if err != nil { + t.Fatal(err) + } + if got := strings.TrimSpace(staged); got != "staged.txt" { + t.Errorf("staged after --dry-run = %q, want exactly what the user had staged (staged.txt)", got) + } +} + func TestGitCommit_DryRunCommitsNothing(t *testing.T) { dir, mprPath := doctorNoteRepo(t) doctorEnv(t, dir) diff --git a/cmd/mxcli/gitnote.go b/cmd/mxcli/gitnote.go index 24a5934d6..b9e55e1bd 100644 --- a/cmd/mxcli/gitnote.go +++ b/cmd/mxcli/gitnote.go @@ -366,8 +366,14 @@ func gitOut(dir string, args ...string) (string, error) { } func gitOutIn(dir string, stdin []byte, args ...string) (string, error) { + return gitOutInEnv(dir, nil, stdin, args...) +} + +// gitOutInEnv is gitOutIn with extra environment entries, such as the +// GIT_INDEX_FILE a dry run stages into. +func gitOutInEnv(dir string, env []string, stdin []byte, args ...string) (string, error) { cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) - cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0", "GIT_PAGER=cat") + cmd.Env = append(append(os.Environ(), "GIT_TERMINAL_PROMPT=0", "GIT_PAGER=cat"), env...) if stdin != nil { cmd.Stdin = bytes.NewReader(stdin) } diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index e48b61ff9..2da3867e1 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -208,7 +208,8 @@ The commit is refused while Studio Pro has the project open (the `.mpr.lock` beside it), and during an unresolved merge — `git add -A` would otherwise commit the conflict markers. `--dry-run` verifies and reports (including ContentsHash drift, which it does not repair — a dry run writes nothing), -stages, reports what it would commit, and unstages again without committing. +and reports what it would commit without committing. It stages into a copy of +the index, so whatever you had staged is left exactly as it was. ## Auditing a repository: `mxcli git doctor` From 6a7934936e209ad31b3ade3809b0859761a09574 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:01:11 +0000 Subject: [PATCH 18/61] fix(git): commit leaves out only the project's own build folders `mxcli git commit` dropped any path with a segment named `.mendix-cache`, `deployment`, `theme-cache` or `.mxcli`, at any depth. So `resources/deployment/endpoints.xml` was silently left out of the commit. The only trace was a count in "Left out 1 file(s) Studio Pro never commits". Those folders are the project's own: Studio Pro, mxbuild and mxcli create them directly beside the .mpr. isJunkPath now matches them there only. When the project sits in a subfolder of the repository, that subfolder is the anchor, taken from gitNoteRepo.MPRPath. The lock, backup and user-settings file names still match at any level. Tests: TestGitCommit_NestedFolderNamedLikeJunkIsCommitted and TestGitCommit_JunkAnchoredAtProjectInSubfolder. Before the fix both fail: "resources/deployment/endpoints.xml is not the project's own folder and must be committed", and the same for javascriptsource/theme-cache/, app/resources/deployment/ and tools/deployment/. The project's own deployment/ and theme-cache/ stay out, as the existing spaced-path test also checks. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...sted-folders-named-like-build-folders.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_commit.go | 37 ++++++--- cmd/mxcli/gitdoctor_test.go | 79 +++++++++++++++++++ docs-site/src/tools/outside-studio-pro.md | 6 +- 5 files changed, 112 insertions(+), 13 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json new file mode 100644 index 000000000..b6fbea8c4 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "`mxcli git commit` silently leaves a user file out of the commit when any folder on its path is named like a build folder: `resources/deployment/endpoints.xml` (or `javascriptsource/theme-cache/x.js`) stays untracked after the commit, with only `Left out 1 file(s) Studio Pro never commits (project-settings.user.json, *.mpr.lock, caches)` to show for it", "cause": "isJunkPath matched `.mendix-cache`, `deployment`, `theme-cache` and `.mxcli` on every path segment, so any folder of that name at any depth was dropped. Those are the project's own folders, the ones Studio Pro, mxbuild and mxcli create directly beside the .mpr; the file-name rules (lock, backup, user settings) are the ones that rightly match at any level", "file": "`cmd/mxcli/cmd_git_commit.go` (`isJunkPath(rel, projDir)`, projDir from `path.Dir(gitNoteRepo.MPRPath)`, which already resolves the project folder inside the repository, symlinks included); tests `TestGitCommit_NestedFolderNamedLikeJunkIsCommitted`, `TestGitCommit_JunkAnchoredAtProjectInSubfolder` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**Anchor a path rule where the thing it names is created**: the build and cache folders only ever appear beside the .mpr, and a rule that matches the name at every depth also catches user folders that share it. **The report that something was left out did not name it**, which is what made the drop silent; a nested-folder fixture next to the root one (the existing test only had root `deployment/`) is the control that tells the two apart. Test the project-in-a-subfolder layout too: the anchor is the project's folder, not the repository root, and the same names outside the project are not its own", "refs": ["ako/mxcli#1670", "mendixlabs/mxcli#1669"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 22b0ac578..3ff88359a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. +- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index 3db953617..3942c276c 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -32,8 +32,9 @@ var gitCommitCmd = &cobra.Command{ rewritten from the file, which is the source of truth; MISSING or ORPHAN entries have no automatic repair and refuse the commit; 2. everything is staged, then the files Studio Pro never commits are - unstaged again: project-settings.user.json, *.mpr.lock, *.mpr.bak, - .mendix-cache/, deployment/, theme-cache/, .mxcli/; + unstaged again: project-settings.user.json, *.mpr.lock, *.mpr.bak, and + the project's own .mendix-cache/, deployment/, theme-cache/ and .mxcli/ + beside the .mpr (a folder of the same name further down is committed); 3. git commit runs with your message and your git identity; 4. the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check it and show what changed. @@ -134,9 +135,10 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error // -z: paths come back NUL-separated, so spaces and quotes stay inside one path. cached, _ := git("diff", "--cached", "--name-only", "-z") staged := nulFields(cached) + projDir := path.Dir(repo.MPRPath) // the project's folder in the repository; "." at its root var junk []string for _, f := range staged { - if isJunkPath(f) { + if isJunkPath(f, projDir) { junk = append(junk, f) } } @@ -247,9 +249,14 @@ func nulFields(s string) []string { } // isJunkPath reports whether a repo-relative path is one of the files Studio -// Pro never commits. Matched on any path level and without case, the way -// Studio Pro matches its own lock file (whose .gitignore entry is lowercase). -func isJunkPath(rel string) bool { +// Pro never commits. The files match by name at any level and without case, +// the way Studio Pro matches its own lock file (whose .gitignore entry is +// lowercase). The folders are the project's own, the ones Studio Pro, mxbuild +// and mxcli create directly in projDir (the project's folder relative to the +// repository root, "." at the root): matched at any depth, a user folder of +// the same name, resources/deployment/ say, was silently left out of the +// commit. +func isJunkPath(rel, projDir string) bool { f := filepath.ToSlash(rel) base := path.Base(f) switch { @@ -258,11 +265,21 @@ func isJunkPath(rel string) bool { case strings.HasSuffix(strings.ToLower(base), ".mpr.lock"), strings.HasSuffix(strings.ToLower(base), ".mpr.bak"): return true } - for _, seg := range strings.Split(f, "/") { - switch strings.ToLower(seg) { - case ".mendix-cache", "deployment", "theme-cache", ".mxcli": - return true + inProject := f + if projDir != "." && projDir != "" { + prefix := projDir + "/" + if len(f) <= len(prefix) || !strings.EqualFold(f[:len(prefix)], prefix) { + return false } + inProject = f[len(prefix):] + } + folder, rest, inFolder := strings.Cut(inProject, "/") + if !inFolder || rest == "" { + return false + } + switch strings.ToLower(folder) { + case ".mendix-cache", "deployment", "theme-cache", ".mxcli": + return true } return false } diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index edb2b38ec..5457a1b31 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -592,6 +592,85 @@ func TestGitCommit_SpacedPathsStagedWholeAndJunkLeft(t *testing.T) { } } +// writeFiles creates each repo-relative path with its parent folders. +func writeFiles(t *testing.T, dir string, paths ...string) { + t.Helper() + for _, p := range paths { + full := filepath.Join(dir, filepath.FromSlash(p)) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(full, []byte(p), 0o644); err != nil { + t.Fatal(err) + } + } +} + +// committedSet is the set of paths HEAD changed. +func committedSet(t *testing.T, dir string) map[string]bool { + t.Helper() + set := map[string]bool{} + for _, f := range strings.Split(strings.TrimSpace(committedFiles(t, dir)), "\n") { + set[f] = true + } + return set +} + +// The never-commit folders are the project's own, beside the .mpr, where +// Studio Pro's .gitignore anchors them (/deployment/). Matched at any depth, a +// user folder of the same name was silently left out of the commit. +func TestGitCommit_NestedFolderNamedLikeJunkIsCommitted(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + writeFiles(t, dir, "resources/deployment/endpoints.xml", "javascriptsource/theme-cache/x.js", "deployment/run.log", "theme-cache/a.css") + + var out bytes.Buffer + if err := commitRun(mprPath, "nested", false, false, &out); err != nil { + t.Fatalf("commitRun: %v\n%s", err, out.String()) + } + files := committedSet(t, dir) + for _, keep := range []string{"resources/deployment/endpoints.xml", "javascriptsource/theme-cache/x.js"} { + if !files[keep] { + t.Errorf("%s is not the project's own folder and must be committed:\n%s", keep, out.String()) + } + } + for _, junk := range []string{"deployment/run.log", "theme-cache/a.css"} { + if files[junk] { + t.Errorf("%s is the project's own and must be left out", junk) + } + } +} + +// With the project in a subfolder of the repository, its folders are under +// that subfolder; the same names elsewhere in the repository are not its own. +func TestGitCommit_JunkAnchoredAtProjectInSubfolder(t *testing.T) { + root := t.TempDir() + doctorEnv(t, root) + gitIn(t, root, "init", "-q", "-b", "feature") + app := filepath.Join(root, "app") + if err := os.MkdirAll(app, 0o755); err != nil { + t.Fatal(err) + } + writeProject(t, app, "11.13.0", baseUnits(t)) + gitIn(t, root, "add", "-A") + gitIn(t, root, "commit", "-q", "-m", "base") + writeFiles(t, root, "app/deployment/run.log", "app/resources/deployment/endpoints.xml", "tools/deployment/deploy.sh") + + var out bytes.Buffer + if err := commitRun(filepath.Join(app, "App.mpr"), "sub", false, false, &out); err != nil { + t.Fatalf("commitRun: %v\n%s", err, out.String()) + } + files := committedSet(t, root) + if files["app/deployment/run.log"] { + t.Error("app/deployment/ is the project's own and must be left out") + } + for _, keep := range []string{"app/resources/deployment/endpoints.xml", "tools/deployment/deploy.sh"} { + if !files[keep] { + t.Errorf("%s must be committed:\n%s", keep, out.String()) + } + } +} + // An unresolved merge must refuse the commit: `git add -A` would otherwise // stage the conflict markers and commit them. func TestGitCommit_RefusedDuringUnresolvedMerge(t *testing.T) { diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 2da3867e1..8219bcf62 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -196,8 +196,10 @@ In order it: 1. repairs ContentsHash mismatches from the files on disk (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit); 2. stages everything, then unstages the files Studio Pro never commits: - `project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, - `deployment/`, `theme-cache/`, `.mxcli/`; + `project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's + own `.mendix-cache/`, `deployment/`, `theme-cache/` and `.mxcli/`, the + folders Studio Pro, mxbuild and mxcli create beside the `.mpr`. A folder of + the same name further down, such as `resources/deployment/`, is committed; 3. runs `git commit` with your message and your git identity; 4. computes the mx_metadata note from the new commit and attaches it, and prints the `git push origin refs/notes/mx_metadata` command — From 58e8346f8be08791fefd3d6c242ab709d2de5cce Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:58:01 +0000 Subject: [PATCH 19/61] fix(widgets): read switch arms and statement-head guards in editorConfig hide rules A pluggable widget's enumeration that its editorConfig hides must hold its default, or mx check reports CE0463. mxcli refused that for the one Timeline rule it read (`groupEvents: false`), but `widget describe timeline` showed "2 of 6 editor hide-rules recognized". So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed check and exec and failed mx check (ako/mxcli#1762, mendixlabs/mxcli#1668). Two shapes were unread: - terser folds the statements before a `switch` into its head, `switch(A ? hide : hide, B || hide, e.groupByKey)`, so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard took for a conditional group. A `switch(`/`if(` head directly in a function body is now statement-level; a call's paren still is not. - a hide in a `case`/`default` arm had no reading. switchArmCondition now states the arm's values as eq/in, or ne/notin over the other cases for a default arm, following fall-through and grouped labels. Anything it cannot state exactly (a guarded hide inside the arm, a non-literal label, an arm whose exit is unclear, a switch under an `if`) yields no rule, as before. Timeline is 6 of 6 (3.1.1, 3.2.2, 3.2.3), so each hidden enumeration now gets MDL-WIDGET10's existing verdict: an error for a non-default value, which exec's preflight refuses before writing; a warning at the default; written at the default when left out. Across every installed package dumped before and after, only Timeline and HeatMap gained rules (HeatMap folds an `if(` head the same way). MDL-WIDGET10 names an in/notin set instead of "matches". WidgetDefGeneratorVersion goes to 20 so cached definitions regenerate. The custom-mode warning on title/description/timeIndication came from the hand-written fallback table, whose comment claimed only TextTemplates can cause CE0463. That comment, and the writer's, are corrected. File Uploader's ternary-wrapped hidePropertiesIn (ako/mxcli#258 item 1) was already lifted by the else-if chain support of mendixlabs/mxcli#1583. It is pinned by a test against File Uploader 2.5.0. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: the six hidden variants were CE0463 and the three controls 0 errors before the fix; check now refuses exactly those six. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...meline-hidden-enumeration-switch-arms.json | 1 + CHANGELOG.md | 1 + .../exec_preflight_timeline_1762_test.go | 103 ++++ docs-site/src/guides/pluggable-widgets.md | 13 + .../1762-timeline-hidden-enumeration.mdl | 114 ++++ mdl/backend/widgetobj/builder.go | 12 +- mdl/executor/editorconfig_extract.go | 47 +- mdl/executor/editorconfig_switch.go | 496 ++++++++++++++++++ mdl/executor/editorconfig_switch_1762_test.go | 235 +++++++++ .../validate_timeline_hidden_1762_test.go | 136 +++++ mdl/executor/validate_widgets.go | 19 + mdl/executor/widget_defs.go | 21 +- mdl/executor/widget_engine.go | 7 +- 13 files changed, 1186 insertions(+), 19 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json create mode 100644 cmd/mxcli/exec_preflight_timeline_1762_test.go create mode 100644 mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl create mode 100644 mdl/executor/editorconfig_switch.go create mode 100644 mdl/executor/editorconfig_switch_1762_test.go create mode 100644 mdl/executor/validate_timeline_hidden_1762_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json new file mode 100644 index 000000000..e8d3f6949 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-28","symptom":"A Timeline enumeration the widget hides, set to a non-default value — `customVisualization: true, groupByDayOptions: 'dayMonth'`, or `groupByKey: 'day', groupByMonthOptions: 'monthYear'` — passes `mxcli check` and `exec`, and mx check reports \"[CE0463] The definition of this widget has changed\" at Timeline 'tlD' / 'tlG'. `mxcli widget describe timeline` says \"2 of 6 editor hide-rules recognized\"; `groupEvents: false` plus a hidden value was already an MDL-WIDGET10 error (ako/mxcli#1762, mendixlabs/mxcli#1668)","cause":"Two editorConfig shapes went unread. terser folds the statements before a `switch` into its head — `switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey)` — so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard rejected as a conditional group (the else branch was lifted, through trailingExpr). And a hide in a `case`/`default` arm had no reading at all: its `case \"day\":` colon reached the ternary-else path and found no `?`. The template warnings the reporter saw in custom mode came from the hand-written fallback table in widget_defs.go, which listed only the TextTemplate keys of that same hide, on the stated premise that enumerations cannot cause CE0463","file":"mdl/executor/editorconfig_switch.go (switchArmCondition, switchArms, statementHeadParen, atFunctionStatementLevel), editorconfig_extract.go (the switch-arm branch before parseGuard; parseGuard's `(` boundary), validate_widgets.go (visibilityCondWord in/notin), widget_engine.go (WidgetDefGeneratorVersion 20)","insight":"**A known rule elsewhere is not the extractor knowing it.** The custom-mode warning made rule (1) look recognised, and it was the reason to ask why the enumerations in the same list were missed: `widget describe` (extractor only) showed no `title` rule, while the generated .def.json had one, merged from the hand table. Compare the two sources first when a rule half-works. **The reporter's framing (`the true branch of a ternary`) was not the cause**: the existing TimelinePattern test already lifted a statement-level THEN branch, and only the minified file showed that terser had moved it into the `switch(` head. Read the shipped JS, not the de-minified source. **Measure the matrix, then make it the oracle**: one widget per variant through `mx check --no-update-widgets` on 11.13.0 gave 6 CE0463 and 3 clean, and the extractor test, the check test and the bug-test all encode those nine. **Policy follows the existing split, not a second one**: hidden + non-default is refused (MDL-WIDGET10 error, exec preflight) and hidden + left out is written at the default, as rule (3) and the column-SET reset (mendixlabs/mxcli#1519) already do. **Bound the blast radius by dumping lifted rules for every installed .mpk before and after**: only Timeline and HeatMap changed. HeatMap folds into an `if(` head the same way, and its new rule is correct. **The scratchpad can be shared between agents**: work in a private subdirectory","refs":["ako/mxcli#1762","mendixlabs/mxcli#1668","ako/mxcli#258","mendixlabs/mxcli#1583"],"ce":["CE0463"],"rules":["MDL-WIDGET10"],"test":"mdl/executor/editorconfig_switch_1762_test.go; validate_timeline_hidden_1762_test.go; cmd/mxcli/exec_preflight_timeline_1762_test.go; mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl"} diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ff88359a..c33507cc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/cmd/mxcli/exec_preflight_timeline_1762_test.go b/cmd/mxcli/exec_preflight_timeline_1762_test.go new file mode 100644 index 000000000..e948758ed --- /dev/null +++ b/cmd/mxcli/exec_preflight_timeline_1762_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/deprecation" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#1762 (mendixlabs/mxcli#1668), the reporter's script. Reported: check +// and exec both pass, and mx check says "[CE0463] The definition of this widget +// has changed" at Timeline 'tlD' (groupByDayOptions, hidden in custom mode) and +// 'tlG' (groupByMonthOptions, hidden when groupByKey is 'day'). exec must refuse +// it before writing anything, naming both — as it already did for the one +// Timeline rule it knew, `groupEvents: false`. +const timeline1762Setup = `create module ModTL; +create non-persistent entity ModTL.Ev ( Line: String(200), WhoText: String(200), WhenText: String(50), At: DateTime ); +create microflow ModTL.DS_Events () returns List of ModTL.Ev as $Events +begin + $Events = create list of ModTL.Ev; + $E = create ModTL.Ev ( Line = 'Created', WhoText = 'someone', WhenText = '10:00', At = [%CurrentDateTime%] ); + add $E to $Events; + return $Events; +end; +` + +func timeline1762Pages(dayOptions, monthOptions string) string { + return `create page ModTL.PageD ( Title: 'D', Layout: Atlas_Core.Atlas_Default ) { + timeline tlD ( + data: MICROFLOW ModTL.DS_Events, + customVisualization: true, groupEvents: true, groupAttribute: At, + groupByKey: 'day', ` + dayOptions + `ungroupedEventsPosition: 'end' + ) { + customtitle d2 { dynamictext d2t (Content: '{1}', ContentParams: ({1} = Line)) } + } +}; +create page ModTL.PageG ( Title: 'G', Layout: Atlas_Core.Atlas_Default ) { + timeline tlG ( + data: MICROFLOW ModTL.DS_Events, + title: '{1}', titleParams: ({1} = $currentObject/Line), + customVisualization: false, groupEvents: true, groupAttribute: At, + groupByKey: 'day', ` + monthOptions + `ungroupedEventsPosition: 'end' + ) +}; +` +} + +func TestExecPreflightRefusesHiddenTimelineEnumerations(t *testing.T) { + src := filepath.Join("..", "..", "testdata", "expr-checker") + if _, err := os.Stat(filepath.Join(src, "minimal.mpr")); err != nil { + t.Skipf("expr-checker fixture not found: %v", err) + } + dir := t.TempDir() + if err := os.CopyFS(dir, os.DirFS(src)); err != nil { + t.Fatal(err) + } + mpr := filepath.Join(dir, "minimal.mpr") + exec := executor.New(io.Discard) + exec.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + if err := exec.Execute(&ast.ConnectStmt{Path: mpr}); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = exec.Execute(&ast.DisconnectStmt{}) }) + + preflight := func(script string) (string, string) { + t.Helper() + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatal(errs[0]) + } + var out bytes.Buffer + refusal := execPreflight(exec, prog, mpr, "s.mdl", false, false, false, deprecation.Warn, &out, false) + return refusal, out.String() + } + + refusal, out := preflight(timeline1762Setup + + timeline1762Pages("groupByDayOptions: 'dayMonth', ", "groupByMonthOptions: 'monthYear', ")) + if refusal == "" || !strings.Contains(refusal, "Nothing was written") { + t.Fatalf("want the pre-flight to refuse the script, got %q\n%s", refusal, out) + } + for _, want := range []string{"`tlD`", "`groupByDayOptions`", "`tlG`", "`groupByMonthOptions`"} { + if !strings.Contains(out, want) { + t.Errorf("pre-flight output must name %s:\n%s", want, out) + } + } + + // CONTROL: the same pages without the hidden values (variant b of the + // report, which mx check passes) run. + if refusal, out := preflight(timeline1762Setup + timeline1762Pages("", "")); refusal != "" { + t.Errorf("the pages without hidden values were refused: %s\n%s", refusal, out) + } +} diff --git a/docs-site/src/guides/pluggable-widgets.md b/docs-site/src/guides/pluggable-widgets.md index a83a59eb0..6ef086397 100644 --- a/docs-site/src/guides/pluggable-widgets.md +++ b/docs-site/src/guides/pluggable-widgets.md @@ -51,6 +51,19 @@ version you actually have at the moment you create the widget. rules** and applies them, so a freshly-created widget carries exactly the defaults Studio Pro would give it. + A property the widget hides must hold its default, or the build fails with CE0463. + So a hidden property you leave out is written at its default; one you set to its + default draws an MDL-WIDGET10 warning (the value is ignored); and one you set to + anything else is an MDL-WIDGET10 **error**, which `exec` refuses before writing. + For example, a + Timeline grouped by `'day'` hides `groupByMonthOptions`, and one with + `customVisualization: true` hides both format enumerations. + + This only covers rules mxcli recognises. `mxcli widget describe -p app.mpr` + ends with "*N* of *M* editor hide-rules recognized". A rule it cannot read hides + nothing as far as `check` knows, so a value set behind one reaches `mx check` + unwarned. + The result: widgets created by mxcli open cleanly across Mendix 10.x and 11.x with the widget packages bundled with those releases — verified on Mendix 11.12 with Data Widgets 3.4: a project authored entirely by mxcli reports **0** CE0463. diff --git a/mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl b/mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl new file mode 100644 index 000000000..3cbb491b3 --- /dev/null +++ b/mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl @@ -0,0 +1,114 @@ +mdl 1; +-- ============================================================================ +-- Issue ako/mxcli#1762 (mendixlabs/mxcli#1668) — a Timeline enumeration the +-- widget hides, set to a non-default value, fails the build with CE0463 +-- ============================================================================ +-- +-- The Timeline's editorConfig hides properties six ways: +-- +-- e.customVisualization ? hide(title, description, icon, timeIndication, +-- groupByDayOptions, groupByMonthOptions) (1) +-- : hide(customIcon, customTitle, …) (2) +-- e.groupEvents || hide(groupAttribute, groupByKey, groupByDayOptions, …) (3) +-- switch (e.groupByKey) { case "day": hide(groupByMonthOptions) (4) +-- case "month": hide(groupByDayOptions) (5) +-- default: hide(groupByDayOptions, +-- groupByMonthOptions) } (6) +-- +-- mxcli read (2) and (3) only. Measured on mxbuild 11.13.0 with Timeline 3.2.2, +-- one widget per variant, nothing else hidden set: +-- +-- customVisualization: true, groupByDayOptions: 'dayMonth' → CE0463 (1) +-- customVisualization: true, groupByKey: 'month', +-- groupByMonthOptions: 'monthYear' → CE0463 (1) +-- groupByKey: 'day', groupByMonthOptions: 'monthYear' → CE0463 (4) +-- groupByKey: 'month', groupByDayOptions: 'fullDate' → CE0463 (5) +-- groupByKey: 'year', groupByDayOptions / groupByMonthOptions set → CE0463 (6) +-- the same values where they are visible, or left out → 0 errors +-- +-- `check` and `exec` passed all of them. Now each is refused with MDL-WIDGET10 +-- — the verdict (3) already got — naming the property, the rule and the +-- default; one the script leaves out is written at that default. +-- +-- Verify (needs -p: the rules are lifted from the project's installed .mpk): +-- +-- mxcli check 1762-timeline-hidden-enumeration.mdl -p app.mpr --references +-- → passes; uncomment a statement below to see the MDL-WIDGET10 error +-- mxcli exec 1762-timeline-hidden-enumeration.mdl -p app.mpr +-- mxcli docker check -p app.mpr --no-update-widgets # 0 errors +-- describe page M1762.PageD, then exec it # unchanged +-- ============================================================================ + +create module M1762; + +create non-persistent entity M1762.Ev ( + Line: String(200), + WhoText: String(200), + At: DateTime +); + +create microflow M1762.DS_Events () returns List of M1762.Ev as $Events +begin + $Events = create list of M1762.Ev; + $E = create M1762.Ev ( Line = 'Created', WhoText = 'someone', At = [%CurrentDateTime%] ); + add $E to $Events; + return $Events; +end; + +-- The reporter's tlD, corrected: custom mode hides the format enumerations, so +-- they are left out (written at their defaults). +create or modify page M1762.PageD ( Title: 'D', Layout: Atlas_Core.Atlas_Default ) { + timeline tlD ( + data: MICROFLOW M1762.DS_Events, + customVisualization: true, groupEvents: true, groupAttribute: At, + groupByKey: 'day', ungroupedEventsPosition: 'end' + ) { + customtitle d2 { dynamictext d2t (Content: '{1}', ContentParams: ({1} = Line)) } + } +}; + +-- The reporter's tlG, corrected: grouping by day shows only groupByDayOptions. +create or modify page M1762.PageG ( Title: 'G', Layout: Atlas_Core.Atlas_Default ) { + timeline tlG ( + data: MICROFLOW M1762.DS_Events, + title: '{1}', titleParams: ({1} = $currentObject/Line), + customVisualization: false, groupEvents: true, groupAttribute: At, + groupByKey: 'day', groupByDayOptions: 'fullDate', ungroupedEventsPosition: 'end' + ) +}; + +-- Grouping by month shows only groupByMonthOptions. +create or modify page M1762.PageM ( Title: 'M', Layout: Atlas_Core.Atlas_Default ) { + timeline tlM ( + data: MICROFLOW M1762.DS_Events, + title: '{1}', titleParams: ({1} = $currentObject/Line), + groupAttribute: At, groupByKey: 'month', groupByMonthOptions: 'monthYear' + ) +}; + +-- The failing shapes, kept as documentation. Uncomment one and re-run +-- `mxcli check ... -p app.mpr`: MDL-WIDGET10 is an error, and `mx check` +-- reports CE0463 on the same widget. +-- +-- (1) hidden in custom mode: +-- +-- create or modify page M1762.PageD1 ( Title: 'x', Layout: Atlas_Core.Atlas_Default ) { +-- timeline tlD1 ( data: MICROFLOW M1762.DS_Events, customVisualization: true, +-- groupAttribute: At, groupByKey: 'day', groupByDayOptions: 'dayMonth' ) { +-- customtitle d3 { dynamictext d3t (Content: '{1}', ContentParams: ({1} = Line)) } +-- } +-- }; +-- +-- (4) grouping by day hides the month format: +-- +-- create or modify page M1762.PageG1 ( Title: 'x', Layout: Atlas_Core.Atlas_Default ) { +-- timeline tlG1 ( data: MICROFLOW M1762.DS_Events, title: '{1}', titleParams: ({1} = $currentObject/Line), +-- groupAttribute: At, groupByKey: 'day', groupByMonthOptions: 'monthYear' ) +-- }; +-- +-- (6) grouping by year hides both: +-- +-- create or modify page M1762.PageY1 ( Title: 'x', Layout: Atlas_Core.Atlas_Default ) { +-- timeline tlY1 ( data: MICROFLOW M1762.DS_Events, title: '{1}', titleParams: ({1} = $currentObject/Line), +-- groupAttribute: At, groupByKey: 'year', groupByDayOptions: 'dayMonth' ) +-- }; diff --git a/mdl/backend/widgetobj/builder.go b/mdl/backend/widgetobj/builder.go index f89bf4e2d..8f1d0408c 100644 --- a/mdl/backend/widgetobj/builder.go +++ b/mdl/backend/widgetobj/builder.go @@ -882,9 +882,11 @@ func (ob *Builder) EnsureRequiredObjectLists() {} // ApplyPropertyVisibility nulls the TextTemplate of any TextTemplate-typed // property the rules mark as hidden under the widget's current configuration. -// Non-TextTemplate properties are left untouched: only the populated-vs-null -// ClientTemplate choice triggers CE0463, so Phase 1 scopes the action to -// TextTemplate. The widget's current primitive values (read from the assembled +// Non-TextTemplate properties are left untouched HERE, not because they are +// harmless: a hidden enumeration at a non-default value is CE0463 too +// (ako/mxcli#1762). The engine writes one the script does not name at its +// declared default (hiddenUnnamedProperties), and MDL-WIDGET10 refuses one it +// does. The widget's current primitive values (read from the assembled // object) drive rule evaluation, so a rule keyed on e.g. `type` sees the value // just set from MDL. @@ -897,8 +899,8 @@ func (ob *Builder) ApplyPropertyVisibility(rules []types.WidgetVisibilityRule) { // the updated object. It is the engine-agnostic form of ApplyPropertyVisibility // so build paths that construct a widget object without a full Builder (e.g. the // filter-widget build path) can apply the same #574 nulling. Non-TextTemplate -// properties are left untouched — only the populated-vs-null ClientTemplate -// choice triggers CE0463. +// properties are left untouched here; see ApplyPropertyVisibility for where a +// hidden enumeration's default is enforced instead. func ApplyVisibilityRules(object bson.D, propertyTypeIDs map[string]pages.PropertyTypeIDEntry, rules []types.WidgetVisibilityRule) bson.D { if len(rules) == 0 { return object diff --git a/mdl/executor/editorconfig_extract.go b/mdl/executor/editorconfig_extract.go index 6ced44279..2450a8b72 100644 --- a/mdl/executor/editorconfig_extract.go +++ b/mdl/executor/editorconfig_extract.go @@ -127,8 +127,9 @@ func extractVisibilityRulesFromMPK(mpkPath, widgetID string) []types.WidgetVisib // widget's compiled editorConfig.js into declarative WidgetVisibilityRules. // // It recognizes the dominant `getProperties` idioms — `"V"===ref && hide(...)`, -// `"V"!==ref && hide(...)`, `ref && hide(...)`, `ref || hide(...)`, and -// `ref ? hide(...) : …` — where `ref` is `obj.prop` or a locally-aliased +// `"V"!==ref && hide(...)`, `ref && hide(...)`, `ref || hide(...)`, +// `ref ? hide(...) : …`, and a hide in a `switch(ref){case "V": …}` arm — where +// `ref` is `obj.prop` or a locally-aliased // identifier resolved within the enclosing function scope. Everything it cannot // lift (object-list-nested hides, compound/computed guards, unresolved aliases) // is counted in the returned stats so callers can gauge coverage; unrecognized @@ -159,7 +160,21 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit if listKey != "" { itemIdent = enclosingForEachParam(js, callStart) } - cond, guardText, ok, conjunctive, chain := parseGuard(js, callStart, itemIdent) + // A hide in a `switch` arm runs on the arm's case values, not on a guard + // beside it, so it is read from the switch and never from parseGuard — + // whose `:` handling would take the arm's label for a ternary's else. + var ( + cond types.WidgetVisibilityCondition + guardText string + conjunctive bool + chain []types.WidgetVisibilityCondition + ) + armCond, inSwitchArm, armOK := switchArmCondition(js, callStart, itemIdent) + if inSwitchArm { + cond, ok = armCond, armOK + } else { + cond, guardText, ok, conjunctive, chain = parseGuard(js, callStart, itemIdent) + } // A guard inside `outer ? ( … inner && hide(x) … )` states only the INNER // term; the branch runs on outer too. Collect the enclosing group guards // so the rule carries the whole conjunction — Combo box nests three deep, @@ -254,13 +269,19 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit // guard alone would mark both branches hidden together (#956). if len(condKeys) > 0 { aliases := enclosingAliases(js, callStart) + // In a switch arm the element's condition holds only where the arm + // runs, so the arm's condition is a term of its own. + condExtra := extra + if inSwitchArm { + condExtra = append([]types.WidgetVisibilityCondition{cond}, extra...) + } for _, tk := range condKeys { c, ok := guardToCondition(tk.guard, tk.falsy, aliases, itemIdent) if !ok { stats.SkippedComplex++ continue } - sig := listKey + "\x00" + tk.key + "\x00" + c.PropertyKey + c.Operator + c.Value + c.Scope + condsSig(extra) + sig := listKey + "\x00" + tk.key + "\x00" + c.PropertyKey + c.Operator + c.Value + c.Scope + condsSig(condExtra) if seen[sig] { continue } @@ -270,7 +291,7 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit PropertyKey: tk.key, ListPropertyKey: listKey, HiddenWhen: &cc, - And: extra, + And: condExtra, }) } } @@ -810,6 +831,22 @@ func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibil if !falsy { return types.WidgetVisibilityCondition{}, guard, false, false, nil } + case '(': + // The paren heads a STATEMENT rather than grouping a guarded branch: + // terser folds the expression statements before a `switch`/`if` into + // its head, and their contents run unconditionally. Timeline's first + // hide is the THEN branch of + // + // switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey){…} + // + // and was dropped as "one operand of a compound condition" while its + // else branch, read through trailingExpr, was lifted (ako/mxcli#1762). + // A call's paren is not this case — the call can itself be guarded. + // boundaryAt indexes js only while pre is still a prefix of it, which + // an else-branch guard (cut out by ternaryCondition) is not. + if elseBranch || !strings.HasPrefix(js, pre) || !statementHeadParen(js, boundaryAt) { + return types.WidgetVisibilityCondition{}, guard, false, false, nil + } default: return types.WidgetVisibilityCondition{}, guard, false, false, nil } diff --git a/mdl/executor/editorconfig_switch.go b/mdl/executor/editorconfig_switch.go new file mode 100644 index 000000000..056bff4dd --- /dev/null +++ b/mdl/executor/editorconfig_switch.go @@ -0,0 +1,496 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// A widget's getProperties can choose what to hide with a `switch` on one of its +// enumerations. Timeline 3.2 does, and three of its six hide rules were unread +// (ako/mxcli#1762, mendixlabs/mxcli#1668): +// +// switch (…, e.groupByKey) { +// case "day": hidePropertyIn(t, e, "groupByMonthOptions"); break; +// case "month": hidePropertyIn(t, e, "groupByDayOptions"); break; +// default: hidePropertiesIn(t, e, ["groupByDayOptions", "groupByMonthOptions"]); +// } +// +// A hide in an arm fires on the values that enter the arm: its own labels, the +// labels of every arm that falls through into it, and — when a default arm is +// among those — every value no case names. That is one `eq`/`in` condition, or +// one `ne`/`notin` over the cases the switch does name. +// +// Exact or nothing, like the rest of the extractor: a hide carrying its own +// guard inside the arm, a label that is not a string literal, a switch that is +// not a statement of a function body, or a discriminant that is not one of the +// widget's properties yields no rule — which reads as "not hidden", the state +// every one of these was in before. + +// switchArm is one `case`/`default` arm of a switch body. +type switchArm struct { + labels []string // the case values; empty for a lone default + isDefault bool + start, end int // the arm's statements, as offsets into the switch body + exit armExit // whether control falls into the next arm +} + +// armExit is how control leaves a switch arm. +type armExit int + +const ( + armFallsThrough armExit = iota // runs on into the next arm + armTerminates // ends in break/return/throw/continue + armUnknown // cannot be told from its text (a block, a conditional break) +) + +// switchArmCondition reports whether the hide at callStart sits in a switch arm +// (inArm) and, when it does, the condition under which that arm runs. ok is +// false for an arm whose condition cannot be stated exactly; the caller then +// emits no rule and does NOT fall back to the guard parser, whose reading of +// the arm's `case "v":` label as a ternary's else could only be wrong. +func switchArmCondition(js string, callStart int, itemIdent string) (cond types.WidgetVisibilityCondition, inArm, ok bool) { + none := types.WidgetVisibilityCondition{} + open := enclosingOpenerQuoted(js, callStart) + if open < 0 || js[open] != '{' { + return none, false, false + } + head := strings.TrimRight(js[:open], " ") + if !strings.HasSuffix(head, ")") { + return none, false, false + } + parenOpen := matchingOpenParen(head) + if parenOpen < 0 { + return none, false, false + } + kwHead := strings.TrimRight(head[:parenOpen], " ") + if !hasKeywordSuffix(kwHead, "switch") { + return none, false, false + } + + // From here on the hide is in a switch arm. + kwStart := len(kwHead) - len("switch") + if !atFunctionStatementLevel(js, kwStart) { + return none, true, false // the switch runs only under some outer condition + } + // terser folds the preceding statements into the head — `switch(A, B, e.k)` — + // and the switch is on the LAST of them. + parts := splitTopLevelCommas(head[parenOpen+1 : len(head)-1]) + m := refRE.FindStringSubmatch(strings.TrimSpace(parts[len(parts)-1])) + if m == nil || m[1] == "!" { + return none, true, false + } + key, scope, resolved := resolveRef(m[2], enclosingAliases(js, kwStart), itemIdent) + if !resolved { + return none, true, false // the platform argument, or a computed local + } + body, balanced := balancedArgs(js, open+1) + if !balanced { + return none, true, false + } + arms, parsed := switchArms(body) + if !parsed { + return none, true, false + } + at := callStart - (open + 1) + idx := -1 + for i, a := range arms { + if at >= a.start && at < a.end { + idx = i + } + } + if idx < 0 || !onlyHideStatementsBefore(body[arms[idx].start:at]) { + return none, true, false + } + + first := idx +walk: + for first > 0 { + switch arms[first-1].exit { + case armTerminates: + break walk + case armUnknown: + // Treating it either way could be wrong: as falling through, this + // rule claims its values; as terminating, it misses them. + return none, true, false + } + first-- // that arm falls through into this one + } + var entered []string + hasDefault := false + for _, a := range arms[first : idx+1] { + entered = append(entered, a.labels...) + hasDefault = hasDefault || a.isDefault + } + op, vals := "in", uniqueStrings(entered) + if hasDefault { + // Entered by every value no case names, plus this run's own cases: the + // complement of the OTHER arms' cases. + var all []string + for _, a := range arms { + all = append(all, a.labels...) + } + op, vals = "notin", withoutStrings(uniqueStrings(all), vals) + } + switch { + case len(vals) == 0: + return none, true, false // every value enters: unconditional, not a condition + case len(vals) == 1 && op == "in": + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "eq", Value: vals[0], Scope: scope}, true, true + case len(vals) == 1: + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "ne", Value: vals[0], Scope: scope}, true, true + } + for _, v := range vals { + if strings.ContainsRune(v, ',') { + return none, true, false // the joined set would decode wrong + } + } + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: op, Value: strings.Join(vals, ","), Scope: scope}, true, true +} + +// switchArms splits a switch body into its arms. ok is false for anything but +// `case "literal":` and `default:` labels, or for text before the first label. +func switchArms(body string) ([]switchArm, bool) { + var arms []switchArm + depth := 0 + inStr := byte(0) + for i := 0; i < len(body); i++ { + c := body[i] + if inStr != 0 { + if c == '\\' { + i++ + } else if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + continue + case '(', '[', '{': + depth++ + continue + case ')', ']', '}': + depth-- + continue + } + if depth != 0 || (i > 0 && (isIdentByte(body[i-1]) || body[i-1] == '.')) { + continue + } + var arm switchArm + var next int + switch { + case hasKeywordAt(body, i, "case"): + label, end, ok := switchCaseLabel(body, i+len("case")) + if !ok { + return nil, false + } + arm.labels, next = []string{label}, end + case hasKeywordAt(body, i, "default"): + j := skipSpaces(body, i+len("default")) + if j >= len(body) || body[j] != ':' { + return nil, false + } + arm.isDefault, next = true, j+1 + default: + continue + } + if len(arms) == 0 { + if strings.TrimSpace(body[:i]) != "" { + return nil, false + } + } else { + arms[len(arms)-1].end = i + } + arm.start, arm.end = next, len(body) + arms = append(arms, arm) + i = next - 1 + } + if len(arms) == 0 { + return nil, false + } + for i := range arms { + arms[i].exit = exitOf(body[arms[i].start:arms[i].end]) + } + return arms, true +} + +// switchCaseLabel reads the `"value":` after a `case` keyword, returning the value and +// the offset just past the colon. +func switchCaseLabel(body string, from int) (string, int, bool) { + j := skipSpaces(body, from) + if j >= len(body) || (body[j] != '"' && body[j] != '\'') { + return "", 0, false + } + q := body[j] + k := strings.IndexByte(body[j+1:], q) + if k < 0 { + return "", 0, false + } + value := body[j+1 : j+1+k] + if strings.ContainsRune(value, '\\') { + return "", 0, false + } + end := skipSpaces(body, j+1+k+1) + if end >= len(body) || body[end] != ':' { + return "", 0, false + } + return value, end + 1, true +} + +// abruptKeywords leave a switch arm. +var abruptKeywords = []string{"break", "return", "throw", "continue"} + +// exitOf reports how control leaves an arm. Only two readings are certain: the +// last statement is itself a break/return/throw/continue, or the arm holds none +// of them and no block that could hide one. Anything else — `{ …; break }`, +// `if (c) break; …` — is unknown. +func exitOf(arm string) armExit { + stmts := splitTopLevel(arm, ';') + for i := len(stmts) - 1; i >= 0; i-- { + s := strings.TrimSpace(stmts[i]) + if s == "" { + continue + } + for _, kw := range abruptKeywords { + if hasKeywordAt(s, 0, kw) { + return armTerminates + } + } + if mayLeaveEarly(arm) { + return armUnknown + } + return armFallsThrough + } + return armFallsThrough // an empty arm (a grouped label) falls through +} + +// mayLeaveEarly reports whether s, outside its string literals, holds a block or +// an abrupt keyword anywhere. +func mayLeaveEarly(s string) bool { + inStr := byte(0) + for i := 0; i < len(s); i++ { + c := s[i] + if inStr != 0 { + if c == '\\' { + i++ + } else if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + continue + case '{': + return true + } + if i > 0 && (isIdentByte(s[i-1]) || s[i-1] == '.') { + continue + } + for _, kw := range abruptKeywords { + if hasKeywordAt(s, i, kw) { + return true + } + } + } + return false +} + +// onlyHideStatementsBefore reports whether the text of an arm before a hide is +// nothing but earlier hide calls — `case "v": hide(a), hide(b); hide(c)` — so +// the hide is guarded by the arm alone and not by a condition of its own. +func onlyHideStatementsBefore(prefix string) bool { + p := trimNamespacePrefix(strings.TrimRight(prefix, " ")) + for { + if strings.TrimSpace(p) == "" { + return true + } + if !strings.HasSuffix(p, ";") && !strings.HasSuffix(p, ",") { + return false + } + call := strings.TrimRight(p[:len(p)-1], " ") + if !strings.HasSuffix(call, ")") { + return false + } + open := matchingOpenParen(call) + if open < 0 { + return false + } + name := strings.TrimRight(call[:open], " ") + loc := hideCallTailRE.FindStringIndex(name) + if loc == nil || loc[1] != len(name) { + return false + } + p = trimNamespacePrefix(strings.TrimRight(name[:loc[0]], " ")) + } +} + +func trimNamespacePrefix(p string) string { + if loc := nsPrefixRE.FindStringIndex(p); loc != nil { + return strings.TrimRight(p[:loc[0]], " ") + } + return p +} + +// statementHeadParen reports whether the '(' at open is the head of a `switch` +// or `if` statement sitting directly in a function body. Its contents then run +// whenever the function does, so a guard bounded by it is statement-level. +func statementHeadParen(js string, open int) bool { + head := strings.TrimRight(js[:open], " ") + for _, kw := range []string{"switch", "if"} { + if hasKeywordSuffix(head, kw) { + return atFunctionStatementLevel(js, len(head)-len(kw)) + } + } + return false +} + +// atFunctionStatementLevel reports whether the statement starting at pos is one +// of a function body's own statements — not an `else`'s, not under an `if` +// without braces, not inside a nested block. Anything else runs under a +// condition the rule would not carry. +func atFunctionStatementLevel(js string, pos int) bool { + before := strings.TrimRight(js[:pos], " ") + if before != "" { + switch before[len(before)-1] { + case '{', ';', '}': + default: + return false + } + } + open := enclosingOpenerQuoted(js, pos) + return open >= 0 && js[open] == '{' && isFunctionBody(js, open) +} + +// isFunctionBody reports whether the '{' at open opens a function's body: +// `function(…){`, `function name(…){`, or an arrow's `=>{`. +func isFunctionBody(js string, open int) bool { + head := strings.TrimRight(js[:open], " ") + if strings.HasSuffix(head, "=>") { + return true + } + if !strings.HasSuffix(head, ")") { + return false + } + params := matchingOpenParen(head) + if params < 0 { + return false + } + name := strings.TrimRight(head[:params], " ") + if hasKeywordSuffix(name, "function") { + return true + } + i := len(name) + for i > 0 && isIdentByte(name[i-1]) { + i-- + } + return i < len(name) && hasKeywordSuffix(strings.TrimRight(name[:i], " "), "function") +} + +// enclosingOpenerQuoted is enclosingOpener skipping quoted strings, so a caption +// holding a bracket does not throw the count off. +func enclosingOpenerQuoted(js string, pos int) int { + depth := 0 + inStr := byte(0) + for i := pos - 1; i >= 0; i-- { + c := js[i] + if inStr != 0 { + if c == inStr && (i == 0 || js[i-1] != '\\') { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case '}', ')', ']': + depth++ + case '{', '(', '[': + if depth == 0 { + return i + } + depth-- + } + } + return -1 +} + +// hasKeywordSuffix reports whether s ends with the keyword kw as a whole word. +func hasKeywordSuffix(s, kw string) bool { + if !strings.HasSuffix(s, kw) { + return false + } + i := len(s) - len(kw) + return i == 0 || (!isIdentByte(s[i-1]) && s[i-1] != '.') +} + +// hasKeywordAt reports whether the keyword kw starts at s[i] as a whole word. +func hasKeywordAt(s string, i int, kw string) bool { + if !strings.HasPrefix(s[i:], kw) { + return false + } + end := i + len(kw) + return end == len(s) || !isIdentByte(s[end]) +} + +func skipSpaces(s string, i int) int { + for i < len(s) && s[i] == ' ' { + i++ + } + return i +} + +// splitTopLevel splits s on sep where it is not nested in a bracket or string. +func splitTopLevel(s string, sep byte) []string { + var parts []string + depth := 0 + inStr := byte(0) + start := 0 + for i := 0; i < len(s); i++ { + c := s[i] + if inStr != 0 { + if c == '\\' { + i++ + } else if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + case sep: + if depth == 0 { + parts = append(parts, s[start:i]) + start = i + 1 + } + } + } + return append(parts, s[start:]) +} + +func withoutStrings(ss, drop []string) []string { + out := ss[:0:0] + for _, s := range ss { + keep := true + for _, d := range drop { + if s == d { + keep = false + break + } + } + if keep { + out = append(out, s) + } + } + return out +} diff --git a/mdl/executor/editorconfig_switch_1762_test.go b/mdl/executor/editorconfig_switch_1762_test.go new file mode 100644 index 000000000..6864c6c3c --- /dev/null +++ b/mdl/executor/editorconfig_switch_1762_test.go @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The Timeline's hide rules (ako/mxcli#1762, mendixlabs/mxcli#1668). +// +// A pluggable widget's enumeration that its editorConfig hides must be stored at +// its XML default, or mx check reports CE0463. mxcli refuses that for the one +// Timeline rule it recognised (`groupEvents: false`), and passed the rest: +// `mxcli widget describe timeline` reported "2 of 6 editor hide-rules +// recognized". Measured on mxbuild 11.13.0 with Timeline 3.2.2, one widget per +// variant, nothing else hidden on the page: +// +// customVisualization: true, groupByDayOptions: 'dayMonth' → CE0463 (1) +// customVisualization: true, groupByKey: 'month', groupByMonthOptions: 'monthYear' → CE0463 (1) +// groupByKey: 'day', groupByMonthOptions: 'monthYear' → CE0463 (4) +// groupByKey: 'month', groupByDayOptions: 'fullDate' → CE0463 (5) +// groupByKey: 'year', groupByDayOptions: 'dayMonth' → CE0463 (6) +// groupByKey: 'year', groupByMonthOptions: 'monthYear' → CE0463 (6) +// controls — custom mode with neither set; 'day' + groupByDayOptions; +// 'month' + groupByMonthOptions → 0 errors +// +// That matrix is this file's oracle. Two shapes went unread. terser folds the +// statements before the `switch` into its head, `switch(A ? hide : hide, B || +// hide, e.groupByKey)`, so rule (1)'s guard is bounded by the `switch(` paren, +// which the guard parser took for a conditional group. And the `case`/`default` +// arms of a switch had no reading at all. +package executor + +import ( + "slices" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// timelineGetProperties is Timeline 3.2.3's getProperties, verbatim from the +// marketplace package (3.2.2 differs only in the namespace identifier). +const timelineGetProperties = `exports.getProperties=function(e,t){switch(e.customVisualization?I.hidePropertiesIn(t,e,["title","description","icon","timeIndication","groupByDayOptions","groupByMonthOptions"]):I.hidePropertiesIn(t,e,["customIcon","customTitle","customEventDateTime","customDescription","customGroupHeader"]),e.groupEvents||I.hidePropertiesIn(t,e,["groupAttribute","groupByKey","groupByDayOptions","groupByMonthOptions","ungroupedEventsPosition","customGroupHeader"]),e.groupByKey){case"day":I.hidePropertyIn(t,e,"groupByMonthOptions");break;case"month":I.hidePropertyIn(t,e,"groupByDayOptions");break;default:I.hidePropertiesIn(t,e,["groupByDayOptions","groupByMonthOptions"])}return t};` + +// timelineCase is one configuration and whether prop is hidden under it. +type timelineCase struct { + prop string + cv string // customVisualization + ge string // groupEvents + key string // groupByKey + hidden bool +} + +var timelineCases = []timelineCase{ + // Rule (1), the ternary's THEN branch inside the switch head. + {"groupByDayOptions", "true", "true", "day", true}, // measured: CE0463 + {"groupByMonthOptions", "true", "true", "month", true}, // measured: CE0463 + {"title", "true", "true", "day", true}, + {"description", "true", "true", "day", true}, + {"timeIndication", "true", "true", "day", true}, + {"icon", "true", "true", "day", true}, + {"title", "false", "true", "day", false}, + {"icon", "false", "true", "day", false}, + // Rule (2), its ELSE branch — recognised before; the control. + {"customTitle", "false", "true", "day", true}, + {"customTitle", "true", "true", "day", false}, + // Rule (3) — recognised before; the control. + {"ungroupedEventsPosition", "false", "false", "day", true}, + {"ungroupedEventsPosition", "false", "true", "day", false}, + // Rules (4)–(6), the switch's arms. + {"groupByMonthOptions", "false", "true", "day", true}, // (4) measured: CE0463 + {"groupByDayOptions", "false", "true", "month", true}, // (5) measured: CE0463 + {"groupByDayOptions", "false", "true", "year", true}, // (6) measured: CE0463 + {"groupByMonthOptions", "false", "true", "year", true}, // (6) measured: CE0463 + {"groupByDayOptions", "false", "true", "day", false}, // measured: clean + {"groupByMonthOptions", "false", "true", "month", false}, // measured: clean +} + +func checkTimelineCases(t *testing.T, rules []types.WidgetVisibilityRule) { + t.Helper() + for _, c := range timelineCases { + values := map[string]string{"customVisualization": c.cv, "groupEvents": c.ge, "groupByKey": c.key} + hidden, found := ruleHidesUnder(rules, c.prop, values) + if !found && c.hidden { + t.Errorf("%s: no rule at all; want hidden under %v", c.prop, values) + continue + } + if hidden != c.hidden { + t.Errorf("%s under %v: hidden=%v, want %v", c.prop, values, hidden, c.hidden) + } + } +} + +func TestTimelineHideRules_AllSixRecognised(t *testing.T) { + rules, stats := extractVisibilityRulesFromJS(timelineGetProperties) + if stats.TotalHideCalls != 6 || stats.Recognized != 6 { + t.Errorf("coverage %d of %d; want 6 of 6", stats.Recognized, stats.TotalHideCalls) + } + checkTimelineCases(t, rules) +} + +// The same through the committed package (testdata/expr-checker ships Timeline +// 3.2.2), which is what `widget describe` and the .def.json generator read. +func TestTimelineHideRules_FromTheInstalledPackage(t *testing.T) { + const mpkPath = "../../testdata/expr-checker/widgets/com.mendix.widget.web.Timeline.mpk" + rules, recognized, total := ExtractWidgetVisibilityStats(mpkPath, "com.mendix.widget.web.timeline.Timeline") + if total != 6 || recognized != 6 { + t.Errorf("%d of %d editor hide-rules recognized; want 6 of 6", recognized, total) + } + checkTimelineCases(t, rules) +} + +// A switch's arms, including the shapes Timeline does not use: grouped labels, +// fall-through into the next arm, a default arm that is not last, comma and +// semicolon siblings. +func TestSwitchArms_FallThroughAndDefault(t *testing.T) { + js := `exports.getProperties=function(e,t){switch(e.k){` + + `case"a":x.hidePropertyIn(t,e,"p1");` + // falls through into b/c + `case"b":case"c":x.hidePropertyIn(t,e,"p2");break;` + + `case"d":x.hidePropertyIn(t,e,"p3"),x.hidePropertyIn(t,e,"p4");x.hidePropertyIn(t,e,"p5");break;` + + `default:x.hidePropertyIn(t,e,"p6");` + // falls through into e + `case"e":x.hidePropertyIn(t,e,"p7")` + + `}return t}` + rules, stats := extractVisibilityRulesFromJS(js) + if stats.Recognized != stats.TotalHideCalls { + t.Errorf("coverage %d of %d", stats.Recognized, stats.TotalHideCalls) + } + hiddenFor := map[string][]string{ + "p1": {"a"}, + "p2": {"a", "b", "c"}, + "p3": {"d"}, "p4": {"d"}, "p5": {"d"}, + "p6": {"z"}, // only a value no case names + "p7": {"e", "z"}, // its own case, and the default that falls into it + } + for prop, want := range hiddenFor { + for _, k := range []string{"a", "b", "c", "d", "e", "z"} { + hidden, _ := ruleHidesUnder(rules, prop, map[string]string{"k": k}) + if hidden != slices.Contains(want, k) { + t.Errorf("%s with k=%q: hidden=%v, want %v", prop, k, hidden, slices.Contains(want, k)) + } + } + } +} + +// Shapes a switch rule cannot state soundly are lifted as nothing — the safe +// direction, which is what each was before. +func TestSwitchArms_UnrepresentableShapesProduceNoRule(t *testing.T) { + cases := map[string]string{ + // the arm's hide carries its own guard: hidden on `k=="a" AND flag` + "q1": `function(e,t){switch(e.k){case"a":e.flag&&x.hidePropertyIn(t,e,"q1");break}return t}`, + // a label that is not a literal makes the default arm's complement unknown + "q2": `function(e,t){switch(e.k){case n:x.hidePropertyIn(t,e,"q0");break;default:x.hidePropertyIn(t,e,"q2")}return t}`, + // a switch under an `if` runs only when the if holds + "q3": `function(e,t){if(e.on){switch(e.k){case"a":x.hidePropertyIn(t,e,"q3")}}return t}`, + // the discriminant is not one of the widget's properties (the platform) + "q4": `function(e,t,r){switch(r){case"web":x.hidePropertyIn(t,e,"q4")}return t}`, + // an arm every value reaches is unconditional, which no condition states + "q5": `function(e,t){switch(e.k){default:x.hidePropertyIn(t,e,"q5")}return t}`, + // whether the arm before falls through cannot be read off its last + // statement — a block, a conditional break — so q6's arm is not known + "q6": `function(e,t){switch(e.k){case"a":{x.hidePropertyIn(t,e,"q0");break}case"b":x.hidePropertyIn(t,e,"q6")}return t}`, + "q7": `function(e,t){switch(e.k){case"a":if(e.f)break;x.hidePropertyIn(t,e,"q0");case"b":x.hidePropertyIn(t,e,"q7")}return t}`, + } + for prop, js := range cases { + rules, _ := extractVisibilityRulesFromJS(js) + if r := findRule(rules, prop); r != nil { + t.Errorf("%s: lifted %+v from %s; want no rule", prop, r, js) + } + } +} + +// A switch inside an object list's forEach reads the item. +func TestSwitchArms_NestedItemScope(t *testing.T) { + js := `exports.getProperties=function(e,t){e.items.forEach((function(r,n){switch(r.mode){` + + `case"a":x.hideNestedPropertiesIn(t,e,"items",n,["s1"]);break;` + + `default:x.hidePropertyIn(t,e,"items",n,"s2")}}));return t}` + rules, _ := extractVisibilityRulesFromJS(js) + s1, s2 := findRule(rules, "s1"), findRule(rules, "s2") + if s1 == nil || s1.ListPropertyKey != "items" || s1.HiddenWhen == nil || + *s1.HiddenWhen != (types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "eq", Value: "a", Scope: types.ConditionScopeItem}) { + t.Errorf("s1 = %+v; want items/s1 hidden when the item's mode is \"a\"", s1) + } + if s2 == nil || s2.ListPropertyKey != "items" || s2.HiddenWhen == nil || + *s2.HiddenWhen != (types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "ne", Value: "a", Scope: types.ConditionScopeItem}) { + t.Errorf("s2 = %+v; want items/s2 hidden when the item's mode is not \"a\"", s2) + } +} + +// terser folds statements into an `if(` head the same way it does a `switch(`; +// a CALL's paren is different, because the call can itself be conditional. +func TestStatementHeadParenIsNotAGroup(t *testing.T) { + js := `exports.getProperties=function(e,t){` + + `if(e.a?x.hidePropertyIn(t,e,"r1"):x.hidePropertyIn(t,e,"r2"),e.b)return t;` + + `e.y&&foo(e.x&&x.hidePropertyIn(t,e,"r3"));return t}` + rules, _ := extractVisibilityRulesFromJS(js) + if h, _ := ruleHidesUnder(rules, "r1", map[string]string{"a": "true"}); !h { + t.Error("r1: want hidden when a is set") + } + if h, _ := ruleHidesUnder(rules, "r1", map[string]string{"a": "false"}); h { + t.Error("r1: hidden when a is not set") + } + if h, _ := ruleHidesUnder(rules, "r2", map[string]string{"a": "false"}); !h { + t.Error("r2 (the else branch, the control): want hidden when a is not set") + } + if r := findRule(rules, "r3"); r != nil { + t.Errorf("r3: lifted %+v; a guard inside a call's arguments is one conjunct of `e.y && …`", r) + } +} + +// fileUploaderGetProperties is File Uploader 2.5.0's getProperties, verbatim from +// the marketplace package (content 235351). +const fileUploaderGetProperties = `exports.getProperties=function(e,t){return e.readOnlyMode?A.hidePropertiesIn(t,e,["files"===e.uploadMode?"associatedImages":"associatedFiles","createImageAction","createFileAction","allowedFileFormats","maxFilesPerUpload","maxFilesPerBatch","maxFileSize","objectCreationTimeout"]):"files"===e.uploadMode?(A.hidePropertiesIn(t,e,["associatedImages","createImageAction"]),function(e,t){var r;e.allowedFileFormats.forEach(function(r,n){"simple"===r.configMode?A.hideNestedPropertiesIn(t,e,"allowedFileFormats",n,["mimeType","extensions","typeFormatDescription"]):A.hideNestedPropertiesIn(t,e,"allowedFileFormats",n,["predefinedType"])});var n=null===(r=t[0].properties)||void 0===r?void 0:r.find(function(e){return"allowedFileFormats"===e.key});if(n){var o;n.objectHeaders=["File type",""],null===(o=n.objects)||void 0===o||o.forEach(function(t,r){var n=e.allowedFileFormats[r];"simple"===n.configMode?t.captions=[E[n.predefinedType].description," "]:t.captions=[n.typeFormatDescription,"".concat(z(n.mimeType,n.extensions))]})}}(e,t)):A.hidePropertiesIn(t,e,["associatedFiles","createFileAction","allowedFileFormats"]),"files"===e.uploadMode?A.hidePropertiesIn(t,e,["onUploadSuccessImage","onUploadFailureImage"]):A.hidePropertiesIn(t,e,["onUploadSuccessFile","onUploadFailureFile"]),e.enableCustomButtons?e.customButtons.forEach(function(r,n){A.hideNestedPropertiesIn(t,e,"customButtons",n,"files"===e.uploadMode?["buttonActionImage"]:["buttonActionFile"])}):A.hidePropertiesIn(t,e,["customButtons"]),t};` + +// ako/mxcli#258 item 1: the ternary-wrapped `hidePropertiesIn` that gates the +// create* actions on uploadMode. It was unread when filed ("6 of 9"); the +// else-if chain reading added for mendixlabs/mxcli#1583 lifts it, so this pins +// that, and that the switch/statement-head changes for #1762 leave it alone. +func TestFileUploaderUploadModeGate_258(t *testing.T) { + rules, _ := extractVisibilityRulesFromJS(fileUploaderGetProperties) + cases := []struct { + prop, readOnly, mode string + hidden bool + }{ + {"createImageAction", "false", "files", true}, + {"createImageAction", "false", "images", false}, + {"createImageAction", "true", "images", true}, + {"createFileAction", "false", "images", true}, + {"createFileAction", "false", "files", false}, + {"createFileAction", "true", "files", true}, + {"onUploadSuccessImage", "false", "files", true}, + {"onUploadSuccessImage", "false", "images", false}, + {"onUploadSuccessFile", "false", "images", true}, + {"onUploadSuccessFile", "false", "files", false}, + } + for _, c := range cases { + values := map[string]string{"readOnlyMode": c.readOnly, "uploadMode": c.mode} + if hidden, _ := ruleHidesUnder(rules, c.prop, values); hidden != c.hidden { + t.Errorf("%s under %v: hidden=%v, want %v", c.prop, values, hidden, c.hidden) + } + } +} diff --git a/mdl/executor/validate_timeline_hidden_1762_test.go b/mdl/executor/validate_timeline_hidden_1762_test.go new file mode 100644 index 000000000..b5e4302ee --- /dev/null +++ b/mdl/executor/validate_timeline_hidden_1762_test.go @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: Apache-2.0 + +// A Timeline enumeration its editorConfig hides, set to a non-default value +// (ako/mxcli#1762, mendixlabs/mxcli#1668). Reported: `mxcli check` and `exec` +// both pass, and mx check reports "[CE0463] The definition of this widget has +// changed" at Timeline 'tlD' and 'tlG'. +// +// The check already refused this shape for the one rule it recognised — +// `groupEvents: false` plus a non-default hidden value is an MDL-WIDGET10 +// error, "a non-default value there fails the build with CE0463" — so the +// remaining rules get the same verdict rather than a second policy: refused +// when the script names the value, written at the default when it does not. +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// timelineVariants mirrors the measurement in editorconfig_switch_1762_test.go: +// one widget per configuration, with what mx check said about it on 11.13.0. +var timelineVariants = []struct { + name, props string + hiddenProp string // "" for a control mx check passed +}{ + {"tlB", `customVisualization: true`, ""}, + {"tlD", `customVisualization: true, groupByKey: 'day', groupByDayOptions: 'dayMonth'`, "groupByDayOptions"}, + {"tlDm", `customVisualization: true, groupByKey: 'month', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions"}, + {"tlC", `groupByKey: 'day', groupByDayOptions: 'dayMonth'`, ""}, + {"tlM", `groupByKey: 'month', groupByMonthOptions: 'monthYear'`, ""}, + {"tlG", `groupByKey: 'day', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions"}, + {"tlF", `groupByKey: 'month', groupByDayOptions: 'fullDate'`, "groupByDayOptions"}, + {"tlY", `groupByKey: 'year', groupByDayOptions: 'dayMonth'`, "groupByDayOptions"}, + {"tlYm", `groupByKey: 'year', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions"}, +} + +func timelinePage(name, props string) string { + return "create page ModTL.P_" + name + " (Title: 'T', Layout: Atlas_Core.Atlas_Default) {\n" + + " timeline " + name + " (data: MICROFLOW ModTL.DS_Events, groupAttribute: At, " + props + ")\n};\n" +} + +func widget10For(vs []linter.Violation, widget string) []linter.Violation { + var out []linter.Violation + for _, v := range vs { + if v.RuleID == "MDL-WIDGET10" && strings.Contains(v.Message, "widget `"+widget+"`") { + out = append(out, v) + } + } + return out +} + +// check -p against the installed Timeline 3.2.2 agrees with mxbuild on every +// measured variant. +func TestTimelineHiddenEnumeration_CheckAgreesWithMxbuild(t *testing.T) { + project := projectFixture(t) + var src strings.Builder + for _, v := range timelineVariants { + src.WriteString(timelinePage(v.name, v.props)) + } + prog, errs := visitor.Build(src.String()) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + all := ValidateWidgetProperties(prog, project) + + for _, v := range timelineVariants { + got := widget10For(all, v.name) + if v.hiddenProp == "" { + for _, g := range got { + if g.Severity == linter.SeverityError { + t.Errorf("%s (mx check: 0 errors): refused with %s", v.name, g.Message) + } + } + continue + } + refused := false + for _, g := range got { + if g.Severity == linter.SeverityError && strings.Contains(g.Message, "`"+v.hiddenProp+"`") && + strings.Contains(g.Message, "CE0463") { + refused = true + } + } + if !refused { + t.Errorf("%s (mx check: CE0463): want an MDL-WIDGET10 error naming `%s`, got %+v", v.name, v.hiddenProp, got) + } + } + // The default arm's rule says which values hide it, not "matches". + if got := widget10For(all, "tlY"); len(got) == 0 || + !strings.Contains(got[0].Message, "`groupByKey` is none of \"day\", \"month\"") { + t.Errorf("tlY: message %+v; want it to name the values the default arm covers", got) + } +} + +// The value split the check already makes: a hidden enumeration set to its +// default is redundant, not a build failure, so it stays a warning. +func TestTimelineHiddenEnumeration_DefaultIsOnlyAWarning(t *testing.T) { + project := projectFixture(t) + prog, errs := visitor.Build(timelinePage("tlDef", `customVisualization: true, groupByDayOptions: 'dayName'`)) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + got := widget10For(ValidateWidgetProperties(prog, project), "tlDef") + if len(got) != 1 || got[0].Severity != linter.SeverityWarning { + t.Fatalf("got %+v; want one MDL-WIDGET10 warning (the value will be ignored)", got) + } +} + +const timelineSetup = `create module ModTL; +create non-persistent entity ModTL.Ev ( Line: String(200), At: DateTime ); +create microflow ModTL.DS_Events () returns List of ModTL.Ev as $Events +begin + $Events = create list of ModTL.Ev; + return $Events; +end; +` + +// The other half of the policy: a hidden enumeration the script does NOT name is +// written at its default (hiddenUnnamedProperties), now that the rules hiding +// it are known. exec's refusal of a named one is its pre-flight running this +// same check — see cmd/mxcli/exec_preflight_timeline_1762_test.go. +func TestTimelineHiddenEnumeration_UnnamedOnesAreWrittenAtTheirDefault(t *testing.T) { + exec, _ := pageFixtureExecutor(t) + run(t, exec, timelineSetup) + if err := runProgErr(t, exec, timelinePage("tlB", `customVisualization: true`)); err != nil { + t.Fatalf("exec of the variant mx check passed: %v", err) + } + id := unitIDByName(t, exec, "page", "P_tlB") + for key, want := range map[string]string{"groupByDayOptions": "dayName", "groupByMonthOptions": "month"} { + if got := storedPrimitive854(t, exec, id, "tlB", key); got != want { + t.Errorf("%s stored %q; a hidden enumeration must be stored at its default %q", key, got, want) + } + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 8b82c3878..7f3d8737c 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -939,11 +939,30 @@ func visibilityCondWord(c *types.WidgetVisibilityCondition) string { return "is enabled" case "falsy": return "is disabled" + case "empty": + return "is empty" + case "notempty": + return "is not empty" + case "in": + return "is one of " + quotedCSV(c.Value) + case "notin": + // A switch's default arm (Timeline's `groupByKey` other than day/month). + return "is none of " + quotedCSV(c.Value) default: return "matches" } } +// quotedCSV renders an `in`/`notin` condition's comma-separated values as a +// quoted, comma-separated list. +func quotedCSV(csv string) string { + parts := strings.Split(csv, ",") + for i, p := range parts { + parts[i] = fmt.Sprintf("%q", p) + } + return strings.Join(parts, ", ") +} + // validateObjectListItemEnums flags an enumeration sub-property of an object-list // item (e.g. a Maps marker `LocationType`, a chart series `Interpolation`) whose // value isn't one of the widget's declared member keys. Studio Pro silently diff --git a/mdl/executor/widget_defs.go b/mdl/executor/widget_defs.go index 3aca88263..f2841c8fa 100644 --- a/mdl/executor/widget_defs.go +++ b/mdl/executor/widget_defs.go @@ -95,9 +95,8 @@ func RefreshWidgetDefinitions(projectPath string, force bool, output io.Writer) // Lift property-visibility rules from the widget's editorConfig.js // (#574 Phase 2) so the generated .def.json carries the version- // specific applicability logic. Merge with the hand-transcribed table - // rather than replace it: the static extractor skips compound/ternary - // guards (e.g. Timeline's `customVisualization ? hide([title,...])`), - // so the hand-authored fallback fills the keys the extractor misses. + // rather than replace it: the static extractor skips some guard + // shapes, so the hand-authored fallback fills the keys it misses. // Extracted rules win on conflict (they're version-specific). if rules := extractVisibilityRulesFromMPK(mpkPath, mpkDef.ID); len(rules) > 0 { defJSON.PropertyVisibility = mergeVisibilityRules(rules, widgetVisibilityRules[mpkDef.ID]) @@ -442,20 +441,26 @@ func GenerateDefJSON(mpkDef *mpk.WidgetDefinition, mdlName string) *WidgetDefini // configurations. Until the JS extractor lands (#574 Phase 2), these are // transcribed by hand from each widget's compiled editorConfig.js. // -// Only TextTemplate-typed hidden properties need entries: the populated-vs-null -// ClientTemplate choice is what triggers CE0463. Properties hidden as -// Expression/enum/Widgets slots don't carry a ClientTemplate and are omitted. +// The entries cover TextTemplate properties only, and that is NOT a statement +// that only TextTemplates matter. Any hidden property must hold its default: an +// enumeration hidden at a non-default value is CE0463 just the same (Timeline's +// groupByDayOptions in custom mode, ako/mxcli#1762). This table once made that +// rule look recognised — `check` warned about the Timeline's hidden templates, +// from here, while the enumerations in the very same hide went unchecked. // // VideoPlayer (editorConfig.js): // "expression"===e.type && hidePropertiesIn(["videoUrl","posterUrl"]) // Timeline (editorConfig.js): // e.customVisualization ? hidePropertiesIn(["title","description","icon","timeIndication",...]) : ... // +// The extractor now lifts the whole Timeline hide (3.1.1, 3.2.2 and 3.2.3 all +// read 6 of 6), so its entry only matters for a package that ships no +// editorConfig. +// // mergeVisibilityRules returns the extracted rules plus any hand-authored fallback // rules whose PropertyKey the extractor did not cover. Extracted rules are // version-specific (lifted from the installed .mpk's editorConfig.js) and win on -// conflict; the fallback fills the compound/ternary guards the static extractor -// skips (e.g. Timeline `title`/`description` hidden when `customVisualization`). +// conflict; the fallback fills guards the static extractor skips. func mergeVisibilityRules(extracted, fallback []types.WidgetVisibilityRule) []types.WidgetVisibilityRule { if len(fallback) == 0 { return extracted diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index d008a6bcd..ae79fc557 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -100,7 +100,12 @@ const defaultSlotContainer = "template" // `editable:` when the package declares no Editability); before, it was // known-but-unmapped and the Calendar's `editable` could not be written // (mendixlabs/mxcli#1578). -const WidgetDefGeneratorVersion = 19 +// 20 — lift editorConfig hides in a `switch` arm, and a guard bounded by the +// head of a `switch(`/`if(` statement (terser folds earlier statements +// into it). Timeline went from 2 to 6 of 6 rules, so its hidden +// enumerations are refused with MDL-WIDGET10 rather than reaching mx check +// as CE0463 (ako/mxcli#1762). Bump forces existing projects to regenerate. +const WidgetDefGeneratorVersion = 20 // WidgetDefinition describes how to construct a pluggable widget from MDL syntax. // Loaded from embedded JSON definition files (*.def.json). From afe48e67c72c17c8e8e281d73ac844cde5b2e861 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 23:02:50 +0000 Subject: [PATCH 20/61] fix(describe): leave out a pluggable widget property it hides at its default Once the Timeline's hide rules are read (ako/mxcli#1762), one of its two format enumerations is always hidden: by customVisualization, or by the groupByKey switch. The generic pluggable-widget describe printed every stored primitive, so each Timeline came back with a hidden property at its default and drew an MDL-WIDGET10 warning on mxcli's own output (11 warnings on six measured pages). The same happened before for any widget hiding an enumeration by a rule the extractor already read, such as a Timeline with `groupEvents: false`. Such a property carries nothing: exec writes a hidden property the script leaves out at that same default. extractExplicitProperties now drops a top-level property that the widget's own rules hide in the stored configuration while it equals the stored schema's default. The rules are the registry's, the ones `check` evaluates, so the two agree about what is hidden. A hidden property holding a non-default value is still printed, because that widget fails mx check with CE0463, and printing it lets `check` say so. describe -> check -> exec on three Timeline configurations: no MDL-WIDGET10, and the stored page is byte-identical afterwards. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...meline-hidden-enumeration-switch-arms.json | 2 +- CHANGELOG.md | 2 +- docs-site/src/guides/pluggable-widgets.md | 2 +- mdl/executor/cmd_pages_describe_pluggable.go | 56 ++++++++++++++- ...ages_describe_timeline_hidden_1762_test.go | 70 +++++++++++++++++++ 5 files changed, 128 insertions(+), 4 deletions(-) create mode 100644 mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json index e8d3f6949..ee7743e72 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json @@ -1 +1 @@ -{"area":"mdl/executor","date":"2026-10-28","symptom":"A Timeline enumeration the widget hides, set to a non-default value — `customVisualization: true, groupByDayOptions: 'dayMonth'`, or `groupByKey: 'day', groupByMonthOptions: 'monthYear'` — passes `mxcli check` and `exec`, and mx check reports \"[CE0463] The definition of this widget has changed\" at Timeline 'tlD' / 'tlG'. `mxcli widget describe timeline` says \"2 of 6 editor hide-rules recognized\"; `groupEvents: false` plus a hidden value was already an MDL-WIDGET10 error (ako/mxcli#1762, mendixlabs/mxcli#1668)","cause":"Two editorConfig shapes went unread. terser folds the statements before a `switch` into its head — `switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey)` — so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard rejected as a conditional group (the else branch was lifted, through trailingExpr). And a hide in a `case`/`default` arm had no reading at all: its `case \"day\":` colon reached the ternary-else path and found no `?`. The template warnings the reporter saw in custom mode came from the hand-written fallback table in widget_defs.go, which listed only the TextTemplate keys of that same hide, on the stated premise that enumerations cannot cause CE0463","file":"mdl/executor/editorconfig_switch.go (switchArmCondition, switchArms, statementHeadParen, atFunctionStatementLevel), editorconfig_extract.go (the switch-arm branch before parseGuard; parseGuard's `(` boundary), validate_widgets.go (visibilityCondWord in/notin), widget_engine.go (WidgetDefGeneratorVersion 20)","insight":"**A known rule elsewhere is not the extractor knowing it.** The custom-mode warning made rule (1) look recognised, and it was the reason to ask why the enumerations in the same list were missed: `widget describe` (extractor only) showed no `title` rule, while the generated .def.json had one, merged from the hand table. Compare the two sources first when a rule half-works. **The reporter's framing (`the true branch of a ternary`) was not the cause**: the existing TimelinePattern test already lifted a statement-level THEN branch, and only the minified file showed that terser had moved it into the `switch(` head. Read the shipped JS, not the de-minified source. **Measure the matrix, then make it the oracle**: one widget per variant through `mx check --no-update-widgets` on 11.13.0 gave 6 CE0463 and 3 clean, and the extractor test, the check test and the bug-test all encode those nine. **Policy follows the existing split, not a second one**: hidden + non-default is refused (MDL-WIDGET10 error, exec preflight) and hidden + left out is written at the default, as rule (3) and the column-SET reset (mendixlabs/mxcli#1519) already do. **Bound the blast radius by dumping lifted rules for every installed .mpk before and after**: only Timeline and HeatMap changed. HeatMap folds into an `if(` head the same way, and its new rule is correct. **The scratchpad can be shared between agents**: work in a private subdirectory","refs":["ako/mxcli#1762","mendixlabs/mxcli#1668","ako/mxcli#258","mendixlabs/mxcli#1583"],"ce":["CE0463"],"rules":["MDL-WIDGET10"],"test":"mdl/executor/editorconfig_switch_1762_test.go; validate_timeline_hidden_1762_test.go; cmd/mxcli/exec_preflight_timeline_1762_test.go; mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl"} +{"area":"mdl/executor","date":"2026-10-28","symptom":"A Timeline enumeration the widget hides, set to a non-default value — `customVisualization: true, groupByDayOptions: 'dayMonth'`, or `groupByKey: 'day', groupByMonthOptions: 'monthYear'` — passes `mxcli check` and `exec`, and mx check reports \"[CE0463] The definition of this widget has changed\" at Timeline 'tlD' / 'tlG'. `mxcli widget describe timeline` says \"2 of 6 editor hide-rules recognized\"; `groupEvents: false` plus a hidden value was already an MDL-WIDGET10 error (ako/mxcli#1762, mendixlabs/mxcli#1668)","cause":"Two editorConfig shapes went unread. terser folds the statements before a `switch` into its head — `switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey)` — so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard rejected as a conditional group (the else branch was lifted, through trailingExpr). And a hide in a `case`/`default` arm had no reading at all: its `case \"day\":` colon reached the ternary-else path and found no `?`. The template warnings the reporter saw in custom mode came from the hand-written fallback table in widget_defs.go, which listed only the TextTemplate keys of that same hide, on the stated premise that enumerations cannot cause CE0463","file":"mdl/executor/editorconfig_switch.go (switchArmCondition, switchArms, statementHeadParen, atFunctionStatementLevel), editorconfig_extract.go (the switch-arm branch before parseGuard; parseGuard's `(` boundary), validate_widgets.go (visibilityCondWord in/notin), cmd_pages_describe_pluggable.go (withoutHiddenDefaults), widget_engine.go (WidgetDefGeneratorVersion 20)","insight":"**A known rule elsewhere is not the extractor knowing it.** The custom-mode warning made rule (1) look recognised, and it was the reason to ask why the enumerations in the same list were missed: `widget describe` (extractor only) showed no `title` rule, while the generated .def.json had one, merged from the hand table. Compare the two sources first when a rule half-works. **The reporter's framing (`the true branch of a ternary`) was not the cause**: the existing TimelinePattern test already lifted a statement-level THEN branch, and only the minified file showed that terser had moved it into the `switch(` head. Read the shipped JS, not the de-minified source. **Measure the matrix, then make it the oracle**: one widget per variant through `mx check --no-update-widgets` on 11.13.0 gave 6 CE0463 and 3 clean, and the extractor test, the check test and the bug-test all encode those nine. **Policy follows the existing split, not a second one**: hidden + non-default is refused (MDL-WIDGET10 error, exec preflight) and hidden + left out is written at the default, as rule (3) and the column-SET reset (mendixlabs/mxcli#1519) already do. **Knowing more rules made describe noisy**: every Timeline hides one of its two formats, and the generic describe printed every stored primitive, so its own output drew MDL-WIDGET10 warnings. describe now drops a hidden property at its default, using the same registry rules the check reads. **Bound the blast radius by dumping lifted rules for every installed .mpk before and after**: only Timeline and HeatMap changed. HeatMap folds into an `if(` head the same way, and its new rule is correct. **The scratchpad can be shared between agents**: work in a private subdirectory","refs":["ako/mxcli#1762","mendixlabs/mxcli#1668","ako/mxcli#258","mendixlabs/mxcli#1583"],"ce":["CE0463"],"rules":["MDL-WIDGET10"],"test":"mdl/executor/editorconfig_switch_1762_test.go; validate_timeline_hidden_1762_test.go; cmd_pages_describe_timeline_hidden_1762_test.go; cmd/mxcli/exec_preflight_timeline_1762_test.go; mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl"} diff --git a/CHANGELOG.md b/CHANGELOG.md index c33507cc8..b7a9cfb96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed -- **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. +- **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/docs-site/src/guides/pluggable-widgets.md b/docs-site/src/guides/pluggable-widgets.md index 6ef086397..6d0812a4d 100644 --- a/docs-site/src/guides/pluggable-widgets.md +++ b/docs-site/src/guides/pluggable-widgets.md @@ -55,7 +55,7 @@ version you actually have at the moment you create the widget. So a hidden property you leave out is written at its default; one you set to its default draws an MDL-WIDGET10 warning (the value is ignored); and one you set to anything else is an MDL-WIDGET10 **error**, which `exec` refuses before writing. - For example, a + `describe` leaves out a hidden property that holds its default. For example, a Timeline grouped by `'day'` hides `groupByMonthOptions`, and one with `customVisualization: true` hides both format enumerations. diff --git a/mdl/executor/cmd_pages_describe_pluggable.go b/mdl/executor/cmd_pages_describe_pluggable.go index 91eed9d63..7c1579073 100644 --- a/mdl/executor/cmd_pages_describe_pluggable.go +++ b/mdl/executor/cmd_pages_describe_pluggable.go @@ -6,6 +6,8 @@ import ( "context" "strings" "sync" + + "github.com/mendixlabs/mxcli/mdl/types" ) // buildPropertyTypeKeyMap builds a map from PropertyType $ID to PropertyKey for a CustomWidget. @@ -1320,6 +1322,8 @@ func extractExplicitProperties(ctx *ExecContext, w map[string]any, enclosingEnti } var result []rawExplicitProp + stored := map[string]string{} // primitive values, for the visibility rules + defaults := map[string]string{} // the stored schema's DefaultValue per key props := getBsonArrayElements(obj["Properties"]) for _, prop := range props { propMap, ok := prop.(map[string]any) @@ -1335,6 +1339,10 @@ func extractExplicitProperties(ctx *ExecContext, w map[string]any, enclosingEnti if !ok { continue } + if pv := extractString(value["PrimitiveValue"]); pv != "" { + stored[propKey] = pv + } + defaults[propKey] = schemas[typePointerID].Default // Check for AttributeRef (attribute binding) if attrRef, ok := value["AttributeRef"].(map[string]any); ok && attrRef != nil { @@ -1401,7 +1409,53 @@ func extractExplicitProperties(ctx *ExecContext, w map[string]any, enclosingEnti }) } } - return result + return withoutHiddenDefaults(ctx, extractCustomWidgetID(ctx, w), result, stored, defaults) +} + +// withoutHiddenDefaults drops the described properties the widget's own +// editorConfig hides in its stored configuration while they hold their +// default. Such a property says nothing — exec writes a hidden property the +// script leaves out at that same default (hiddenUnnamedProperties) — and +// printing it drew an MDL-WIDGET10 warning on describe's own output: every +// Timeline hides one of its two format enumerations (ako/mxcli#1762). +// +// The rules are the ones `check` evaluates (the project's widget registry), so +// the two cannot disagree about what is hidden. A hidden property holding a +// NON-default value is kept: that widget fails mx check with CE0463, and +// printing the value is what lets `check` say so instead of describe hiding it. +func withoutHiddenDefaults(ctx *ExecContext, widgetID string, props []rawExplicitProp, stored, defaults map[string]string) []rawExplicitProp { + reg := ctx.GetWidgetRegistry() + if reg == nil || widgetID == "" || len(props) == 0 { + return props + } + def, ok := reg.GetByWidgetID(widgetID) + if !ok { + return props + } + hidden := map[string]bool{} + for _, rule := range visibilityRulesFor(def, reg) { + if rule.Nested() || rule.HiddenWhen == nil { + continue + } + fires, determinable := rule.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + v, ok := stored[c.PropertyKey] + return v, ok + }) + if determinable && fires { + hidden[rule.PropertyKey] = true + } + } + if len(hidden) == 0 { + return props + } + out := props[:0:0] + for _, p := range props { + if hidden[p.Key] && !p.IsRef && len(p.Params) == 0 && defaults[p.Key] != "" && p.Value == defaults[p.Key] { + continue + } + out = append(out, p) + } + return out } // extractImageProperties extracts properties from a pluggable Image CustomWidget. diff --git a/mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go b/mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go new file mode 100644 index 000000000..a46215405 --- /dev/null +++ b/mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" +) + +// describe → check → exec on a Timeline, once the hide rules are known +// (ako/mxcli#1762). One of the two format enumerations is ALWAYS hidden — by +// customVisualization, or by the groupByKey switch — and a generic pluggable +// widget was described with every stored primitive, so each Timeline came back +// with a hidden property at its default and drew an MDL-WIDGET10 warning on +// mxcli's own output. Such a property carries nothing: exec writes a hidden +// property the script does not name at that same default. +func TestTimelineDescribe_HiddenDefaultsAreNotPrinted(t *testing.T) { + exec, out := pageFixtureExecutor(t) + run(t, exec, timelineSetup) + variants := []struct{ name, props, mustPrint, mustNotPrint string }{ + {"tlB", `customVisualization: true`, "customVisualization", "groupByDayOptions"}, + {"tlS", `groupByKey: 'year'`, "groupByKey: 'year'", "groupByMonthOptions"}, + {"tlM", `groupByKey: 'month', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions: 'monthYear'", "groupByDayOptions"}, + } + for _, v := range variants { + run(t, exec, timelinePage(v.name, v.props)) + } + for _, v := range variants { + id := unitIDByName(t, exec, "page", "P_"+v.name) + before := storedBytes(t, exec, id) + + out.Reset() + run(t, exec, "describe page ModTL.P_"+v.name) + described := out.String() + if !strings.Contains(described, v.mustPrint) { + t.Errorf("%s: describe lost %q:\n%s", v.name, v.mustPrint, described) + } + if strings.Contains(described, v.mustNotPrint) { + t.Errorf("%s: describe printed the hidden %s:\n%s", v.name, v.mustNotPrint, described) + } + + prog, errs := visitor.Build(described) + if len(errs) > 0 { + t.Fatalf("%s: describe output does not parse: %v\n%s", v.name, errs, described) + } + if got := widget10For(ValidateWidgetProperties(prog, exec.backend.Path()), v.name); len(got) > 0 { + t.Errorf("%s: check of describe's own output reports %+v", v.name, got) + } + + if err := runErr(t, exec, described); err != nil { + t.Fatalf("%s: exec of the description: %v", v.name, err) + } + if !bytes.Equal(before, storedBytes(t, exec, id)) { + t.Errorf("%s: describe → exec changed the stored page", v.name) + } + } +} + +func storedBytes(t *testing.T, exec *Executor, unitID string) []byte { + t.Helper() + raw, err := exec.backend.GetRawUnitBytes(model.ID(unitID)) + if err != nil { + t.Fatal(err) + } + return append([]byte(nil), raw...) +} From 75ee5480565abcf16f8e2f075c4ffe6f835ae9d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 23:16:27 +0000 Subject: [PATCH 21/61] fix(microflows): edit an activity replaced by one of the same kind in place Re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new $IDs, while a decision whose condition changed the same way kept its own, and exec reported both as `spliced: 1 replaced`. Studio Pro compares and merges by $ID, so it showed the edit as one activity removed and another added (ako/mxcli#1761, mendixlabs/mxcli#1667). A splice Replace by one activity of the same activity and action $Type now edits the stored activity: the replacement is written into the stored slot, at the stored size, and its elements are paired structurally with the stored ones (same $Type at the same key; list entries by the member, parameter or language they name; unnamed entries by position only for equal $Type sequences). The stored $IDs go on at Bytes, rewriting every reference in the same pass, so the executor can still name the fragment's activity until then. This applies at top level, in a loop body and to an activity with a custom error handler, for create or modify and for alter ... replace. A run of adjacent statements each changed in its own place, pairwise of one plain kind, is diffed one-for-one instead of as a replace-and-drop of the whole run, so each is edited in place. The report tells the two outcomes apart: `edited in place` for a node that kept its $ID (an activity of the same kind, a decision's condition, a return's value, a node's notes) and `replaced with new element IDs` for one that did not. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...e-kind-activity-replace-mints-new-ids.json | 1 + CHANGELOG.md | 1 + ...create-or-modify-edit-inside-loop-body.mdl | 5 +- .../1489-error-handler-activity-splices.mdl | 4 +- ...notes-on-decision-and-end-event-splice.mdl | 2 +- .../1610-decision-caption-change-splices.mdl | 2 +- ...1761-same-kind-activity-edit-keeps-ids.mdl | 64 +++ ...e-856-list-operation-where-bare-member.mdl | 4 +- mdl/backend/mfmutator/splice.go | 54 ++- mdl/backend/mfmutator/splice_edit.go | 332 +++++++++++++ .../mfmutator/splice_edit_in_place_test.go | 441 ++++++++++++++++++ mdl/backend/mfmutator/splice_handler_test.go | 92 ++-- mdl/backend/microflow_mutation.go | 10 +- mdl/executor/cmd_alter_flow.go | 24 +- .../cmd_flow_loop_body_edit_pedapp_test.go | 18 +- mdl/executor/cmd_flow_modify.go | 59 ++- .../flow_splice_decision_1694_pedapp_test.go | 12 +- ...w_splice_edit_in_place_1761_pedapp_test.go | 364 +++++++++++++++ .../flow_splice_error_handler_pedapp_test.go | 42 +- ...ow_splice_list_op_where_856_pedapp_test.go | 17 +- .../flow_splice_loop_body_1693_pedapp_test.go | 2 +- .../flow_commit_events_rerun_test.go | 4 +- mdl/roundtrip/flow_idempotent_shapes_test.go | 4 +- mdl/roundtrip/flow_modify_end_test.go | 4 +- mdl/roundtrip/flow_modify_notes_test.go | 4 +- mdl/roundtrip/flow_modify_test.go | 12 +- mdl/roundtrip/flow_respelling_rerun_test.go | 16 +- mdl/roundtrip/flow_splice_guard_test.go | 6 +- .../flow_splice_handler_return_test.go | 2 +- .../flow_splice_member_qualified_test.go | 4 +- mdl/roundtrip/flow_splice_parity_test.go | 4 +- mdl/roundtrip/rerun_idempotent_test.go | 2 +- 32 files changed, 1476 insertions(+), 136 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json create mode 100644 mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl create mode 100644 mdl/backend/mfmutator/splice_edit.go create mode 100644 mdl/backend/mfmutator/splice_edit_in_place_test.go create mode 100644 mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json new file mode 100644 index 000000000..576a61b5a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json @@ -0,0 +1 @@ +{"area":"mdl/backend","date":"2026-10-28","symptom":"Re-running `create or modify microflow|nanoflow` with only a filter's expression changed gave the activity, its ListOperationsAction and its FilterByExpression new $IDs, while a decision whose condition changed the same way kept its own; exec printed `spliced: 1 replaced` for both, and check, mx check, diff and describe were clean, so only a BSON dump or Studio Pro's version control (one activity removed, another added) showed it","cause":"The statement diff turns a changed activity into a Replace, and Mutator.Replace at top level appended the fragment's freshly built activity and removed the stored one. The in-place swaps that existed (swapInPlace: loop bodies #1106, handled activities #1489) kept the stored slot and geometry but still wrote the fragment's $IDs. The patch write (UpdateRawUnitPatch) deliberately skips canon.TransplantIDs, so nothing downstream re-paired them. Separately, two adjacent changed activities form one run, which became 'replace the first by both, drop the second' and could never be an edit","file":"mdl/backend/mfmutator/splice_edit.go (sameKind, swapKeeping, editInPlace, carryPairs, pairEntries, applyCarried), mdl/backend/mfmutator/splice.go (Replace, replaceInLoop, replaceKeepingHandler, Bytes), mdl/executor/cmd_alter_flow.go (noteEdited), mdl/executor/cmd_flow_modify.go (patchSummary, pairwiseSameKind)","fix":"A replace by one activity of the same activity and action $Type is written into the stored slot and paired structurally with the stored activity: same $Type at the same key, list entries by what they name (Attribute/Association plus Type, Parameter, LanguageCode, Name, Key), unnamed entries by position only for equal $Type sequences. The pairs are applied at Bytes over every binary in the unit, so the executor can keep naming the fragment's $ID (LinkNote of a stored note) until then. MicroflowMutator.EditedInPlace feeds the report: `edited in place` vs `replaced with new element IDs`. A run of N changed statements pairwise of one plain kind is split into N one-for-one gaps","test":"mdl/backend/mfmutator/splice_edit_in_place_test.go (six kinds byte-exact, members by name, another kind/operation is new, notes, loop body, error handler); mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go (six kinds x microflow/nanoflow on the real codec, a control that a log for a filter is new, and PedApp's Studio Pro-authored PopulateUserAttributes edited in one run). With the carry stubbed out the engine and executor tests fail with new $IDs; measured on 10.24.24: unfixed 1ccaa1aa->1c24b792 for the activity, fixed all kept, mx check 0 errors","insight":"The identity carry has to happen where the pairing is known, and that is the splice, not canon: the patch writer is told the splice owns its $IDs (ContentsOwnElementIDs), so a fix that hopes Reconcile will re-pair finds it switched off on purpose. Pair by name before position — a member list paired by index hands a removed member's $ID to its neighbour, which is the GUID-class mistake one level down. And apply the carry late: the executor still names the fragment's $IDs after Replace returns (linkStoredNotes draws a stored note's line to the fragment's activity), so the stored $IDs go on at Bytes, where one pass rewrites every reference too. The report half matters as much: a count that prints 'replaced' for both outcomes is why the bug read as harmless; the test that a decision 'kept its IDs' and a filter did not was only visible in a BSON dump","refs":["ako/mxcli#1761","mendixlabs/mxcli#1667","ako/mxcli#1106","mendixlabs/mxcli#1489","ako/mxcli#888"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index b7a9cfb96..9734de71e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. +- **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl b/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl index c3df1a7a6..4fb535f2b 100644 --- a/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl +++ b/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl @@ -14,9 +14,10 @@ -- To reproduce: exec this file against a project, then change -- `formatDecimal($It/Amount, '0.#')` to `toString(round($It/Amount, 1))` and -- exec it again. The second run now reports --- Modified microflow: BugTest1106.BuildJson (spliced: 1 replaced) +-- Modified microflow: BugTest1106.BuildJson (spliced: 1 edited in place) -- and a third run writes nothing. The loop and the other activities keep their --- $IDs and positions; only the edited `set` is replaced, in place. +-- $IDs and positions; the edited `set` is the stored activity with its new +-- expression, and keeps its $ID too (ako/mxcli#1761). -- -- Still refused (needs the body laid out again): a statement added, removed, -- reordered or moved inside the body, or one kind of statement swapped for diff --git a/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl b/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl index 26da19f0c..712d8f341 100644 --- a/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl +++ b/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl @@ -13,8 +13,8 @@ -- -- To reproduce: exec this file, then change `P = $P` to `P = 'x'` (or the -- handler's log text) and exec it again. The second run reports --- `Modified microflow: BugTest1489.Flow (spliced: 1 replaced)`; the handler and --- the merge it rejoins at keep their $IDs and positions. A third run writes +-- `Modified microflow: BugTest1489.Flow (spliced: 1 edited in place)`; the call, +-- the handler and the merge it rejoins at keep their $IDs and positions. A third run writes -- nothing. Changing the handler's kind is still refused. mdl 1; diff --git a/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl b/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl index 0e4e40e0a..bb0b0ef43 100644 --- a/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl +++ b/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl @@ -11,7 +11,7 @@ -- To reproduce: exec this file, then add `@annotation 'guard'` on the line -- before `if` (or `@annotation 'the end'` before `return 'done';`) and exec it -- again. The second run reports `Modified microflow: BugTest1517.F (spliced: --- 1 replaced)`; the decision and the end event keep their $IDs and flows, and +-- 1 edited in place)`; the decision and the end event keep their $IDs and flows, and -- the note is drawn where a full build would draw it. A third run writes -- nothing. A note shared with another activity (`id:`) is still refused. diff --git a/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl b/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl index 86b3f8f82..9bb0254bc 100644 --- a/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl +++ b/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl @@ -14,7 +14,7 @@ -- -- To reproduce: exec this file against a project, then add -- `@caption 'Big?'` on the line before `if` and exec it again. The second run --- reports `Modified microflow: BugTest1610.F (spliced: 1 replaced)`, and a +-- reports `Modified microflow: BugTest1610.F (spliced: 1 edited in place)`, and a -- third run writes nothing. Taking the caption off again captions the -- decision with its condition, as a new one would be. diff --git a/mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl b/mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl new file mode 100644 index 000000000..735410c30 --- /dev/null +++ b/mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl @@ -0,0 +1,64 @@ +-- ============================================================================ +-- ako/mxcli#1761 (mendixlabs/mxcli#1667) — an edited filter gets new $IDs +-- ============================================================================ +-- +-- Re-running `create or modify microflow` where the only change is a filter's +-- expression did not edit the activity, it replaced it. Measured after the +-- second run (Mendix 10.24.28): +-- +-- $BigList: the Microflows$ActionActivity, Microflows$ListOperationsAction +-- and Microflows$FilterByExpression all got new $IDs. [...] The decision's +-- ExclusiveSplit and SplitCondition kept theirs. +-- +-- and exec reported both as `spliced: 1 replaced`. Studio Pro compares and +-- merges by $ID, so it showed the filter as one activity removed and another +-- added; check, mx check, diff and describe all looked fine. +-- +-- An activity replaced by one of the same kind — the same action — is now an +-- edit of the stored one: it keeps its $ID, and so do its action and every +-- element under it that pairs up (a member by its attribute or association, +-- an argument by its parameter). Several activities each changed in their own +-- place are each edited so. +-- +-- To reproduce: exec this file against a project, then change every `> 10` +-- to `>= 10`, `= 7` to `= 8` and `Factor = 2` to `Factor = 3`, and exec it +-- again. The second run reports +-- Modified microflow: BugTest1761.FilterSales (spliced: 1 edited in place) +-- Modified microflow: BugTest1761.Decide (spliced: 1 edited in place) +-- Modified microflow: BugTest1761.Kinds (spliced: 3 edited in place) +-- and `mxcli bson dump --type microflow --object BugTest1761.FilterSales` +-- shows the same $IDs as before it; a third run writes nothing. An activity +-- replaced by another kind — +-- a log where the filter was — still reports `replaced with new element IDs`. + +mdl 1; + +create module BugTest1761; + +create or modify persistent entity BugTest1761.Sale ( Amount: Integer, Note: String(200) ); + +create or modify microflow BugTest1761.FilterSales ( $SaleList: List of BugTest1761.Sale ) +returns List of BugTest1761.Sale as $BigList +begin + $SmallList = filter $SaleList where $currentObject/Amount < 5; + $BigList = filter $SaleList where $currentObject/Amount > 10; + return $BigList; +end; + +create or modify microflow BugTest1761.Decide ( $N: Integer ) returns Boolean as $Result +begin + if $N > 10 then return true; else return false; end if; +end; + +create or modify microflow BugTest1761.Sub ( $Sale: BugTest1761.Sale, $Factor: Integer ) returns Boolean as $Ok +begin + return true; +end; + +create or modify microflow BugTest1761.Kinds ( $SaleList: List of BugTest1761.Sale ) returns Boolean as $Ok +begin + $Found = find $SaleList where $currentObject/Amount > 10; + change $Found (Amount = 7, Note = 'x'); + $Ok = call microflow BugTest1761.Sub (Sale = $Found, Factor = 2); + return $Ok; +end; diff --git a/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl b/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl index 18ab2f0ec..29ee651ef 100644 --- a/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl +++ b/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl @@ -25,10 +25,10 @@ -- -- To reproduce: exec this file against a project. Expected: -- Created microflow: BugTest856.F --- Modified microflow: BugTest856.F (spliced: 1 replaced) +-- Modified microflow: BugTest856.F (spliced: 1 edited in place) -- Before the fix the second statement was refused and exec rolled the whole -- script back. Running the file again sets the declare back and forth, and --- each statement again replaces the declare alone (spliced: 1 replaced). +-- each statement again edits the declare alone (spliced: 1 edited in place). mdl 1; diff --git a/mdl/backend/mfmutator/splice.go b/mdl/backend/mfmutator/splice.go index a1f7a1f6f..e43e7e5c4 100644 --- a/mdl/backend/mfmutator/splice.go +++ b/mdl/backend/mfmutator/splice.go @@ -17,10 +17,14 @@ package mfmutator // every other property of a rewired flow (its $ID, case value, error-handler // flag, the curve at its other end) is kept; // - nodes are moved only to make room, and only by a translation; -// - nothing else is touched, and no $ID is ever rewritten. A removed element -// leaves no reference behind: Save refuses a document in which any binary -// still names an element that was removed (CLAUDE.md rule 1), and one in -// which two elements share an $ID. +// - nothing else is touched, and no stored $ID is ever rewritten. A removed +// element leaves no reference behind: Save refuses a document in which any +// binary still names an element that was removed (CLAUDE.md rule 1), and +// one in which two elements share an $ID; +// - an activity replaced by one of the same kind is edited, not replaced: +// the replacement takes the stored activity's $IDs, and every reference to +// it is rewritten with them in the same pass (splice_edit.go, +// ako/mxcli#1761). // // What the splice cannot do safely it refuses, with the reason: an insert // after a decision (which branch?), before an activity several flows enter @@ -81,6 +85,11 @@ type Mutator struct { // pinnedIn are the same for the nodes of a loop body, by the loop's $ID, // in the loop box's coordinates (PinInLoop). pinnedIn map[string]map[point]bool + // carried maps the $ID a same-kind replacement was built with to the + // stored $ID it takes over, put on at Bytes (ako/mxcli#1761, + // splice_edit.go); edited are the stored activities a Replace edited so. + carried map[string][]byte + edited map[string]bool } var _ backend.MicroflowMutator = (*Mutator)(nil) @@ -96,11 +105,15 @@ func New(doc bson.D, unitID model.ID, deps Deps) (*Mutator, error) { if dDoc(doc, "ObjectCollection") == nil { return nil, fmt.Errorf("unit %s has no object collection", unitID) } - return &Mutator{deps: deps, unitID: unitID, doc: doc, removed: map[string]bool{}}, nil + return &Mutator{deps: deps, unitID: unitID, doc: doc, removed: map[string]bool{}, + carried: map[string][]byte{}, edited: map[string]bool{}}, nil } // Bytes returns the patched unit, after the integrity checks Save applies. +// An activity a replace edited in place takes its stored identity back first +// (applyCarried). func (m *Mutator) Bytes() ([]byte, error) { + m.applyCarried() if err := m.checkIntegrity(); err != nil { return nil, err } @@ -467,6 +480,11 @@ func (m *Mutator) Replace(target model.ID, frag *backend.MicroflowFragment) erro if err != nil { return err } + // One activity of the same kind is an edit of the stored one, which keeps + // its identity (ako/mxcli#1761). + if edited, err := m.editInPlace(g, x, frag); edited || err != nil { + return err + } y := g.nodes[out.dest] if y == nil { return fmt.Errorf("flow %s points at an object that is not in the flow", out.id) @@ -528,7 +546,8 @@ func (m *Mutator) Replace(target model.ID, frag *backend.MicroflowFragment) erro // swapped in place (ako/mxcli#1106): the new activity takes the stored one's // place in the loop's collection, is drawn where it was at the size it was, // and the flows that entered and left it now enter and leave the new one, so -// nothing else in the body moves. By several (ako/mxcli#1693) the fragment +// nothing else in the body moves; of the same kind, it also takes the stored +// one's identity (ako/mxcli#1761). By several (ako/mxcli#1693) the fragment // starts where the activity started, what is past it in the body moves along // when the fragment is longer, and the loop box grows when the body no longer // fits it. The activity may be the body's first (no flow in) or last (no flow @@ -554,7 +573,7 @@ func (m *Mutator) replaceInLoop(g *graph, x *node, frag *backend.MicroflowFragme if d, err := m.serializeInPlace(x, frag); err != nil { return err } else if dString(d, "$Type") == x.typ { - return m.swapInPlace(g, x, dDoc(loop.doc, "ObjectCollection"), d, frag.Entry) + return m.swapKeeping(g, x, dDoc(loop.doc, "ObjectCollection"), d, frag.Entry) } } in := g.incoming(x.id) @@ -828,7 +847,8 @@ func (m *Mutator) dropInLoop(g *graph, x *node) error { // handler by one activity handling errors the same way, in place // (mendixlabs/mxcli#1489): the new activity takes the stored one's place, is // drawn where it was at the size it was, and every flow that entered or left -// it — its error flow included — now enters or leaves the new one. The +// it — its error flow included — now enters or leaves the new one; of the +// same kind, it takes the stored one's identity too (ako/mxcli#1761). The // handler's own activities, and the merge its path rejoins at, are not // touched: changing one of them is a change of its own, addressed by its own // handle. Anything else — more than one activity, or one that does not raise @@ -864,22 +884,12 @@ func (m *Mutator) replaceKeepingHandler(g *graph, x *node, frag *backend.Microfl return fmt.Errorf("cannot replace %s by a %s: it has an error handler, which the splice keeps only on an "+ "activity", describeNode(x), t) } - if err := m.swapInPlace(g, x, dDoc(m.doc, "ObjectCollection"), d, frag.Entry); err != nil { + if err := m.swapKeeping(g, x, dDoc(m.doc, "ObjectCollection"), d, frag.Entry); err != nil { return err } - if notes == nil { - return nil - } - // A note drawn relative to the activity moves with it, from where the - // builder drew the activity to where the stored one is; one whose canvas - // position the script states stays there (mendixlabs/mxcli#1598). - for _, n := range notes.frag.Objects { - if !notes.frag.Pinned[n.GetID()] { - p := n.GetPosition() - n.SetPosition(model.Point{X: p.X + x.pos.X - notes.entryAt.X, Y: p.Y + x.pos.Y - notes.entryAt.Y}) - } - } - return m.addFragment(notes.frag) + // The notes move with the activity, from where the builder drew it to + // where the stored one is. + return m.addNotesAt(notes, x.pos) } // fragmentNotes are the notes a one-activity fragment attaches to its diff --git a/mdl/backend/mfmutator/splice_edit.go b/mdl/backend/mfmutator/splice_edit.go new file mode 100644 index 000000000..45830c419 --- /dev/null +++ b/mdl/backend/mfmutator/splice_edit.go @@ -0,0 +1,332 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mfmutator + +import ( + "bytes" + "fmt" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/model" +) + +// # Editing an activity in place (ako/mxcli#1761) +// +// A replace by one activity of the same kind — the same activity $Type and +// the same action $Type — is an edit of the stored activity, the way Studio +// Pro edits one: the activity keeps its $ID, and so does every element under +// it that the new content has at the same place. Studio Pro compares and +// merges by $ID, so a replace that minted new ones showed an edited filter as +// one activity removed and another added, and turned a teammate's edit of the +// same activity on another branch into a conflict — while a decision whose +// condition changed the same way kept its $IDs (SetCondition). +// +// The replacement is serialized from the statement as for any replace and put +// where the stored activity is (swapInPlace). Its elements are then paired +// with the stored ones structurally (carryPairs), never by a shared value: +// +// - two elements pair when they have the same $Type at the same place: the +// activity itself, or the same key of two paired elements; +// - in a list, entries that name what they are about pair by that name — a +// member change by its attribute or association and whether it sets, adds +// or removes; a parameter mapping by its parameter; a translation by its +// language — so a reordered member keeps its $ID, and a removed member's +// $ID goes onto no other member. A list of unnamed entries pairs by +// position, and only when both lists hold the same $Types in the same +// order; +// - anything else is new: an element whose $Type differs (a find where a +// filter was, a custom range where a constant one was), and everything +// under it. Its stored counterpart is removed, so a reference left to it +// is refused at Save as for any removed element. +// +// The stored $IDs go on at Bytes, not at the replace. Until then the +// replacement carries the $IDs its fragment was built with, which the +// executor still names it by: a stored note linked to it after the replace +// (ako/mxcli#1695), a note the statement states. At Bytes every reference to +// a carried element is rewritten with it, in one pass over every binary in +// the unit, so no pointer is left naming the built $ID (CLAUDE.md rule 1). +// +// What is written is the replacement's content, as before: the statement +// states the activity whole (describe prints every property MDL has a +// spelling for), and the writer completes the property set Studio Pro writes +// (canon.CompletePropertySets). Only which $IDs it carries changes. + +// sameKind reports whether d, a serialized replacement, is the same kind of +// activity as stored: both action activities, whose actions have one $Type. +func sameKind(d, stored bson.D) bool { + const activity = "Microflows$ActionActivity" + if dString(d, "$Type") != activity || dString(stored, "$Type") != activity { + return false + } + na, sa := dDoc(d, "Action"), dDoc(stored, "Action") + return na != nil && sa != nil && dString(na, "$Type") != "" && dString(na, "$Type") == dString(sa, "$Type") +} + +// swapKeeping is swapInPlace that, when d is the same kind of activity as x +// (sameKind), carries x's identity onto it: x's $ID and those of the elements +// under it that pair up (carryPairs), put on at Bytes. +func (m *Mutator) swapKeeping(g *graph, x *node, oc, d bson.D, id model.ID) error { + same := sameKind(d, x.doc) + if err := m.swapInPlace(g, x, oc, d, id); err != nil { + return err + } + if same { + carryPairs(d, x.doc, m.carried) + m.edited[x.id] = true + } + return nil +} + +// EditedInPlace reports whether a Replace of target edited the stored +// activity, keeping its identity, rather than putting new elements in its +// place. +func (m *Mutator) EditedInPlace(target model.ID) bool { return m.edited[string(target)] } + +// editInPlace replaces the top-level activity x by frag's one activity when +// the two are the same kind, and reports whether it did. The replacement is +// written where x is in the collection, drawn where the script states — or, +// when it states nothing, where x is — at x's size (a Studio Pro activity may +// have been resized), and takes x's identity (swapKeeping). The notes frag +// states for the activity come with it, drawn where they are relative to it. +// Anything else — more than one activity, or another kind — is left to the +// replace, with nothing changed. +func (m *Mutator) editInPlace(g *graph, x *node, frag *backend.MicroflowFragment) (bool, error) { + rest, notes := withoutNotes(frag) + if !singleActivity(rest) { + return false, nil + } + obj := rest.Objects[0] + d, err := m.deps.SerializeObject(obj) + if err != nil { + return false, fmt.Errorf("serialize %T: %w", obj, err) + } + if d == nil || !sameKind(d, x.doc) { + return false, nil + } + at := x.pos + if frag.Placed { + p := obj.GetPosition() + at = point{p.X, p.Y} + } + obj.SetPosition(model.Point{X: at.X, Y: at.Y}) + dSet(d, "RelativeMiddlePoint", at.String()) + dSet(d, "Size", x.size.String()) + if err := m.swapKeeping(g, x, dDoc(m.doc, "ObjectCollection"), d, rest.Entry); err != nil { + return true, err + } + return true, m.addNotesAt(notes, at) +} + +// addNotesAt adds the notes a one-activity fragment attaches to its activity, +// now drawn at at: a note drawn relative to the activity moves with it, from +// where the builder drew the activity; one whose canvas position the script +// states stays there (mendixlabs/mxcli#1598). +func (m *Mutator) addNotesAt(notes *fragmentNotes, at point) error { + if notes == nil { + return nil + } + for _, n := range notes.frag.Objects { + if !notes.frag.Pinned[n.GetID()] { + p := n.GetPosition() + n.SetPosition(model.Point{X: p.X + at.X - notes.entryAt.X, Y: p.Y + at.Y - notes.entryAt.Y}) + } + } + return m.addFragment(notes.frag) +} + +// carryPairs records in into, for d and every element under it, the $ID of +// the element of stored it stands for: built $ID (its 16 bytes as a string) → +// stored $ID. d stands for stored when the two have one $Type; see the +// package comment above for how their children and list entries pair. +func carryPairs(d, stored bson.D, into map[string][]byte) { + if t := dString(d, "$Type"); t == "" || t != dString(stored, "$Type") { + return + } + if n, s := elementID(d), elementID(stored); n != nil && s != nil { + into[string(n)] = bytes.Clone(s) + } + for _, e := range d { + if e.Key == "$ID" || e.Key == "$Type" { + continue + } + switch nv := e.Value.(type) { + case bson.D: + if sd, ok := dGet(stored, e.Key).(bson.D); ok { + carryPairs(nv, sd, into) + } + case bson.A: + if sa, ok := dGet(stored, e.Key).(bson.A); ok { + for _, p := range pairEntries(nv, sa) { + carryPairs(p[0], p[1], into) + } + } + } + } +} + +// elementID is d's 16-byte binary $ID, or nil. +func elementID(d bson.D) []byte { + if b, ok := dGet(d, "$ID").(bson.Binary); ok && len(b.Data) == 16 { + return b.Data + } + return nil +} + +// entryNames are the properties by which a list entry names what it is about, +// in the order they are tried: a member change names its attribute or its +// association (one of the two is empty), a parameter mapping its parameter, a +// translation its language. +var entryNames = []string{"Attribute", "Association", "Parameter", "LanguageCode", "Name", "Key"} + +// entryKey is what a list entry is about — its $Type, the first of +// entryNames it states, and its Type where it has one (a member change that +// sets an association is not one that adds to it) — or "" for an entry that +// names nothing. +func entryKey(d bson.D) string { + for _, k := range entryNames { + if s := dString(d, k); s != "" { + key := dString(d, "$Type") + "\x00" + k + "=" + s + if t := dString(d, "Type"); t != "" { + key += "\x00" + t + } + return key + } + } + return "" +} + +// pairEntries pairs the entries of two lists of elements: by entryKey when +// every entry on both sides names what it is about (the n-th entry with a key +// with the n-th stored one with that key), by position when none does and the +// two lists hold the same $Types in the same order, and not at all otherwise. +func pairEntries(news, olds bson.A) [][2]bson.D { + nd, ok1 := entryDocs(news) + od, ok2 := entryDocs(olds) + if !ok1 || !ok2 || len(nd) == 0 || len(od) == 0 { + return nil + } + named, unnamed := 0, 0 + for _, d := range append(append([]bson.D(nil), nd...), od...) { + if entryKey(d) == "" { + unnamed++ + } else { + named++ + } + } + var out [][2]bson.D + switch { + case unnamed == 0: + byKey := map[string][]bson.D{} + for _, d := range od { + k := entryKey(d) + byKey[k] = append(byKey[k], d) + } + for _, d := range nd { + k := entryKey(d) + if q := byKey[k]; len(q) > 0 { + out = append(out, [2]bson.D{d, q[0]}) + byKey[k] = q[1:] + } + } + case named == 0: + if len(nd) != len(od) { + return nil + } + for i := range nd { + if dString(nd[i], "$Type") != dString(od[i], "$Type") { + return nil + } + } + for i := range nd { + out = append(out, [2]bson.D{nd[i], od[i]}) + } + } + return out +} + +// entryDocs returns the elements of a Mendix list, without its marker; false +// when it holds anything but elements (strings, pointers). +func entryDocs(a bson.A) ([]bson.D, bool) { + var out []bson.D + for i, v := range a { + switch t := v.(type) { + case bson.D: + if dString(t, "$Type") == "" { + return nil, false + } + out = append(out, t) + case int32: + if i != 0 { + return nil, false + } + default: + return nil, false + } + } + return out, true +} + +// applyCarried puts on each carried element the stored $ID it takes over, and +// rewrites every reference to it in the same pass: every 16-byte binary in the +// unit that holds the built $ID. A pair applies only while the element is in +// the unit and no element holds the stored $ID — an operation after the +// replace that took the replacement out again leaves the stored $ID removed. +// The stored $IDs that go back on are no longer removed elements. +func (m *Mutator) applyCarried() { + if len(m.carried) == 0 { + return + } + held := map[string]bool{} + collectIDs(m.doc, held) + sub := map[string][]byte{} + taken := map[string]bool{} + for n, s := range m.carried { + if held[n] && !held[string(s)] && !taken[string(s)] { + sub[n] = s + taken[string(s)] = true + } + } + substituteIDs(m.doc, sub) + for _, s := range sub { + delete(m.removed, string(s)) + } + m.carried = map[string][]byte{} +} + +// substituteIDs rewrites, in place, every 16-byte binary in v that sub maps. +func substituteIDs(v any, sub map[string][]byte) { + repl := func(x any) (any, bool) { + b, ok := x.(bson.Binary) + if !ok || len(b.Data) != 16 { + return nil, false + } + if s, ok := sub[string(b.Data)]; ok { + return bson.Binary{Subtype: b.Subtype, Data: bytes.Clone(s)}, true + } + return nil, true + } + switch t := v.(type) { + case bson.D: + for i := range t { + if r, isBin := repl(t[i].Value); isBin { + if r != nil { + t[i].Value = r + } + continue + } + substituteIDs(t[i].Value, sub) + } + case bson.A: + for i := range t { + if r, isBin := repl(t[i]); isBin { + if r != nil { + t[i] = r + } + continue + } + substituteIDs(t[i], sub) + } + } +} diff --git a/mdl/backend/mfmutator/splice_edit_in_place_test.go b/mdl/backend/mfmutator/splice_edit_in_place_test.go new file mode 100644 index 000000000..df1e36492 --- /dev/null +++ b/mdl/backend/mfmutator/splice_edit_in_place_test.go @@ -0,0 +1,441 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mfmutator + +import ( + "bytes" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1761 (mendixlabs/mxcli#1667): re-running `create or modify +// microflow` with only a filter's expression changed did not edit the +// activity, it replaced it — the activity, its action and its operation all +// got new $IDs, while a decision whose condition changed the same way kept its +// own. Studio Pro compares and merges by $ID, so it showed the edited filter +// as one activity removed and another added. +// +// A replace by one activity of the same kind (same activity and action $Type) +// is an edit of the stored activity: the new content is written where the +// stored one is, and the stored $IDs are carried onto every element that pairs +// up — the same key of two paired elements, a list entry with the same name +// (an attribute, an association, a parameter), or the same position in a list +// of unnamed entries of the same length. What does not pair is new. + +// editDeps serializes an activity the way fakeDeps does, with action as its +// Action — the document the codec would write for the fragment's statement. +type editDeps struct { + fakeDeps + action bson.D +} + +func (d *editDeps) SerializeObject(o microflows.MicroflowObject) (bson.D, error) { + doc, err := d.fakeDeps.SerializeObject(o) + if err != nil || d.action == nil { + return doc, err + } + if _, ok := o.(*microflows.ActionActivity); ok && !dSet(doc, "Action", d.action) { + doc = append(doc, bson.E{Key: "Action", Value: d.action}) + } + return doc, nil +} + +// el is an element: its $ID is the named one, or a freshly minted one for "", +// as the builder mints them for a fragment. +func el(name, typ string, props ...bson.E) bson.D { + id := bin(name) + if name == "" { + id = bson.Binary{Data: types.UUIDToBlob(types.GenerateID())} + } + return append(bson.D{{Key: "$ID", Value: id}, {Key: "$Type", Value: typ}}, props...) +} + +func kv(k string, v any) bson.E { return bson.E{Key: k, Value: v} } + +// list is a stored list with its marker. +func list(marker int32, els ...bson.D) bson.A { + out := bson.A{marker} + for _, e := range els { + out = append(out, e) + } + return out +} + +// The six kinds the issue names. Each returns the action with the given $IDs +// ("" mints) and the value under test. +func filterAction(act, op, expr string) bson.D { + return el(act, "Microflows$ListOperationsAction", kv("ErrorHandlingType", "Rollback"), + kv("NewOperation", el(op, "Microflows$FilterByExpression", kv("Expression", expr), kv("ListName", "SaleList"))), + kv("ResultVariableName", "BigList")) +} + +func findAction(act, op, expr string) bson.D { + return el(act, "Microflows$ListOperationsAction", kv("ErrorHandlingType", "Rollback"), + kv("NewOperation", el(op, "Microflows$FindByExpression", kv("Expression", expr), kv("ListName", "SaleList"))), + kv("ResultVariableName", "Found")) +} + +func retrieveAction(act, src, rng, sorts, xpath string) bson.D { + return el(act, "Microflows$RetrieveAction", kv("ErrorHandlingType", "Rollback"), + kv("RetrieveSource", el(src, "Microflows$DatabaseRetrieveSource", kv("Entity", "M.Sale"), + kv("Range", el(rng, "Microflows$ConstantRange", kv("SingleObject", false))), + kv("XpathConstraint", xpath), + kv("NewSortings", el(sorts, "Microflows$SortingsList", kv("Sortings", list(2)))))), + kv("ResultVariableName", "Big")) +} + +func memberItem(id, attr, assoc, value string) bson.D { + return el(id, "Microflows$ChangeActionItem", kv("Attribute", attr), kv("Association", assoc), + kv("Type", "Set"), kv("Value", value)) +} + +func changeAction(typ, act string, items ...bson.D) bson.D { + return el(act, typ, kv("ErrorHandlingType", "Rollback"), kv("Items", list(2, items...)), + kv("RefreshInClient", false), kv("Commit", "No"), kv("ChangeVariableName", "Found")) +} + +func mapping(id, param, arg string) bson.D { + return el(id, "Microflows$MicroflowCallParameterMapping", kv("Argument", arg), kv("Parameter", param)) +} + +func callAction(act, call string, mappings ...bson.D) bson.D { + return el(act, "Microflows$MicroflowCallAction", kv("ErrorHandlingType", "Rollback"), + kv("MicroflowCall", el(call, "Microflows$MicroflowCall", kv("Microflow", "M.SUB"), + kv("ParameterMappings", list(2, mappings...)))), + kv("ResultVariableName", "Ok")) +} + +// withAction is line() with the activity a carrying action. +func withAction(action bson.D) []bson.D { + objs := line() + objs[1] = append(objs[1], bson.E{Key: "Action", Value: action}) + return objs +} + +// replaceA replaces a, which holds stored, by one activity serialized with +// replacement, and returns the mutator, what was saved and the fragment. +func replaceA(t *testing.T, stored, replacement bson.D) (*Mutator, []byte, *backend.MicroflowFragment) { + t.Helper() + raw, err := bson.Marshal(unit(withAction(stored), lineFlows())) + if err != nil { + t.Fatal(err) + } + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + deps := &editDeps{action: replacement} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + frag := oneActivity() + if err := m.Replace(model.ID(uid("a")), frag); err != nil { + t.Fatalf("replace: %v", err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + return m, deps.saved, frag +} + +func TestSplice_ReplaceBySameKindEditsTheStoredActivity(t *testing.T) { + for _, tc := range []struct { + name string + // stored is a's action as stored; replacement the fragment's, with + // fresh $IDs; want the replacement with the stored $IDs. + stored, replacement, want bson.D + }{ + {"filter expression", + filterAction("act", "op", "$currentObject/Amount > 10"), + filterAction("", "", "$currentObject/Amount >= 10"), + filterAction("act", "op", "$currentObject/Amount >= 10")}, + {"find expression", + findAction("act", "op", "$currentObject/Amount = 3"), + findAction("", "", "$currentObject/Amount = 4"), + findAction("act", "op", "$currentObject/Amount = 4")}, + {"retrieve where", + retrieveAction("act", "src", "rng", "srt", "[Amount > 100]"), + retrieveAction("", "", "", "", "[Amount > 200]"), + retrieveAction("act", "src", "rng", "srt", "[Amount > 200]")}, + {"change member value", + changeAction("Microflows$ChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "7"), memberItem("i2", "M.Sale.Note", "", "'x'")), + changeAction("Microflows$ChangeAction", "", memberItem("", "M.Sale.Amount", "", "8"), memberItem("", "M.Sale.Note", "", "'x'")), + changeAction("Microflows$ChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "8"), memberItem("i2", "M.Sale.Note", "", "'x'"))}, + {"create member value", + changeAction("Microflows$CreateChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "1")), + changeAction("Microflows$CreateChangeAction", "", memberItem("", "M.Sale.Amount", "", "2")), + changeAction("Microflows$CreateChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "2"))}, + {"call microflow argument", + callAction("act", "call", mapping("p1", "M.SUB.Sale", "$New"), mapping("p2", "M.SUB.Factor", "2")), + callAction("", "", mapping("", "M.SUB.Sale", "$New"), mapping("", "M.SUB.Factor", "3")), + callAction("act", "call", mapping("p1", "M.SUB.Sale", "$New"), mapping("p2", "M.SUB.Factor", "3"))}, + } { + t.Run(tc.name, func(t *testing.T) { + m, saved, _ := replaceA(t, tc.stored, tc.replacement) + // What the edit should leave: the stored unit with a's action + // holding the new value under the stored $IDs — a keeps its + // $ID, its slot in the collection, its place and size, and the + // flows through it are untouched. + want, _ := bson.Marshal(unit(withAction(tc.want), lineFlows())) + if !bytes.Equal(saved, want) { + var got bson.D + _ = bson.Unmarshal(saved, &got) + t.Errorf("the replace did not edit the stored activity in place:\n got %v\nwant %v", got, unit(withAction(tc.want), lineFlows())) + } + if !m.EditedInPlace(model.ID(uid("a"))) { + t.Error("EditedInPlace reports a swapped activity") + } + }) + } +} + +// itemIDs maps each entry of the list at path under a's action to its $ID, by +// the entry's name property. +func itemIDs(t *testing.T, saved []byte, nameKey string, path ...string) map[string]string { + t.Helper() + var d bson.D + if err := bson.Unmarshal(saved, &d); err != nil { + t.Fatal(err) + } + m, _ := New(d, "unit", &fakeDeps{}) + a := m.graph().nodes[uid("a")] + if a == nil { + t.Fatal("a is not in the unit: the activity got a new $ID") + } + cur := dDoc(a.doc, "Action") + for _, k := range path[:len(path)-1] { + cur = dDoc(cur, k) + } + out := map[string]string{} + for _, e := range arrayElements(dGet(cur, path[len(path)-1])) { + if ed, ok := e.(bson.D); ok { + out[dString(ed, nameKey)+dString(ed, "Association")] = binaryID(dGet(ed, "$ID")) + } + } + return out +} + +// A member list pairs by member, never by position: a reordered member keeps +// its own $ID, a removed member's $ID goes with it — onto no other member — +// and an added member is new. +func TestSplice_ReplaceBySameKindPairsMembersByName(t *testing.T) { + t.Run("change members reordered, one dropped, one added", func(t *testing.T) { + _, saved, _ := replaceA(t, + changeAction("Microflows$ChangeAction", "act", + memberItem("i1", "M.Sale.Amount", "", "7"), + memberItem("i2", "M.Sale.Note", "", "'x'"), + memberItem("i3", "", "M.Sale_Customer", "$Cust")), + changeAction("Microflows$ChangeAction", "", + memberItem("", "M.Sale.Note", "", "'y'"), + memberItem("", "M.Sale.Amount", "", "8"), + memberItem("", "M.Sale.Code", "", "'c'"))) + ids := itemIDs(t, saved, "Attribute", "Items") + if ids["M.Sale.Note"] != uid("i2") || ids["M.Sale.Amount"] != uid("i1") { + t.Errorf("a reordered member lost its $ID: %v", ids) + } + if c := ids["M.Sale.Code"]; c == uid("i1") || c == uid("i2") || c == uid("i3") || c == "" { + t.Errorf("the added member took a stored $ID: %s", c) + } + if bytes.Contains(saved, types.UUIDToBlob(uid("i3"))) { + t.Error("the dropped association member's $ID is still in the unit") + } + }) + t.Run("call arguments reordered, one dropped", func(t *testing.T) { + _, saved, _ := replaceA(t, + callAction("act", "call", mapping("p1", "M.SUB.Sale", "$New"), mapping("p2", "M.SUB.Factor", "2")), + callAction("", "", mapping("", "M.SUB.Factor", "3"))) + ids := itemIDs(t, saved, "Parameter", "MicroflowCall", "ParameterMappings") + if len(ids) != 1 || ids["M.SUB.Factor"] != uid("p2") { + t.Errorf("want Factor alone, keeping p2's $ID, got %v", ids) + } + if bytes.Contains(saved, types.UUIDToBlob(uid("p1"))) { + t.Error("the dropped argument's $ID is still in the unit") + } + }) + t.Run("an association set member is not an add member", func(t *testing.T) { + add := memberItem("", "", "M.Sale_Customer", "$Cust") + dSet(add, "Type", "Add") + _, saved, _ := replaceA(t, + changeAction("Microflows$ChangeAction", "act", memberItem("i3", "", "M.Sale_Customer", "$Cust")), + changeAction("Microflows$ChangeAction", "", add)) + if bytes.Contains(saved, types.UUIDToBlob(uid("i3"))) { + t.Error("a member that now adds to the association kept the $ID of the one that set it") + } + }) +} + +// A replace by another kind of activity is not an edit: the activity is new, +// with new $IDs throughout, and the stored one is gone. The same action with +// another operation keeps the activity and the action, and only the operation +// is new. +func TestSplice_ReplaceByAnotherKindIsNew(t *testing.T) { + t.Run("another action", func(t *testing.T) { + m, saved, frag := replaceA(t, + filterAction("act", "op", "$currentObject/Amount > 10"), + changeAction("Microflows$ChangeAction", "", memberItem("", "M.Sale.Amount", "", "8"))) + for _, gone := range []string{"a", "act", "op"} { + if bytes.Contains(saved, types.UUIDToBlob(uid(gone))) { + t.Errorf("%s's $ID is still in the unit", gone) + } + } + if !edges(t, saved)[[2]string{uid("start"), string(frag.Entry)}] { + t.Error("the flow into the replaced activity does not enter the new one") + } + if m.EditedInPlace(model.ID(uid("a"))) { + t.Error("EditedInPlace reports an activity of another kind as edited") + } + }) + t.Run("another operation", func(t *testing.T) { + _, saved, _ := replaceA(t, + filterAction("act", "op", "$currentObject/Amount > 10"), + findAction("", "", "$currentObject/Amount > 10")) + var d bson.D + _ = bson.Unmarshal(saved, &d) + m, _ := New(d, "unit", &fakeDeps{}) + a := m.graph().nodes[uid("a")] + if a == nil { + t.Fatal("the activity got a new $ID") + } + act := dDoc(a.doc, "Action") + if binaryID(dGet(act, "$ID")) != uid("act") { + t.Error("the action got a new $ID") + } + op := dDoc(act, "NewOperation") + if dString(op, "$Type") != "Microflows$FindByExpression" || binaryID(dGet(op, "$ID")) == uid("op") { + t.Errorf("the find kept the filter's $ID: %v", op) + } + if bytes.Contains(saved, types.UUIDToBlob(uid("op"))) { + t.Error("the filter's $ID is still in the unit") + } + }) +} + +// The notes a create or modify states for the activity come with the edit, +// their lines attached to the stored activity; and a stored note linked to the +// fragment's activity after the replace (LinkNote, ako/mxcli#1695) is linked +// to the stored one. +func TestSplice_ReplaceBySameKindKeepsNotesOnTheStoredActivity(t *testing.T) { + objs := append(withAction(filterAction("act", "op", "x > 10")), obj("old", "Microflows$Annotation", 250, 330)) + raw, _ := bson.Marshal(unit(objs, lineFlows())) + var d bson.D + _ = bson.Unmarshal(raw, &d) + deps := &editDeps{action: filterAction("", "", "x >= 10")} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + frag := oneActivity() + note := µflows.Annotation{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Position: model.Point{X: 360, Y: 100}, + }, Caption: "edited"} + frag.Objects = append(frag.Objects, note) + frag.AnnotationFlows = []*microflows.AnnotationFlow{{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + OriginID: note.ID, DestinationID: frag.Entry, + }} + if err := m.Replace(model.ID(uid("a")), frag); err != nil { + t.Fatal(err) + } + if err := m.LinkNote(model.ID(uid("old")), frag.Entry); err != nil { + t.Fatalf("link the stored note to the replacement: %v", err) + } + if err := m.Save(); err != nil { + t.Fatal(err) + } + if bytes.Contains(deps.saved, types.UUIDToBlob(string(frag.Entry))) { + t.Error("the fragment's activity $ID is in the unit: the stored activity was not kept") + } + var got bson.D + _ = bson.Unmarshal(deps.saved, &got) + g, _ := New(got, "unit", &fakeDeps{}) + lines := map[string]bool{} + for _, f := range g.graph().annotationFlows(uid("a")) { + lines[f.origin] = true + } + if !lines[string(note.ID)] || !lines[uid("old")] { + t.Errorf("want the stated note and the stored one attached to a, got %v", lines) + } + if p := g.graph().nodes[string(note.ID)]; p == nil || p.pos != (point{250, 100}) { + t.Errorf("the note is not drawn where it is relative to the activity: %v", p) + } +} + +// Inside a loop body, and on an activity with a custom error handler, the +// replace already swapped one activity in place (ako/mxcli#1106, +// mendixlabs/mxcli#1489); of the same kind, it now keeps the stored identity +// there too. +func TestSplice_ReplaceBySameKindInPlaceKeepsIdentityEverywhere(t *testing.T) { + t.Run("in a loop body", func(t *testing.T) { + inner := append(obj("inner", "Microflows$ActionActivity", 100, 60), bson.E{Key: "Action", Value: filterAction("act", "op", "x > 10")}) + loop := append(obj("loop", "Microflows$LoopedActivity", 420, 200), bson.E{Key: "ObjectCollection", Value: bson.D{ + {Key: "$ID", Value: bin("loopoc")}, + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), inner, obj("inner2", "Microflows$ActionActivity", 260, 60)}}, + }}) + raw, _ := bson.Marshal(unit([]bson.D{obj("start", "Microflows$StartEvent", 100, 200), loop}, + []bson.D{flow("f1", "start", "loop", 1, 3, false), flow("fi", "inner", "inner2", 1, 3, false)})) + var d bson.D + _ = bson.Unmarshal(raw, &d) + deps := &editDeps{action: filterAction("", "", "x >= 10")} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + if err := m.Replace(model.ID(uid("inner")), oneActivity()); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatal(err) + } + for _, kept := range []string{"inner", "act", "op"} { + if !bytes.Contains(deps.saved, types.UUIDToBlob(uid(kept))) { + t.Errorf("%s's $ID is gone", kept) + } + } + if !edges(t, deps.saved)[[2]string{uid("inner"), uid("inner2")}] { + t.Error("the body flow no longer leaves the stored activity") + } + if !m.EditedInPlace(model.ID(uid("inner"))) { + t.Error("EditedInPlace reports a swapped activity") + } + }) + t.Run("with an error handler", func(t *testing.T) { + objs, flows := handlerFlow() + act := callAction("act", "call", mapping("p1", "M.SUB.Factor", "2")) + dSet(act, "ErrorHandlingType", string(microflows.ErrorHandlingTypeCustomWithoutRollback)) + objs[1] = append(objs[1][:len(objs[1])-1], bson.E{Key: "Action", Value: act}) + raw, _ := bson.Marshal(unit(objs, flows)) + var d bson.D + _ = bson.Unmarshal(raw, &d) + repl := callAction("", "", mapping("", "M.SUB.Factor", "3")) + dSet(repl, "ErrorHandlingType", string(microflows.ErrorHandlingTypeCustomWithoutRollback)) + deps := &editDeps{action: repl} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + if err := m.Replace(model.ID(uid("a")), oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback)); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatal(err) + } + for _, kept := range []string{"a", "act", "call", "p1"} { + if !bytes.Contains(deps.saved, types.UUIDToBlob(uid(kept))) { + t.Errorf("%s's $ID is gone", kept) + } + } + e := edges(t, deps.saved) + if !e[[2]string{uid("a"), uid("h")}] || !e[[2]string{uid("start"), uid("a")}] { + t.Errorf("the flows through the activity, its error flow included, do not run through the stored one: %v", e) + } + }) +} diff --git a/mdl/backend/mfmutator/splice_handler_test.go b/mdl/backend/mfmutator/splice_handler_test.go index 5f0df41dd..7e4eb0099 100644 --- a/mdl/backend/mfmutator/splice_handler_test.go +++ b/mdl/backend/mfmutator/splice_handler_test.go @@ -60,35 +60,69 @@ func handlerFlow() ([]bson.D, []bson.D) { } func TestSplice_ReplaceKeepsTheErrorHandler(t *testing.T) { - objs, flows := handlerFlow() - m, deps := newMutator(t, unit(objs, flows)) - frag := oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback) - if err := m.Replace(model.ID(uid("a")), frag); err != nil { - t.Fatal(err) - } - if err := m.Save(); err != nil { - t.Fatalf("save: %v", err) - } - // What the replace should leave: the stored unit with a's document - // swapped for the new one, drawn where a was at a's size, and the three - // flows through a — in, out, and the error flow — pointing at it. - newID := bson.Binary{Data: types.UUIDToBlob(string(frag.Entry))} - wantObjs, wantFlows := handlerFlow() - wantObjs[1] = withEH(bson.D{ - {Key: "$ID", Value: newID}, - {Key: "$Type", Value: "Microflows$ActionActivity"}, - {Key: "RelativeMiddlePoint", Value: "250;200"}, - {Key: "Size", Value: "160;60"}, - }, microflows.ErrorHandlingTypeCustomWithoutRollback) - dSet(wantFlows[0], "DestinationPointer", newID) - dSet(wantFlows[1], "OriginPointer", newID) - dSet(wantFlows[2], "OriginPointer", newID) - raw, _ := bson.Marshal(unit(wantObjs, wantFlows)) - if !bytes.Equal(deps.saved, raw) { - var got bson.D - _ = bson.Unmarshal(deps.saved, &got) - t.Errorf("the replace changed more than the activity and the flows through it:\n got %v\nwant %v", got, unit(wantObjs, wantFlows)) - } + // The same kind of activity (a call for a call) is an edit of the stored + // one, which keeps its $ID (ako/mxcli#1761): the replacement equal to it, + // the unit is the stored one, byte for byte. + t.Run("the same kind of activity", func(t *testing.T) { + objs, flows := handlerFlow() + m, deps := newMutator(t, unit(objs, flows)) + if err := m.Replace(model.ID(uid("a")), oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback)); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + raw, _ := bson.Marshal(unit(handlerFlow())) + if !bytes.Equal(deps.saved, raw) { + var got bson.D + _ = bson.Unmarshal(deps.saved, &got) + t.Errorf("the replace by the same activity changed the unit:\n got %v\nwant %v", got, unit(handlerFlow())) + } + }) + // Another kind is a new activity: what the replace should leave is the + // stored unit with a's document swapped for the new one, drawn where a + // was at a's size, and the three flows through a — in, out, and the + // error flow — pointing at it. + t.Run("another kind of activity", func(t *testing.T) { + objs, flows := handlerFlow() + raw, _ := bson.Marshal(unit(objs, flows)) + var d bson.D + _ = bson.Unmarshal(raw, &d) + java := bson.D{ + {Key: "$Type", Value: "Microflows$JavaActionCallAction"}, + {Key: "ErrorHandlingType", Value: string(microflows.ErrorHandlingTypeCustomWithoutRollback)}, + } + deps := &editDeps{action: java} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + frag := oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback) + if err := m.Replace(model.ID(uid("a")), frag); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + newID := bson.Binary{Data: types.UUIDToBlob(string(frag.Entry))} + wantObjs, wantFlows := handlerFlow() + wantObjs[1] = bson.D{ + {Key: "$ID", Value: newID}, + {Key: "$Type", Value: "Microflows$ActionActivity"}, + {Key: "RelativeMiddlePoint", Value: "250;200"}, + {Key: "Size", Value: "160;60"}, + {Key: "Action", Value: java}, + } + dSet(wantFlows[0], "DestinationPointer", newID) + dSet(wantFlows[1], "OriginPointer", newID) + dSet(wantFlows[2], "OriginPointer", newID) + want, _ := bson.Marshal(unit(wantObjs, wantFlows)) + if !bytes.Equal(deps.saved, want) { + var got bson.D + _ = bson.Unmarshal(deps.saved, &got) + t.Errorf("the replace changed more than the activity and the flows through it:\n got %v\nwant %v", got, unit(wantObjs, wantFlows)) + } + }) } func TestSplice_ReplaceOfAHandledActivityRefusals(t *testing.T) { diff --git a/mdl/backend/microflow_mutation.go b/mdl/backend/microflow_mutation.go index 8b8b4f085..9ae785682 100644 --- a/mdl/backend/microflow_mutation.go +++ b/mdl/backend/microflow_mutation.go @@ -47,8 +47,16 @@ type MicroflowMutator interface { InsertAfter(target model.ID, frag *MicroflowFragment) error // InsertBefore splices frag onto the one flow that enters target. InsertBefore(target model.ID, frag *MicroflowFragment) error - // Replace puts frag in target's place and removes target. + // Replace puts frag in target's place and removes target. A target + // replaced by one activity of the same kind — the same activity and + // action — is edited instead: the new content is written onto the stored + // activity, which keeps its $ID and those of the elements under it that + // pair up (ako/mxcli#1761). Replace(target model.ID, frag *MicroflowFragment) error + // EditedInPlace reports whether a Replace of target edited the stored + // activity, keeping its identity, rather than putting new elements in + // its place — what the report of the change says. + EditedInPlace(target model.ID) bool // Drop removes target and joins its incoming flows to its successor. A // decision of the shape `alter … insert` produces goes with its paths and // its merge (ako/mxcli#1694); any other decision is refused. diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go index 06ed6c4de..30d6c192e 100644 --- a/mdl/executor/cmd_alter_flow.go +++ b/mdl/executor/cmd_alter_flow.go @@ -151,6 +151,7 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato if err := a.setNodeNotes(ctx, mut, op, target); err != nil { return fail(err) } + a.noteEdited(op) continue } if op.SetCondition { @@ -164,6 +165,7 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato return fail(err) } a.noteRewritten(target, "if "+expr) + a.noteEdited(op) continue } if ret, ok := returnValueEdit(op, target); ok { @@ -177,6 +179,7 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato return fail(err) } a.noteRewritten(target, "return "+value) + a.noteEdited(op) continue } frag, vars, err := a.buildFragmentVars(ctx, op.Body) @@ -234,7 +237,11 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato return nil }) } - err = mut.Replace(target.ID, frag) + if err = mut.Replace(target.ID, frag); err == nil && mut.EditedInPlace(target.ID) { + // One activity of the same kind: the stored one, edited + // (ako/mxcli#1761). + a.noteEdited(op) + } default: err = fmt.Errorf("unknown operation") } @@ -309,6 +316,21 @@ type alterFlowContext struct { // each (ako/mxcli#1695). nil until then — and for the patch of a `create or // modify`, which states its notes and has none to resolve. storedNotes map[string]*microflows.Annotation + + // edited are the replace operations that edited the stored node in place + // — a decision's condition, a return's value, a node's notes, an activity + // replaced by one of the same kind — so that it kept its $ID, as against + // one whose target was swapped for new elements (ako/mxcli#1761). The + // report of a `create or modify` tells the two apart. + edited map[*ast.AlterFlowOperation]bool +} + +// noteEdited records that op edited its stored target in place. +func (a *alterFlowContext) noteEdited(op *ast.AlterFlowOperation) { + if a.edited == nil { + a.edited = map[*ast.AlterFlowOperation]bool{} + } + a.edited[op] = true } // candidatesOf returns the stored candidates with the given $IDs. diff --git a/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go b/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go index 856646225..4f1e2c358 100644 --- a/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go +++ b/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go @@ -114,7 +114,7 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { if err := agreeExec(t, exec, edited); err != nil { t.Fatalf("an expression edit inside a loop body is refused: %v", err) } - if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.LoopEdit (spliced: 1 replaced)") { + if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.LoopEdit (spliced: 1 edited in place)") { t.Errorf("want the edit reported as one splice, got:\n%s", out.String()) } desc := describeFlow(t, exec, "LoopEdit") @@ -122,8 +122,9 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { t.Errorf("the edit was not written:\n%s", desc) } - // Everything but the replaced activity keeps its $ID and its place: the - // loop, its other activity, and the flow around it. + // Every object keeps its $ID and its place: the loop, its other + // activity, and the edited activity itself, which is the stored one + // with a new expression (ako/mxcli#1761) — the same kind of activity. after := flowObjectIDs(loopEditFlowOf(t, exec, "LoopEdit").ObjectCollection) var gone, added []model.ID for id := range before { @@ -138,11 +139,8 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { t.Errorf("object %s moved from %v to %v", id, q, p) } } - if len(gone) != 1 || len(added) != 1 { - t.Fatalf("want exactly the edited activity replaced, got %d gone, %d added", len(gone), len(added)) - } - if before[gone[0]] != after[added[0]] { - t.Errorf("the replacement is drawn at %v, the activity it replaces at %v", after[added[0]], before[gone[0]]) + if len(gone) != 0 || len(added) != 0 { + t.Fatalf("want the edited activity kept, got %d gone, %d added", len(gone), len(added)) } // The re-run is a fixed point. @@ -200,8 +198,8 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { if err := agreeExec(t, exec, src); err != nil { t.Fatalf("refused: %v", err) } - if !strings.Contains(out.String(), "2 replaced") { - t.Errorf("want two in-place replaces, got:\n%s", out.String()) + if !strings.Contains(out.String(), "2 edited in place") { + t.Errorf("want two in-place edits, got:\n%s", out.String()) } desc := describeFlow(t, exec, "LoopEdit") if !strings.Contains(desc, "'still in loop'") || !strings.Contains(desc, "'start'") { diff --git a/mdl/executor/cmd_flow_modify.go b/mdl/executor/cmd_flow_modify.go index d6ceb4329..5e93c59f4 100644 --- a/mdl/executor/cmd_flow_modify.go +++ b/mdl/executor/cmd_flow_modify.go @@ -187,7 +187,7 @@ func modifyFlowInPlace(ctx *ExecContext, d *flowDecl) (handled bool, err error) containerID = to } - switch summary := patchSummary(ops, moves, set); { + switch summary := patchSummary(ops, a.edited, moves, set); { case summary != "": ctx.ReportMutation("Modified", "%s: %s (%s)", d.kind(), d.name, summary) case moved: @@ -1029,6 +1029,17 @@ func (pd *patchDiff) gap(ins []ast.MicroflowStatement, stored []ast.MicroflowSta } } } + // A run of activities each changed in its own place is that many changes + // of one activity, so each can be edited in place (ako/mxcli#1761) rather + // than the run being replaced as a whole. + if pairwiseSameKind(ins, del) { + for i := range ins { + if err := pd.gap(ins[i:i+1], stored, si+i, si+i+1); err != nil { + return err + } + } + return nil + } cands := make([]mfmutator.Candidate, len(del)) for i, st := range del { c, err := pd.loc.locate(st) @@ -1453,6 +1464,16 @@ func (pd *patchDiff) bodyGap(lc mfmutator.Candidate, at ast.MicroflowStatement, return nil } } + // A run of activities each changed in its own place: one at a time, as + // at top level (ako/mxcli#1761). + if pairwiseSameKind(ins, del) { + for i := range ins { + if err := pd.bodyGap(lc, at, ins[i:i+1], stored, si+i, si+i+1); err != nil { + return err + } + } + return nil + } if i, j, ok := loopPair(ins, del); ok { if err := pd.bodyGap(lc, at, ins[:i], stored, si, si+j); err != nil { return err @@ -1612,6 +1633,24 @@ func replaceableInPlace(declared, stored ast.MicroflowStatement) bool { return true } +// pairwiseSameKind reports whether a run of more than one changed statement +// is each declared statement standing where the stored one of the same kind +// stood: as many of each, every pair the same kind of plain activity (not a +// decision, a loop, a return, a break or continue, nor one with an error +// handler). Each pair is then a change of one activity, which the splice +// edits in place when the activity stays the same kind (ako/mxcli#1761). +func pairwiseSameKind(ins, del []ast.MicroflowStatement) bool { + if len(ins) < 2 || len(ins) != len(del) { + return false + } + for i := range ins { + if statementKind(ins[i]) != statementKind(del[i]) || notPlainInBody(ins[i]) != "" || notPlainInBody(del[i]) != "" { + return false + } + } + return true +} + // customErrorHandler reports whether a statement has an `on error` with a // body: a handler path of its own, drawn in the body beside it. func customErrorHandler(st ast.MicroflowStatement) bool { @@ -1988,15 +2027,23 @@ func targetLabel(c mfmutator.Candidate) string { } // patchSummary says what a patch did, e.g. "spliced: 1 inserted, 2 moved; -// set: ExportLevel", or "" when it did nothing. -func patchSummary(ops []*ast.AlterFlowOperation, moves []flowMove, set []string) string { - var ins, rep, drop int +// set: ExportLevel", or "" when it did nothing. A replace that edited its +// stored node in place (edited) — which keeps its $ID, and so reads in Studio +// Pro's version control as that node changed — is told apart from one that +// put new elements in its place, which reads as one removed and another added +// (ako/mxcli#1761). +func patchSummary(ops []*ast.AlterFlowOperation, edited map[*ast.AlterFlowOperation]bool, moves []flowMove, set []string) string { + var ins, edit, rep, drop int for _, op := range ops { switch op.Op { case ast.AlterFlowInsertAfter, ast.AlterFlowInsertBefore: ins++ case ast.AlterFlowReplace, ast.AlterFlowReplaceBody: - rep++ + if edited[op] { + edit++ + } else { + rep++ + } case ast.AlterFlowDrop: drop++ } @@ -2005,7 +2052,7 @@ func patchSummary(ops []*ast.AlterFlowOperation, moves []flowMove, set []string) for _, p := range []struct { n int verb string - }{{ins, "inserted"}, {rep, "replaced"}, {drop, "dropped"}, {len(moves), "moved"}} { + }{{ins, "inserted"}, {edit, "edited in place"}, {rep, "replaced with new element IDs"}, {drop, "dropped"}, {len(moves), "moved"}} { if p.n > 0 { parts = append(parts, fmt.Sprintf("%d %s", p.n, p.verb)) } diff --git a/mdl/executor/flow_splice_decision_1694_pedapp_test.go b/mdl/executor/flow_splice_decision_1694_pedapp_test.go index e14fae397..a44db8580 100644 --- a/mdl/executor/flow_splice_decision_1694_pedapp_test.go +++ b/mdl/executor/flow_splice_decision_1694_pedapp_test.go @@ -96,10 +96,12 @@ func TestSpliceDecision1694_ReRunTakesTheInsertedDecisionOut(t *testing.T) { }) } - // The control for the $ID assertion: an activity the script does change - // shows up as one $ID gone and one added, so the comparison sees churn - // where there is some. (A rebuild is no control here: a flow dropped and - // created again in one session gets its stored identities back.) + // The control for the $ID assertion: an activity the script replaces by + // another kind shows up as one $ID gone and one added, so the comparison + // sees churn where there is some. (A rebuild is no control here: a flow + // dropped and created again in one session gets its stored identities + // back; nor is a changed log text, which edits the stored activity in + // place and keeps its $ID, ako/mxcli#1761.) t.Run("control: a replaced activity is churn", func(t *testing.T) { const name = "D1694Control" original := fmt.Sprintf(dec1694Flow, name) @@ -107,7 +109,7 @@ func TestSpliceDecision1694_ReRunTakesTheInsertedDecisionOut(t *testing.T) { t.Fatal(err) } before := flowObjectIDs(loopEditFlowOf(t, exec, name).ObjectCollection) - if err := agreeExec(t, exec, strings.Replace(original, "'start'", "'begin'", 1)); err != nil { + if err := agreeExec(t, exec, strings.Replace(original, "log info node 'X' 'start';", "commit $F;", 1)); err != nil { t.Fatal(err) } if missing, extra := sameIDs(before, flowObjectIDs(loopEditFlowOf(t, exec, name).ObjectCollection)); missing != 1 || extra != 1 { diff --git a/mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go b/mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go new file mode 100644 index 000000000..e0973e3ce --- /dev/null +++ b/mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go @@ -0,0 +1,364 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "fmt" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// ako/mxcli#1761 (mendixlabs/mxcli#1667, reported on Mendix 10.24.28): re-run +// `create or modify microflow` with only a filter's expression changed, and +// mxcli did not edit the activity, it replaced it: +// +// $BigList: the Microflows$ActionActivity, Microflows$ListOperationsAction +// and Microflows$FilterByExpression all got new $IDs. [...] The decision's +// ExclusiveSplit and SplitCondition kept theirs. +// +// and exec reported both as `spliced: 1 replaced`. Studio Pro compares and +// merges by $ID, so it showed the filter as one activity removed and another +// added. An activity replaced by one of the same kind is now an edit of the +// stored one, which keeps its $ID and those of its action and the elements +// under it; exec says which it was. + +const edit1761Setup = `mdl 1; +create persistent entity MyFirstModule.Sale1761 ( Amount: Integer, Note: String(200) ); +create persistent entity MyFirstModule.Customer1761 ( Name: String(200) ); +create association MyFirstModule.Sale1761_Customer1761 from MyFirstModule.Sale1761 to MyFirstModule.Customer1761; +create microflow MyFirstModule.SUB1761 ( $Sale: MyFirstModule.Sale1761, $Factor: Integer ) returns Boolean as $Ok +begin + return true; +end; +create nanoflow MyFirstModule.NSUB1761 ( $Sale: MyFirstModule.Sale1761, $Factor: Integer ) returns Boolean as $Ok +begin + return true; +end;` + +// edit1761Flow is the flow under test, a microflow or a nanoflow (kind), which +// calls callee with call ("call microflow", "call nanoflow"). +func edit1761Flow(kind, call, callee string) string { + return fmt.Sprintf(`mdl 1; +create or modify %s MyFirstModule.Edit1761 ( $SaleList: List of MyFirstModule.Sale1761, $Cust: MyFirstModule.Customer1761 ) +returns Boolean as $Ok +begin + $Small = filter $SaleList where $currentObject/Amount < 5; + $Found = find $SaleList where $currentObject/Amount = 3; + retrieve $Big from MyFirstModule.Sale1761 where Amount > 100; + change $Found (Amount = 7, Note = 'x', MyFirstModule.Sale1761_Customer1761 = $Cust); + $New = create MyFirstModule.Sale1761 (Amount = 1, Note = 'n'); + $Ok = %s MyFirstModule.%s (Sale = $New, Factor = 2); + return $Ok; +end;`, kind, call, callee) +} + +// storedFlowUnit returns the stored bytes of the microflow or nanoflow +// module.name. +func storedFlowUnit(t *testing.T, exec *Executor, module, name string) []byte { + t.Helper() + ctx := exec.newExecContext(t.Context()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + mfs, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range mfs { + if m.Name == name && h.GetModuleName(h.FindModuleID(m.ContainerID)) == module { + raw, err := ctx.Backend.GetRawUnitBytes(m.ID) + if err != nil { + t.Fatal(err) + } + return append([]byte(nil), raw...) + } + } + nfs, err := ctx.Backend.ListNanoflows() + if err != nil { + t.Fatal(err) + } + for _, n := range nfs { + if n.Name == name && h.GetModuleName(h.FindModuleID(n.ContainerID)) == module { + raw, err := ctx.Backend.GetRawUnitBytes(n.ID) + if err != nil { + t.Fatal(err) + } + return append([]byte(nil), raw...) + } + } + t.Fatalf("%s.%s not found", module, name) + return nil +} + +// storedActivity is one activity of a stored flow: its $ID, the $ID of every +// element under it by path, and how many sequence flows enter and leave it. +type storedActivity struct { + id string + ids map[string]string + in, out int + contents []byte +} + +// findActivity returns the top-level activity whose action is an actType that +// outputs v. +func findActivity(t *testing.T, raw []byte, actType, v string) storedActivity { + t.Helper() + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + oc, _ := afGet(d, "ObjectCollection").(bson.D) + var found bson.D + for _, el := range afList(afGet(oc, "Objects")) { + o, _ := el.(bson.D) + a, _ := afGet(o, "Action").(bson.D) + if a == nil || afGet(a, "$Type") != actType { + continue + } + for _, k := range []string{"ResultVariableName", "VariableName", "ChangeVariableName", "OutputVariableName"} { + if s, _ := afGet(a, k).(string); s == v { + found = o + } + } + } + if found == nil { + t.Fatalf("no %s outputting $%s in the stored flow", actType, v) + } + sa := storedActivity{id: afIDOf(afGet(found, "$ID")), ids: map[string]string{}} + sa.contents, _ = bson.Marshal(found) + var walk func(v any, path string) + walk = func(v any, path string) { + switch n := v.(type) { + case bson.D: + if id, ok := afGet(n, "$ID").(bson.Binary); ok { + sa.ids[path] = types.BlobToUUID(id.Data) + } + for _, e := range n { + walk(e.Value, path+"/"+e.Key) + } + case bson.A: + for i, el := range n { + name := fmt.Sprint(i) + if ed, ok := el.(bson.D); ok { + for _, k := range []string{"Attribute", "Association", "Parameter"} { + if s, _ := afGet(ed, k).(string); s != "" { + name = s + } + } + } + walk(el, path+"["+name+"]") + } + } + } + walk(found, "") + for _, el := range afList(afGet(d, "Flows")) { + f, _ := el.(bson.D) + if afGet(f, "$Type") != "Microflows$SequenceFlow" { + continue + } + if afIDOf(afGet(f, "OriginPointer")) == sa.id { + sa.out++ + } + if afIDOf(afGet(f, "DestinationPointer")) == sa.id { + sa.in++ + } + } + return sa +} + +// assertSameIdentity fails unless after is before with every element $ID +// kept and its flows still running through it. +func assertSameIdentity(t *testing.T, before, after storedActivity) { + t.Helper() + if after.id != before.id { + t.Errorf("the activity got a new $ID: %s -> %s", before.id, after.id) + } + for path, id := range before.ids { + if got := after.ids[path]; got != id { + t.Errorf("%s: $ID %s -> %s", path, id, got) + } + } + for path := range after.ids { + if _, ok := before.ids[path]; !ok { + t.Errorf("%s: an element that was not there", path) + } + } + if after.in != before.in || after.out != before.out { + t.Errorf("flows through the activity: %d in, %d out, want %d in, %d out", after.in, after.out, before.in, before.out) + } +} + +func TestCreateOrModify_SameKindActivityIsEditedInPlace(t *testing.T) { + for _, kind := range []struct{ flow, call, callee string }{ + {"microflow", "call microflow", "SUB1761"}, + {"nanoflow", "call nanoflow", "NSUB1761"}, + } { + base := edit1761Flow(kind.flow, kind.call, kind.callee) + for _, tc := range []struct { + name, from, to string + actType, v string + want string // in the stored activity after the edit + }{ + {"filter expression", "$currentObject/Amount < 5", "$currentObject/Amount <= 5", + "Microflows$ListOperationsAction", "Small", "$currentObject/Amount <= 5"}, + {"find expression", "$currentObject/Amount = 3", "$currentObject/Amount = 4", + "Microflows$ListOperationsAction", "Found", "$currentObject/Amount = 4"}, + {"retrieve where", "where Amount > 100", "where Amount > 200", + "Microflows$RetrieveAction", "Big", "[Amount > 200]"}, + {"change member value", "Amount = 7", "Amount = 8", + "Microflows$ChangeAction", "Found", "8"}, + {"create member value", "Amount = 1", "Amount = 2", + "Microflows$CreateChangeAction", "New", "2"}, + {"call argument", "Factor = 2", "Factor = 3", + "", "Ok", "3"}, + } { + t.Run(kind.flow+"/"+tc.name, func(t *testing.T) { + actType := tc.actType + if actType == "" { + actType = "Microflows$MicroflowCallAction" + if kind.flow == "nanoflow" { + actType = "Microflows$NanoflowCallAction" + } + } + exec, out, _ := openPedAppCopy(t) + if err := agreeExec(t, exec, edit1761Setup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + if err := agreeExec(t, exec, base); err != nil { + t.Fatalf("create: %v\n%s", err, out.String()) + } + before := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), actType, tc.v) + + edited := strings.Replace(base, tc.from, tc.to, 1) + if edited == base { + t.Fatalf("%q is not in the script", tc.from) + } + out.Reset() + if err := agreeExec(t, exec, edited); err != nil { + t.Fatalf("the edit is refused: %v\n%s", err, out.String()) + } + if want := "(spliced: 1 edited in place)"; !strings.Contains(out.String(), want) { + t.Errorf("want %q, got:\n%s", want, out.String()) + } + after := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), actType, tc.v) + assertSameIdentity(t, before, after) + if !bytes.Contains(after.contents, []byte(tc.want)) { + t.Errorf("the new value %q was not written", tc.want) + } + // The edit is the whole of the change: running it again + // writes nothing. + out.Reset() + if err := agreeExec(t, exec, edited); err != nil { + t.Fatal(err) + } + if !strings.Contains(out.String(), "Unchanged") { + t.Errorf("re-running the edited script wrote again:\n%s", out.String()) + } + }) + } + } +} + +// Control: an activity replaced by another kind of activity is new — new +// $IDs, and the report says so. +func TestCreateOrModify_AnotherKindOfActivityIsReplacedWithNewIDs(t *testing.T) { + exec, out, _ := openPedAppCopy(t) + if err := agreeExec(t, exec, edit1761Setup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + base := edit1761Flow("microflow", "call microflow", "SUB1761") + if err := agreeExec(t, exec, base); err != nil { + t.Fatalf("create: %v\n%s", err, out.String()) + } + before := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), "Microflows$ListOperationsAction", "Small") + + // A find in place of the filter: the same action, another operation — + // the activity and its action are the stored ones, the operation is new. + out.Reset() + if err := agreeExec(t, exec, strings.Replace(base, "$Small = filter", "$Small = find", 1)); err != nil { + t.Fatalf("%v\n%s", err, out.String()) + } + after := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), "Microflows$ListOperationsAction", "Small") + if after.id != before.id || after.ids["/Action"] != before.ids["/Action"] { + t.Errorf("the activity or its action got a new $ID") + } + if after.ids["/Action/NewOperation"] == before.ids["/Action/NewOperation"] { + t.Errorf("the find kept the filter's $ID") + } + + // A log in place of the filter is another activity. + out.Reset() + if err := agreeExec(t, exec, strings.Replace(base, "$Small = filter $SaleList where $currentObject/Amount < 5;", + "log info node 'T' 'no filter';", 1)); err != nil { + t.Fatalf("%v\n%s", err, out.String()) + } + if want := "(spliced: 1 replaced with new element IDs)"; !strings.Contains(out.String(), want) { + t.Errorf("want %q, got:\n%s", want, out.String()) + } + if raw := storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"); bytes.Contains(raw, types.UUIDToBlob(before.id)) { + t.Error("the filter's $ID is still in the flow") + } +} + +// The subject that can fail: a flow Studio Pro drew (PedApp's +// FeedbackModule.PopulateUserAttributes), whose retrieve and change share one +// note. Its describe output, with the retrieve's where and one change member +// edited, keeps both activities, their actions and every element under them, +// and the note stays on both. +func TestCreateOrModify_StudioProActivitiesAreEditedInPlace(t *testing.T) { + exec, out, _ := openPedAppCopy(t) + var buf bytes.Buffer + saved := exec.output + exec.output = &buf + if err := afRun(t, exec, "describe microflow FeedbackModule.PopulateUserAttributes;"); err != nil { + t.Fatal(err) + } + exec.output = saved + desc := buf.String() + desc = desc[:strings.Index(desc, "\nend;")+len("\nend;")] + + raw := storedFlowUnit(t, exec, "FeedbackModule", "PopulateUserAttributes") + retrieve := findActivity(t, raw, "Microflows$RetrieveAction", "CurrentUser") + change := findActivity(t, raw, "Microflows$ChangeAction", "Feedback") + notes := func(raw []byte) map[string]int { + var d bson.D + _ = bson.Unmarshal(raw, &d) + n := map[string]int{} + for _, el := range afList(afGet(d, "Flows")) { + f, _ := el.(bson.D) + if afGet(f, "$Type") == "Microflows$AnnotationFlow" { + n[afIDOf(afGet(f, "OriginPointer"))+">"+afIDOf(afGet(f, "DestinationPointer"))]++ + } + } + return n + } + notesBefore := notes(raw) + + edited := strings.Replace(desc, "where id = $currentUser", "where Name = $currentUser/Name", 1) + edited = strings.Replace(edited, "SubmitterDisplayName = $CurrentUser/Name", "SubmitterDisplayName = trim($CurrentUser/Name)", 1) + if edited == desc || strings.Count(edited, "trim(") != 1 || !strings.Contains(edited, "where Name = $currentUser/Name") { + t.Fatalf("the describe output does not hold the statements under test:\n%s", desc) + } + out.Reset() + if err := agreeExec(t, exec, edited); err != nil { + t.Fatalf("%v\n%s", err, out.String()) + } + if want := "(spliced: 2 edited in place)"; !strings.Contains(out.String(), want) { + t.Errorf("want %q, got:\n%s", want, out.String()) + } + raw = storedFlowUnit(t, exec, "FeedbackModule", "PopulateUserAttributes") + assertSameIdentity(t, retrieve, findActivity(t, raw, "Microflows$RetrieveAction", "CurrentUser")) + assertSameIdentity(t, change, findActivity(t, raw, "Microflows$ChangeAction", "Feedback")) + if got := notes(raw); fmt.Sprint(got) != fmt.Sprint(notesBefore) { + t.Errorf("the note's lines changed: %v -> %v", notesBefore, got) + } + if !bytes.Contains(raw, []byte("trim($CurrentUser/Name)")) || !bytes.Contains(raw, []byte("[Name = $currentUser/Name]")) { + t.Error("the edits were not written") + } +} diff --git a/mdl/executor/flow_splice_error_handler_pedapp_test.go b/mdl/executor/flow_splice_error_handler_pedapp_test.go index 794668a5a..2e0a9423e 100644 --- a/mdl/executor/flow_splice_error_handler_pedapp_test.go +++ b/mdl/executor/flow_splice_error_handler_pedapp_test.go @@ -70,9 +70,11 @@ func handlerObjects(t *testing.T, exec *Executor, name string) (map[model.ID]mod } // spliceStep runs src — check first, then exec — and asserts it is spliced -// with exactly `changed` objects replaced and `added` new ones, nothing else -// rebuilt or moved, that describe shows want, and that a second run is -// Unchanged. +// with exactly `changed` objects replaced by new ones and `added` new ones, +// nothing else rebuilt or moved, that describe shows want, and that a second +// run is Unchanged. An activity changed into the same kind of activity is +// edited in place and keeps its $ID (ako/mxcli#1761), so it is not among the +// `changed`: describe showing want is what says it was written. func spliceStep(t *testing.T, exec *Executor, out *bytes.Buffer, name, src string, changed, added int, want ...string) { t.Helper() before, _ := handlerObjects(t, exec, name) @@ -133,19 +135,21 @@ func TestCreateOrModify_ActivityWithErrorHandlerIsSplicedInPlace(t *testing.T) { return "mdl 1;\n" + src[strings.Index(src, "create or modify microflow MyFirstModule.HFlow"):] } + // The call is edited in place (ako/mxcli#1761): it keeps its $ID, and + // describe shows the new argument. t.Run("T1 call argument changed, handler unchanged", func(t *testing.T) { _, call := handlerObjects(t, exec, "HFlow") - spliceStep(t, exec, out, "HFlow", flow(strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1)), 1, 0, + spliceStep(t, exec, out, "HFlow", flow(strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1)), 0, 0, "HSub(P = 'x') on error without rollback begin", "log warning node 'Repro' 'failed';") - if _, now := handlerObjects(t, exec, "HFlow"); now == call { - t.Error("the call kept its $ID: the change was not written") + if _, now := handlerObjects(t, exec, "HFlow"); now != call { + t.Error("the call got a new $ID for a changed argument") } }) t.Run("T3 only the handler's log changed", func(t *testing.T) { _, call := handlerObjects(t, exec, "HFlow") src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1) src = strings.Replace(src, "'failed';", "'failed again';", 1) - spliceStep(t, exec, out, "HFlow", flow(src), 1, 0, "'failed again'", "HSub(P = 'x') on error") + spliceStep(t, exec, out, "HFlow", flow(src), 0, 0, "'failed again'", "HSub(P = 'x') on error") if _, now := handlerObjects(t, exec, "HFlow"); now != call { t.Error("the call was replaced for a change inside its handler") } @@ -153,7 +157,7 @@ func TestCreateOrModify_ActivityWithErrorHandlerIsSplicedInPlace(t *testing.T) { t.Run("call and handler changed together", func(t *testing.T) { src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'y') on", 1) src = strings.Replace(src, "'failed';", "'failed twice';", 1) - spliceStep(t, exec, out, "HFlow", flow(src), 2, 0, "HSub(P = 'y') on error", "'failed twice'") + spliceStep(t, exec, out, "HFlow", flow(src), 0, 0, "HSub(P = 'y') on error", "'failed twice'") }) // A note on the activity is part of it: added or reworded with the // activity unchanged otherwise, it is written, not dropped with the @@ -162,21 +166,21 @@ func TestCreateOrModify_ActivityWithErrorHandlerIsSplicedInPlace(t *testing.T) { src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'y') on", 1) src = strings.Replace(src, "'failed';", "'failed twice';", 1) src = strings.Replace(src, " call microflow MyFirstModule.HSub(", " @annotation 'calls the sub'\n call microflow MyFirstModule.HSub(", 1) - spliceStep(t, exec, out, "HFlow", flow(src), 1, 1, "@annotation 'calls the sub'", "HSub(P = 'y') on error") + spliceStep(t, exec, out, "HFlow", flow(src), 0, 1, "@annotation 'calls the sub'", "HSub(P = 'y') on error") }) t.Run("note on the call reworded", func(t *testing.T) { src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'y') on", 1) src = strings.Replace(src, "'failed';", "'failed twice';", 1) src = strings.Replace(src, " call microflow MyFirstModule.HSub(", " @annotation 'calls the sub, once'\n call microflow MyFirstModule.HSub(", 1) - // The stored note goes out with the activity, the reworded one comes - // in: two objects replaced. - spliceStep(t, exec, out, "HFlow", flow(src), 2, 0, "@annotation 'calls the sub, once'") + // The stored note goes out, the reworded one comes in: one object + // replaced. The call itself is the stored one. + spliceStep(t, exec, out, "HFlow", flow(src), 1, 0, "@annotation 'calls the sub, once'") if desc := describeFlow(t, exec, "HFlow"); strings.Contains(desc, "'calls the sub'") { t.Errorf("the old note is still there:\n%s", desc) } }) t.Run("T4 handler that returns", func(t *testing.T) { - spliceStep(t, exec, out, "HRet", strings.Replace(handlerRetSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1), 1, 0, + spliceStep(t, exec, out, "HRet", strings.Replace(handlerRetSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1), 0, 0, "HSub(P = 'x') on error without rollback begin") }) @@ -248,9 +252,17 @@ func TestAlter_ReplaceOfActivityWithErrorHandler(t *testing.T) { if err := agreeExec(t, exec, src); err != nil { t.Fatalf("refused: %v", err) } + // The same kind of activity: the stored call, edited in place + // (ako/mxcli#1761) — every object keeps its $ID, and describe shows + // the new argument. after, now := handlerObjects(t, exec, "HFlow") - if now == call || len(after) != len(before) { - t.Errorf("want the call alone replaced: %d -> %d objects", len(before), len(after)) + if now != call || len(after) != len(before) { + t.Errorf("want the call edited in place: call %s -> %s, %d -> %d objects", call, now, len(before), len(after)) + } + for id := range before { + if _, ok := after[id]; !ok { + t.Errorf("object %s got a new $ID", id) + } } for id, p := range before { if q, ok := after[id]; ok && p != q { diff --git a/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go b/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go index 9bbbd7ac4..ca39b19fb 100644 --- a/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go +++ b/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go @@ -58,15 +58,16 @@ func TestCreateOrModify_ListOperationWhereBareMemberIsItsStoredActivity(t *testi if strings.Contains(out.String(), "MDL-V1-REBUILD") { t.Errorf("the change fell back to rebuilding the flow:\n%s", out.String()) } - if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.ListOpWhere (spliced: 1 replaced)") { - t.Errorf("want only the changed declare replaced, got:\n%s", out.String()) + if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.ListOpWhere (spliced: 1 edited in place)") { + t.Errorf("want only the changed declare edited, got:\n%s", out.String()) } if got := describeFlow(t, exec, "ListOpWhere"); !strings.Contains(got, "declare $N Integer = 1;") { t.Errorf("the change was not written:\n%s", got) } - // Only the declare is new: the find, the filter and the rest keep - // their $IDs and places. + // Nothing is new: the find, the filter and the rest keep their + // $IDs and places, and so does the declare, edited in place + // (ako/mxcli#1761). after := flowObjectIDs(loopEditFlowOf(t, exec, "ListOpWhere").ObjectCollection) var gone, added []model.ID for id := range before { @@ -81,8 +82,8 @@ func TestCreateOrModify_ListOperationWhereBareMemberIsItsStoredActivity(t *testi t.Errorf("object %s moved from %v to %v", id, q, p) } } - if len(gone) != 1 || len(added) != 1 { - t.Errorf("want exactly the declare replaced (1 gone, 1 new), got %d gone, %d new", len(gone), len(added)) + if len(gone) != 0 || len(added) != 0 { + t.Errorf("want every object kept (0 gone, 0 new), got %d gone, %d new", len(gone), len(added)) } }) } @@ -103,8 +104,8 @@ func TestCreateOrModify_ListOperationWhereChangedPredicateIsAChange(t *testing.T if err := agreeExec(t, exec, strings.Replace(listOpWhereFlow, c.from, c.to, 1)); err != nil { t.Fatalf("%s → %s: %v", c.from, c.to, err) } - if !strings.Contains(out.String(), "(spliced: 1 replaced)") { - t.Errorf("%s → %s: want the operation replaced, got:\n%s", c.from, c.to, out.String()) + if !strings.Contains(out.String(), "(spliced: 1 edited in place)") { + t.Errorf("%s → %s: want the operation edited, got:\n%s", c.from, c.to, out.String()) } // Back to the stored statement for the next case. if err := agreeExec(t, exec, listOpWhereFlow); err != nil { diff --git a/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go b/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go index a319980c8..1a1899fc9 100644 --- a/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go +++ b/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go @@ -215,7 +215,7 @@ func TestSpliceLoopBody1693_CreateOrModify(t *testing.T) { {"LB1693Swap", " log info node 'X' 'middle';\n set $S = $S + 'a';\n commit $It;\n", "", []string{"'middle'", "set $S", "commit $It"}, 1, 1}, {"LB1693ReplaceByTwo", " set $S = $S + 'a';\n log info node 'X' 'one';\n log info node 'X' 'two';\n commit $It;\n", - "spliced: 1 replaced", []string{"set $S", "'one'", "'two'", "commit $It"}, 1, 2}, + "spliced: 1 replaced with new element IDs", []string{"set $S", "'one'", "'two'", "commit $It"}, 1, 2}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { diff --git a/mdl/roundtrip/flow_commit_events_rerun_test.go b/mdl/roundtrip/flow_commit_events_rerun_test.go index 20ea9eb6c..1dbc59840 100644 --- a/mdl/roundtrip/flow_commit_events_rerun_test.go +++ b/mdl/roundtrip/flow_commit_events_rerun_test.go @@ -64,8 +64,8 @@ func TestSpliceRerun_CommitWithEventsIsTheStoredCommit(t *testing.T) { if err := h.exec(edited); err != nil { t.Fatalf("change: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { - t.Errorf("change: want one statement replaced:\n%s", h.out.String()) + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { + t.Errorf("change: want one statement edited in place:\n%s", h.out.String()) } if len(before.diff(h.snapshot())) == 0 { t.Fatal("change: the real change wrote nothing") diff --git a/mdl/roundtrip/flow_idempotent_shapes_test.go b/mdl/roundtrip/flow_idempotent_shapes_test.go index da0f7e27d..0b98011bc 100644 --- a/mdl/roundtrip/flow_idempotent_shapes_test.go +++ b/mdl/roundtrip/flow_idempotent_shapes_test.go @@ -191,7 +191,7 @@ end; want string }{ // The final return: the stored else of the described form. - {"a changed final return", []edit{{" return $N;\nend;", " return $N + $Base;\nend;"}}, "spliced: 1 replaced"}, + {"a changed final return", []edit{{" return $N;\nend;", " return $N + $Base;\nend;"}}, "spliced: 1 edited in place"}, // An activity inserted between the guard and the final return. {"an activity after the guard", []edit{{" return $N;\nend;", " declare $M Integer = $N * 2;\n return $M;\nend;"}}, "spliced: 1 inserted"}, // Two changes straddling the guard's return: one each, never one @@ -199,7 +199,7 @@ end; {"changes on both sides of the guard's return", []edit{ {"declare $Base Integer = 1;", "declare $Base Integer = 2;"}, {"declare $Neg Integer = 0 - $N;", "declare $Neg Integer = 1 - $N;"}, - }, "spliced: 2 replaced"}, + }, "spliced: 2 edited in place"}, } { t.Run(c.name, func(t *testing.T) { h.restore() diff --git a/mdl/roundtrip/flow_modify_end_test.go b/mdl/roundtrip/flow_modify_end_test.go index e21931902..205013eea 100644 --- a/mdl/roundtrip/flow_modify_end_test.go +++ b/mdl/roundtrip/flow_modify_end_test.go @@ -100,7 +100,7 @@ func TestFlowModify_ReturnValueIsSetInPlace(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } after := h.flowUnit(t, "ConvertUUIDToURL") @@ -140,7 +140,7 @@ func TestFlowModify_InsertBeforeChangedReturn(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 inserted, 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 inserted, 1 edited in place)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } requireKept(t, before, h.flowUnit(t, "ConvertUUIDToURL"), "") diff --git a/mdl/roundtrip/flow_modify_notes_test.go b/mdl/roundtrip/flow_modify_notes_test.go index 629499f6f..465963fff 100644 --- a/mdl/roundtrip/flow_modify_notes_test.go +++ b/mdl/roundtrip/flow_modify_notes_test.go @@ -197,8 +197,8 @@ end; t.Fatalf("exec 1: %v\n%s", err, h.out.String()) } for _, want := range []string{ - "Modified microflow: MyFirstModule.Rerun_Rollback (spliced: 1 replaced)", - "Modified nanoflow: MyFirstModule.Rerun_RollbackNf (spliced: 1 replaced)", + "Modified microflow: MyFirstModule.Rerun_Rollback (spliced: 1 edited in place)", + "Modified nanoflow: MyFirstModule.Rerun_RollbackNf (spliced: 1 edited in place)", } { // The control is in the same line: the change itself is written. if !strings.Contains(h.out.String(), want) { diff --git a/mdl/roundtrip/flow_modify_test.go b/mdl/roundtrip/flow_modify_test.go index 2b5e75556..5025faf8c 100644 --- a/mdl/roundtrip/flow_modify_test.go +++ b/mdl/roundtrip/flow_modify_test.go @@ -86,8 +86,8 @@ func TestFlowModify_MDL1ReadsStoredStringsAsStored(t *testing.T) { if bytes.Equal(h.flowUnit(t, "VAL_Feedback"), before) { t.Fatal("mdl 1: a string that states another value than the stored one wrote nothing") } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { - t.Errorf("mdl 1: want the one statement replaced by a splice, got:\n%s", h.out.String()) + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { + t.Errorf("mdl 1: want the one statement edited in place by a splice, got:\n%s", h.out.String()) } if again := h.mustDescribeMdl0(t, valFeedback); !strings.Contains(again, "characters\\\\n';") { t.Errorf("mdl 1: want the backslash stored as written:\n%s", again) @@ -199,7 +199,7 @@ func TestFlowModify_DropAndReplace(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec edited definition: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced, 1 dropped)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place, 1 dropped)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } after := h.flowUnit(t, "SUB_Feedback_Sanitize") @@ -231,7 +231,7 @@ func TestFlowModify_BranchEditIsSpliced(t *testing.T) { if err := h.exec(edited); err != nil { t.Fatalf("exec edited definition: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } after := h.flowUnit(t, "VAL_Feedback") @@ -334,7 +334,7 @@ end;` if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("a value edit in the loop body under mdl 1: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("a value edit in the loop body: want a splice, got:\n%s", h.out.String()) } described = h.mustDescribeMdl0(t, target) @@ -371,7 +371,7 @@ end;` if err := h.exec("mdl 1;\n" + after); err != nil { t.Fatalf("control under mdl 1: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("control: want a splice, got:\n%s", h.out.String()) } } diff --git a/mdl/roundtrip/flow_respelling_rerun_test.go b/mdl/roundtrip/flow_respelling_rerun_test.go index 677b60d7c..19ce7fd37 100644 --- a/mdl/roundtrip/flow_respelling_rerun_test.go +++ b/mdl/roundtrip/flow_respelling_rerun_test.go @@ -188,7 +188,9 @@ func respelledTwice(t *testing.T, h *harness, label, header, flow, setup, script } // Control: a real change outside any loop body is written, and - // moves nothing it does not change. + // moves nothing it does not change. The changed statement is the + // same kind of activity, so it is edited in place and keeps its $ID + // too (ako/mxcli#1761). edited := strings.Replace(script, from, to, 1) if edited == script { t.Fatalf("%s: the script has no %q", label, from) @@ -198,8 +200,8 @@ func respelledTwice(t *testing.T, h *harness, label, header, flow, setup, script if err := h.exec(header + edited); err != nil { t.Fatalf("%s: control: %v\n%s", label, err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { - t.Errorf("%s: control: want one statement replaced:\n%s", label, h.out.String()) + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { + t.Errorf("%s: control: want one statement edited in place:\n%s", label, h.out.String()) } if len(before.diff(h.snapshot())) == 0 { t.Fatalf("%s: control: the real change wrote nothing", label) @@ -209,19 +211,19 @@ func respelledTwice(t *testing.T, h *harness, label, header, flow, setup, script for id, p := range stored { q, ok := after[id] if !ok { - continue // the replaced activity + continue } kept++ if q != p { t.Errorf("%s: control: the unchanged %s moved from %s to %s", label, id, p, q) } } - if kept != len(stored)-1 || len(after) != len(stored) { - t.Errorf("%s: control: %d of %d stored objects kept, %d after; want all but the replaced one", + if kept != len(stored) || len(after) != len(stored) { + t.Errorf("%s: control: %d of %d stored objects kept, %d after; want every one, the edited activity too", label, kept, len(stored), len(after)) } if !samePositions(stored, after) { - t.Errorf("%s: control: the replacement is not drawn where the replaced activity was", label) + t.Errorf("%s: control: the edited activity is not drawn where it was", label) } // The twice-exec rule for the change. second := h.snapshot() diff --git a/mdl/roundtrip/flow_splice_guard_test.go b/mdl/roundtrip/flow_splice_guard_test.go index 3dce8eb4c..4ff29ee47 100644 --- a/mdl/roundtrip/flow_splice_guard_test.go +++ b/mdl/roundtrip/flow_splice_guard_test.go @@ -249,7 +249,7 @@ func TestSpliceRerun_StudioProDecisionConditionSetInPlace(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("want the condition set, got:\n%s", h.out.String()) } after := h.flowUnit(t, "ChangePassword") @@ -356,7 +356,7 @@ func TestSpliceRerun_ReplaceEndingInGuard(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 replaced with new element IDs)") { t.Errorf("want one replace, got:\n%s", h.out.String()) } after := h.flowUnit(t, "ShowPasswordForm") @@ -397,7 +397,7 @@ end; if err := h.exec("mdl 1;\n" + guarded); err != nil { t.Fatalf("guard: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 replaced with new element IDs)") { t.Errorf("want one replace, got:\n%s", h.out.String()) } requireDecisionCases(t, h.flowUnit(t, "Grow_Last")) diff --git a/mdl/roundtrip/flow_splice_handler_return_test.go b/mdl/roundtrip/flow_splice_handler_return_test.go index 8a090b57e..95672d719 100644 --- a/mdl/roundtrip/flow_splice_handler_return_test.go +++ b/mdl/roundtrip/flow_splice_handler_return_test.go @@ -54,7 +54,7 @@ begin end error; log info node 'EH' 'done'; end; -`, "(spliced: 1 replaced)", 2}, +`, "(spliced: 1 replaced with new element IDs)", 2}, {"insert, mdl 1, a returned value", "mdl 1;\n", `create or modify microflow MyFirstModule.EH_Insert1 () returns Boolean as $R begin diff --git a/mdl/roundtrip/flow_splice_member_qualified_test.go b/mdl/roundtrip/flow_splice_member_qualified_test.go index bae027fb4..9325bfda9 100644 --- a/mdl/roundtrip/flow_splice_member_qualified_test.go +++ b/mdl/roundtrip/flow_splice_member_qualified_test.go @@ -95,8 +95,8 @@ func TestFlowModify_SplicedMembersAreQualified(t *testing.T) { case err != nil: t.Errorf("%s: exec: %v", label, err) continue - case !strings.Contains(h.out.String(), "(spliced: 1 replaced)"): - t.Errorf("%s: want one statement replaced by a splice, got:\n%s", label, h.out.String()) + case !strings.Contains(h.out.String(), "(spliced: 1 edited in place)"): + t.Errorf("%s: want one statement edited in place by a splice, got:\n%s", label, h.out.String()) } spliced := flowMembers(t, h.flowUnit(t, "T885_Flow")) for _, m := range spliced { diff --git a/mdl/roundtrip/flow_splice_parity_test.go b/mdl/roundtrip/flow_splice_parity_test.go index 10674fb7a..cb0ffa381 100644 --- a/mdl/roundtrip/flow_splice_parity_test.go +++ b/mdl/roundtrip/flow_splice_parity_test.go @@ -137,8 +137,8 @@ func TestPedAppSpliceParity_AuthoredSpellings(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec an edited constraint: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "Modified microflow: MyFirstModule.SpliceParity (spliced: 1 replaced)") { - t.Errorf("an edited constraint must be spliced in as one replaced retrieve:\n%s", h.out.String()) + if !strings.Contains(h.out.String(), "Modified microflow: MyFirstModule.SpliceParity (spliced: 1 edited in place)") { + t.Errorf("an edited constraint must be spliced in as one retrieve edited in place:\n%s", h.out.String()) } if len(created.diff(h.snapshot())) == 0 { t.Error("an edited constraint wrote nothing") diff --git a/mdl/roundtrip/rerun_idempotent_test.go b/mdl/roundtrip/rerun_idempotent_test.go index 5f8d921bf..7246c3705 100644 --- a/mdl/roundtrip/rerun_idempotent_test.go +++ b/mdl/roundtrip/rerun_idempotent_test.go @@ -441,7 +441,7 @@ end; } written := len(before.diff(h.snapshot())) switch { - case run == 1 && (written == 0 || !strings.Contains(h.out.String(), "(spliced: 1 replaced)")): + case run == 1 && (written == 0 || !strings.Contains(h.out.String(), "(spliced: 1 edited in place)")): t.Errorf("run 1: want the edited set spliced in, %d unit(s) written:\n%s", written, h.out.String()) case run == 2 && written != 0: t.Errorf("run 2 of the same script wrote %d unit(s):\n%s", written, h.out.String()) From 76efadf3e9209edfdb856cfc9408af37ad308edc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 23:16:35 +0000 Subject: [PATCH 22/61] docs(microflows): say what a same-kind replace keeps The alter syntax help, the alter-microflow reference page and the microflow and nanoflow skills now say that an activity replaced by one of the same kind is the stored activity edited, keeping its $ID and those of its action and the parts that pair up, and how exec reports it (`edited in place` against `replaced with new element IDs`) (ako/mxcli#1761). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../skills/mendix/write-microflows/SKILL.md | 2 +- .../write-microflows/reference/pitfalls.md | 5 +-- .../skills/mendix/write-nanoflows/SKILL.md | 34 ++++++++++--------- cmd/mxcli/syntax/features_microflow.go | 6 ++++ .../reference/microflow/alter-microflow.md | 21 ++++++++++++ 5 files changed, 49 insertions(+), 19 deletions(-) diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 8e3fccb1e..9cc999c03 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -69,7 +69,7 @@ Choose the mode by who owns the microflow ([choose-edit-mode](../choose-edit-mod (targets from `describe microflow X with handles`). `create or modify` of `describe` output patches too: unchanged writes nothing; a statement (even one whose handler returns), guard clause, `return` value, `if` condition, header clause, parameter (added/retyped; removed only if unused) or stated `@position`/`@start` change is - patched in place (a move keeps the node's flows), as are plain activities added, removed, reordered or changed in a loop body ([an insert needing room past a stated `@position`](reference/pitfalls.md#13-inserting-where-the-script-states-every-position)). A redrawn + patched in place (a move keeps the node's flows; an activity changed into one of the same kind — an expression, a `where`, a member value, an argument — keeps its `$ID` and its action's, reported `edited in place`), as are plain activities added, removed, reordered or changed in a loop body ([an insert needing room past a stated `@position`](reference/pitfalls.md#13-inserting-where-the-script-states-every-position)). A redrawn `@anchor`/`@curve`, a decision/loop/`return` added or changed inside a loop body, error handler or other `return` added/taken away rebuilds under mdl 0 (`MDL-V1-REBUILD`: IDs renumbered, merges and curves lost) and is refused under `mdl 1;`. **To put a decision or loop in a loop body, `alter … replace body of loop $X in $Xs with begin … end;`** — the loop keeps its `$ID` ([pitfalls](reference/pitfalls.md#11-changing-something-inside-a-loop-body)); a decision `alter … insert` put in is taken out by a re-run, `drop` or `replace` ([pitfalls](reference/pitfalls.md#15-taking-out-a-decision-another-script-put-in)). **To rebuild deliberately, `drop` and `create` in ONE script** — the grants carry only within it ([pitfalls](reference/pitfalls.md#drop--create-is-still-a-new-document)). diff --git a/.claude/skills/mendix/write-microflows/reference/pitfalls.md b/.claude/skills/mendix/write-microflows/reference/pitfalls.md index 3d812a9a3..8f1cd8342 100644 --- a/.claude/skills/mendix/write-microflows/reference/pitfalls.md +++ b/.claude/skills/mendix/write-microflows/reference/pitfalls.md @@ -305,8 +305,9 @@ inserted, dropped, moved (dropped and inserted again) or replaced by one or more others (ako/mxcli#1693), and one replaced by one of the same kind in place (ako/mxcli#1106). The new activities are laid out among the body's nodes; when the body no longer fits, the loop box grows on its far side and what is drawn past the -loop moves along. Only the changed activities get new `$ID`s — the loop, the rest of -its body and everything outside it keep theirs. The loop's iterator is in scope, and +loop moves along. Only the new activities get new `$ID`s — one replaced by the same +kind of activity is the stored one edited and keeps its `$ID` and its action's +(ako/mxcli#1761); the loop, the rest of its body and everything outside it keep theirs. The loop's iterator is in scope, and a nested loop's body is edited the same way. `create or modify` does this for a body whose statements change; `alter` for an activity of the body addressed by its handle. diff --git a/.claude/skills/mendix/write-nanoflows/SKILL.md b/.claude/skills/mendix/write-nanoflows/SKILL.md index 6740be6af..e035dd784 100644 --- a/.claude/skills/mendix/write-nanoflows/SKILL.md +++ b/.claude/skills/mendix/write-nanoflows/SKILL.md @@ -11,21 +11,21 @@ description: "Nanoflow syntax in MDL — shared with microflows, but client-side | lines | section | |---|---| | 35-44 | When to Use This Skill | -| 45-81 | Changing an Existing Nanoflow | -| 82-100 | When to Use a Nanoflow vs a Microflow | -| 101-115 | Key Differences from Microflows | -| 116-139 | Nanoflow Structure | -| 140-148 | Naming Convention | -| 149-217 | Supported Activities | -| 218-484 | Real-World Patterns | -| 485-555 | NanoflowCommons JavaScript Actions Reference | -| 556-594 | Disallowed Activities | -| 595-598 | Return Type Restrictions | -| 599-643 | Error Handling in Nanoflows | -| 644-651 | Security (GRANT/REVOKE) | -| 652-664 | Management Commands | -| 665-675 | Common Mistakes | -| 676-690 | Validation Checklist | +| 45-83 | Changing an Existing Nanoflow | +| 84-102 | When to Use a Nanoflow vs a Microflow | +| 103-117 | Key Differences from Microflows | +| 118-141 | Nanoflow Structure | +| 142-150 | Naming Convention | +| 151-219 | Supported Activities | +| 220-486 | Real-World Patterns | +| 487-557 | NanoflowCommons JavaScript Actions Reference | +| 558-596 | Disallowed Activities | +| 597-600 | Return Type Restrictions | +| 601-645 | Error Handling in Nanoflows | +| 646-653 | Security (GRANT/REVOKE) | +| 654-666 | Management Commands | +| 667-677 | Common Mistakes | +| 678-692 | Validation Checklist | # Mendix Nanoflow Skill @@ -52,7 +52,9 @@ Choose the mode by who owns the nanoflow ([choose-edit-mode](../choose-edit-mode (targets by output variable, caption or statement pattern). `create or modify` of `describe` output also works as a patch: an unchanged definition writes nothing, and an inserted, replaced or dropped statement (at the top level or in an `if` branch) is spliced in, - leaving every other node, merge and curve as stored — a guard clause (`if … then return; + leaving every other node, merge and curve as stored — a statement changed into one of + the same kind (an expression, a `where`, a member value, an argument) edits the stored + activity, which keeps its `$ID` and its action's (reported `edited in place`); a guard clause (`if … then return; end if;`) too, its return a new end event, and so is the `return` ending a new activity's error handler; a changed `return` value or `if` condition is set on the stored end event or decision, and a body without a trailing `return` means the stored end. The diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 1f12a16bf..9cfd46949 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -624,6 +624,12 @@ func init() { "-- stored flow before any operation runs; an ambiguous or unknown target is an error.\n" + "-- Only the new activities, the rewired flows and the objects moved to make room change;\n" + "-- every other element keeps its $ID, position and curve.\n" + + "-- REPLACE by one activity of the same kind (the same action: a filter's expression, a\n" + + "-- retrieve's where, a member's value, a call's argument changed) edits the stored\n" + + "-- activity: it keeps its $ID and those of its action and the parts that pair up — a\n" + + "-- member by its attribute or association, an argument by its parameter (ako/mxcli#1761).\n" + + "-- CREATE OR MODIFY reports it as `edited in place`; anything else replaced gets new\n" + + "-- element IDs, reported as `replaced with new element IDs`.\n" + "-- Inside a loop body, plain activities are inserted, replaced and dropped the same way,\n" + "-- the loop box growing when the body no longer fits (ako/mxcli#1693).\n" + "-- REPLACE BODY OF keeps the loop itself (its $ID, position, iterator and list)\n" + diff --git a/docs-site/src/reference/microflow/alter-microflow.md b/docs-site/src/reference/microflow/alter-microflow.md index 69536e27f..1f3685207 100644 --- a/docs-site/src/reference/microflow/alter-microflow.md +++ b/docs-site/src/reference/microflow/alter-microflow.md @@ -32,6 +32,20 @@ against the flow as stored before any operation runs. A fragment is written exactly as the same statements are in `CREATE MICROFLOW`, between `BEGIN` and `END`. +### Editing an activity + +`REPLACE` by one activity of the same kind — the same action, with a +filter's or find's expression, a retrieve's `WHERE`, a member's value or a +call's argument changed — edits the stored activity rather than putting a new +one in its place. The activity keeps its element ID, its place and its size, +and so do its action and every element under it that pairs up: a member of a +change or create by its attribute or association, an argument by its +parameter, anything else by its property and type. A member taken out goes +with its ID, which no other member inherits; a part of another type (a `FIND` +where a `FILTER` was) is new. Studio Pro, which compares and merges by element +ID, then shows the activity as changed rather than one removed and another +added. An activity replaced by another kind of activity, or by several, is new. + ### Inside a loop body Inside a loop body, plain activities are inserted, replaced and dropped the @@ -84,6 +98,13 @@ makes it — for a decision inside a loop body, `REPLACE BODY OF` that loop. Re-running the script that created a flow after another script spliced a decision into it takes that decision out again, when it has the shape above. +A statement changed where it stands edits its stored activity, as above, and +so does each of several adjacent statements changed in their own places. The +report says which: `Modified microflow: M.F (spliced: 1 edited in place)` for +a node that kept its element ID — an activity of the same kind, a decision's +condition, a return's value — and `replaced with new element IDs` for one +that did not. + ## Examples Insert, replace and drop at top level: From 35741a2c69eb816b26407d8e0d66e6a06c4fd6ad Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:38:44 +0000 Subject: [PATCH 23/61] fix(canon): leave the end-of-path marker out of a parallel split below Mendix 10.21 Symptom (ako/mxcli#1163): on Mendix 9.x and 10.0-10.20 a workflow with a `parallel split` made the project unloadable. exec reported "Created workflow", then mx check died with "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity". Cause: the builder and the ALTER mutator end every path with that marker, because from 10.21 the engine runs a path only up to it. The type itself was introduced in 10.21.0 (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared a written $Type with the project's version. Older shape, measured: Studio Pro 9.4.0 (the Marketplace's "Example project workflow extension" 2.0.0) stores a split whose ParallelSplitOutcome is {$ID,$Type,Flow} and whose paths hold their activities and nothing more. Lowering produces exactly that key set. Fix: canon.LowerToProjectShape drops the marker from every Workflows$Flow below 10.21, for every writer at once (create, create or modify, and a path alter workflow inserts). Measured on fresh apps, the bug-test exec'd then mx check: - 9.24.30.51827: before TypeCacheUnknownTypeException, after 0 errors; mx convert 9.24.30 changes no key; describe -> exec "Unchanged". - 10.12.19.81612: before TypeCacheUnknownTypeException, after 0 errors; mx convert 10.12.19 changes no key. - 10.24.24 and 11.13.0: 4 markers, 0 errors, key sets identical to the unfixed build. Control: stubbing the lowering fails TestNormalize_ParallelSplitPathsBelowMendix10_21 on 9.24 and 10.20 with the marker still in each path. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-28-parallel-split-marker-below-10-21.json | 1 + CHANGELOG.md | 3 +- .../src/appendixes/version-compatibility.md | 1 + .../1163-parallel-split-below-10-21.mdl | 64 +++++++++++ .../bug-tests/1163-workflow-on-mendix9.mdl | 6 +- modelsdk/canon/versionshape.go | 49 ++++++++- .../mpr/writer_version_shape_1163_test.go | 102 ++++++++++++++++++ sdk/workflows/workflow.go | 5 + 8 files changed, 225 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json create mode 100644 mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json new file mode 100644 index 000000000..56b20bd0e --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.20 a workflow with a `parallel split` made the project unloadable: exec reported \"Created workflow\", then mx check (9.24.30, 10.12.19) died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity`", "cause": "The builder (workflows.EndParallelSplitPath) and the ALTER mutator (InsertPath) end every path with Workflows$EndOfParallelSplitPathActivity, which 10.21+ needs to run the path. The TYPE is introduced in 10.21.0 (mendixmodelsdk: EndOfParallelSplitPathActivity.versionInfo). The generated version data holds property floors only, so neither the floor nor any gate ever compared a written $Type with the project's version", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: dropEndOfPathMarkers on every Workflows$Flow below 10.21, endOfParallelSplitPathSince)", "insight": "**For a type newer than the project, find the older shape in a Marketplace SAMPLE, not a module**: `mxcli marketplace search workflow` lists Type=Sample apps whose .mpk holds a whole Studio Pro-authored .mpr; \"Example project workflow extension\" 2.0.0 (117680, Studio Pro 9.4.0) has a parallel split whose ParallelSplitOutcome is `{$ID,$Type,Flow}` and whose paths hold their activities and nothing else — exactly what the lowering produces. Workflow Commons, Employee Onboarding (9.1.1) and Multilevel Dynamic Workflow (9.24.3) hold workflows but no split. **The convert-up oracle does not reconstruct writer-only markers**: `mx convert -p` 10.24.24 of the lowered 9.24 project added 16 PersistentIds, BoundaryEvents and OnWorkflowEvent but no end-of-path marker, and 10.24's mx check is clean on it — so it cannot tell you whether 10.21+ needs the marker (the 11.14 runtime measurement in workflows.EndParallelSplitPath does), and a Studio Pro 9 split upgraded that way is unmarked too. Lower strictly below 10.21. **Lower, don't gate**: the older shape is exact and measured, unlike the published OData service, which is a different document. **Audit what else the writers emit against the SDK's type-level `introduced`** (a text extractor over gen/*.js — never execute it): the same run found user tasks written as Workflows$SingleUserTaskActivity (10.12; Studio Pro 9 stores Workflows$UserTask — 9.24 TypeCacheUnknownTypeException) and user targeting under UserTargeting (11.2, declared on the abstract UserTaskActivity so the floor misses it): on 10.12.19 and 10.24.24 a `targeting xpath […]` user task is CE1859 \"XPath constraint is required\", and mx convert 10.12 rewrites it to UserSource with the constraint EMPTIED. Not fixed here. Measured: bug-test on fresh 9.24.30 and 10.12.19 apps → before TypeCacheUnknownTypeException, after 0 errors, mx convert changes no key; describe → exec on 9.24 \"Unchanged workflow\"; 10.24.24 and 11.13.0 keep 4 markers, 0 errors, key sets identical before/after. Control: stubbing the lowering fails TestNormalize_ParallelSplitPathsBelowMendix10_21 with the marker on 9.24 and 10.20. mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 9734de71e..007339543 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). -- **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. +- **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. +- **A workflow with a `parallel split` no longer makes a Mendix 9 or 10.0–10.20 project unloadable** (ako/mxcli#1163) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity". mxcli ends every path with that marker, because from 10.21 the workflow engine runs a path only up to it, but the type itself arrived in 10.21. Before 10.21 Studio Pro stores a path as its activities and nothing more (measured on a Studio Pro 9.4 app). Below 10.21 the marker is no longer written, for `create`, `create or modify` and a path `alter workflow` inserts. Measured on 9.24.30 and 10.12.19: 0 errors. On 9.24, `describe` → `exec` reports the workflow unchanged. 10.21 and later are unchanged. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index e6b4cd22a..78e924507 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -186,6 +186,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | +| The end of a parallel split path (`Workflows$EndOfParallelSplitPathActivity`, a type introduced in 10.21) | 10.21 | Not stored: a path holds its activities and nothing more (Studio Pro 9.4) | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | | A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | diff --git a/mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl b/mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl new file mode 100644 index 000000000..fc0f86d91 --- /dev/null +++ b/mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl @@ -0,0 +1,64 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.20 a workflow with a +-- `parallel split` made the project unloadable. `exec` reported success, then +-- mx check (and Studio Pro) could not load the project at all: +-- +-- TypeCacheUnknownTypeException: The type cache does not contain a type +-- with qualified name Workflows$EndOfParallelSplitPathActivity. +-- +-- mxcli ends every parallel split path with that marker, because from 10.21 +-- the workflow engine runs a path only up to it. The type itself arrived in +-- 10.21 (mendixmodelsdk), and the property floor checks property versions, +-- never a type's own. Before 10.21 Studio Pro stores a path as its activities +-- and nothing else (Studio Pro 9.4.0: the Marketplace's "Example project +-- workflow extension" 2.0.0, a split whose two paths hold a user task each), +-- so below 10.21 the marker is no longer written. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`), exec then `mxcli docker +-- check`: +-- 9.24.30.51827 before: TypeCacheUnknownTypeException; after: 0 errors. +-- `mx convert -p` 9.24.30 changes no key; describe → exec +-- prints "Unchanged workflow". +-- 10.12.19.81612 before: TypeCacheUnknownTypeException; after: 0 errors; +-- `mx convert -p` 10.12.19 changes no key. +-- 10.24.24, 11.13.0 every path still ends in the marker, 0 errors, the +-- same key sets as before the fix. +-- +-- The ALTER below inserts a path through the mutator (InsertPath), outside +-- CREATE's encoder; the lowering runs on every unit write, so it covers both. + +create module Probe1163Ps; + +create persistent entity Probe1163Ps.Request ( + Title: String(100), + Amount: Integer +); + +create microflow Probe1163Ps.ACT_Left ($WorkflowContext: Probe1163Ps.Request) +begin + log info node 'Probe1163Ps' 'left'; +end; + +create microflow Probe1163Ps.ACT_Right ($WorkflowContext: Probe1163Ps.Request) +begin + log info node 'Probe1163Ps' 'right'; +end; + +create workflow Probe1163Ps.WF_Split + parameter $WorkflowContext: Probe1163Ps.Request + display 'Split' +begin + call microflow Probe1163Ps.ACT_Left; + parallel split + path 1 { call microflow Probe1163Ps.ACT_Left; } + path 2 { call microflow Probe1163Ps.ACT_Right; } + path 3 { } + ; + call microflow Probe1163Ps.ACT_Right; +end workflow; + +alter workflow Probe1163Ps.WF_Split { + insert into 'Parallel split' { path { + call microflow Probe1163Ps.ACT_Right; + } } +}; diff --git a/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl index ecea673ad..6d2529ef7 100644 --- a/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl +++ b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl @@ -16,9 +16,9 @@ mdl 1; -- then `mxcli docker check` reports the blank app's error count, and `mx -- convert -p` strips no PersistentId. -- --- Not covered: a `parallel split`. Its paths end in a --- Workflows$EndOfParallelSplitPathActivity, a type Mendix introduced in 10.21, --- so below 10.21 the project does not load (TypeCacheUnknownTypeException). +-- A `parallel split` is in 1163-parallel-split-below-10-21.mdl: its paths end +-- in a Workflows$EndOfParallelSplitPathActivity, a type Mendix introduced in +-- 10.21, which below 10.21 is no longer written. create module Probe1163Wf; diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 0ee201248..7ac90cb7b 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -53,6 +53,12 @@ import ( // WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows // exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's // own was refused; the others slipped past the floor. +// - Workflows$EndOfParallelSplitPathActivity, a TYPE introduced in 10.21.0, +// which ends every parallel split path mxcli writes (ako/mxcli#1163). +// Before, a path is its activities and nothing more (Studio Pro 9.4.0, the +// Marketplace's "Example project workflow extension" 2.0.0), and a project +// holding the marker does not load (TypeCacheUnknownTypeException). The +// floor checks property versions, never a type's own. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -72,9 +78,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), + unmarkedPaths: pv.Compare(endOfParallelSplitPathSince) < 0, } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant && !l.variableArgs { + !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths { return contents, nil } var doc bson.D @@ -95,10 +102,17 @@ type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool - flatConstant, variableArgs bool + flatConstant, variableArgs, unmarkedPaths bool changed bool } +// endOfParallelSplitPathSince is the version that introduced the type +// Workflows$EndOfParallelSplitPathActivity (mendixmodelsdk 4.117.0: +// EndOfParallelSplitPathActivity.versionInfo introduced "10.21.0"). It is +// written down here because the generated version data holds property +// versions only, never a type's own. +var endOfParallelSplitPathSince = version.Version{Major: 10, Minor: 21} + var placeholderWidgets = map[string]bool{"Forms$TextBox": true, "Forms$TextArea": true, "Forms$DatePicker": true} // walk returns v with every element below it in the project's shape. @@ -140,6 +154,9 @@ func (l *shapeLowering) walk(v any) (any, error) { if l.noPersistentID && strings.HasPrefix(typ, "Workflows$") { x = l.dropPersistentID(x) } + if l.unmarkedPaths && typ == "Workflows$Flow" { + l.dropEndOfPathMarkers(x) + } if l.variableArgs { if err = l.lowerFlowArguments(x); err != nil { return nil, err @@ -378,6 +395,34 @@ func (l *shapeLowering) dropPersistentID(d bson.D) bson.D { return append(append(bson.D{}, d[:i]...), d[i+1:]...) } +// dropEndOfPathMarkers removes the Workflows$EndOfParallelSplitPathActivity +// from a flow's activities below 10.21, where the type does not exist. The +// list keeps its leading type marker, so a path that held only the end-of-path +// activity stores `Activities: [3]`, as Studio Pro 9 stores an empty flow. +// Nothing is refused: +// the marker is the writer's, never something a script states, and a path +// before 10.21 is its activities alone (Studio Pro 9.4.0's parallel split). +func (l *shapeLowering) dropEndOfPathMarkers(flow bson.D) { + i := shapeIndex(flow, "Activities") + if i < 0 { + return + } + list, _ := flow[i].Value.(bson.A) + out := make(bson.A, 0, len(list)) + for _, el := range list { + if d, ok := el.(bson.D); ok { + if t, _ := shapeString(d, "$Type"); t == "Workflows$EndOfParallelSplitPathActivity" { + continue + } + } + out = append(out, el) + } + if len(out) != len(list) { + flow[i].Value = out + l.changed = true + } +} + // listItems returns the elements of a stored list, without its type marker. func listItems(v any) bson.A { a, _ := v.(bson.A) diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index 3d33cfcb7..1542401b5 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -164,6 +164,108 @@ func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { } } +// endOfPath is the marker mxcli ends a parallel split path with +// (workflows.EndParallelSplitPath). +func endOfPath() bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$EndOfParallelSplitPathActivity"}, + {Key: "Annotation", Value: nil}, {Key: "Caption", Value: "End of parallel split path"}, + {Key: "Name", Value: "EndOfParallelSplitPath"}, {Key: "PersistentId", Value: pid()}} +} + +// workflowWithParallelSplit is a workflow as mxcli builds it: a parallel split +// whose first path calls a microflow and whose second is empty, each ending in +// the end-of-path marker, followed by another microflow call. +func workflowWithParallelSplit() bson.D { + call := func(name string) bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$CallMicroflowTask"}, + {Key: "BoundaryEvents", Value: bson.A{int32(2)}}, {Key: "Caption", Value: name}, + {Key: "Microflow", Value: "M." + name}, {Key: "Name", Value: name}, {Key: "PersistentId", Value: pid()}} + } + path := func(acts ...bson.D) bson.D { + list := bson.A{int32(3)} + for _, a := range acts { + list = append(list, a) + } + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$ParallelSplitOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: list}}}, + {Key: "PersistentId", Value: pid()}} + } + split := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$ParallelSplitActivity"}, + {Key: "Annotation", Value: nil}, {Key: "Caption", Value: "Parallel split"}, {Key: "Name", Value: "split"}, + {Key: "Outcomes", Value: bson.A{int32(3), path(call("Left"), endOfPath()), path(endOfPath())}}, + {Key: "PersistentId", Value: pid()}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), split, call("After")}}}}, + {Key: "Name", Value: "WF"}, + {Key: "PersistentId", Value: pid()}, + {Key: "Title", Value: "WF"}} +} + +// pathActivityTypes returns, per parallel split path, the $Type of each +// activity its flow holds. +func pathActivityTypes(t *testing.T, d bson.D) [][]string { + t.Helper() + outcomes, _ := get(find(d, "Workflows$ParallelSplitActivity"), "Outcomes") + var out [][]string + for _, o := range listItems(outcomes) { + flow, _ := get(o.(bson.D), "Flow") + acts, _ := get(flow.(bson.D), "Activities") + if a, _ := acts.(bson.A); len(a) == 0 || a[0] != int32(3) { + t.Errorf("a path's Activities is %v, want a list with marker 3", acts) + } + types := []string{} + for _, a := range listItems(acts) { + typ, _ := get(a.(bson.D), "$Type") + types = append(types, typ.(string)) + } + out = append(out, types) + } + return out +} + +// Workflows$EndOfParallelSplitPathActivity is introduced in 10.21.0 +// (mendixmodelsdk: the type's own versionInfo). mxcli ends every parallel split +// path with one, because from 10.21 the engine runs a path only up to it — and +// below 10.21 the type does not exist: mx check 9.24.30 does not load the +// project at all ("TypeCacheUnknownTypeException: The type cache does not +// contain a type with qualified name Workflows$EndOfParallelSplitPathActivity"). +// The property floor holds property versions only, so it never saw the type. +// Studio Pro before 10.21 stores a path as its activities and nothing else: +// the Marketplace's "Example project workflow extension" 2.0.0 (Studio Pro +// 9.4.0) has a split whose two paths hold one user task each, no marker, and +// an empty flow stores `Activities: [3]`. So below 10.21 the marker is not +// written; from 10.21 it is, and must be. +func TestNormalize_ParallelSplitPathsBelowMendix10_21(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want string + }{ + {pv(9, 24, 30), "Workflows$CallMicroflowTask|"}, + {pv(10, 20, 0), "Workflows$CallMicroflowTask|"}, + {pv(10, 21, 0), "Workflows$CallMicroflowTask,Workflows$EndOfParallelSplitPathActivity|Workflows$EndOfParallelSplitPathActivity"}, + {pv(11, 13, 0), "Workflows$CallMicroflowTask,Workflows$EndOfParallelSplitPathActivity|Workflows$EndOfParallelSplitPathActivity"}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithParallelSplit()) + if err != nil { + t.Errorf("%d.%d: a parallel split is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + var paths []string + for _, p := range pathActivityTypes(t, d) { + paths = append(paths, strings.Join(p, ",")) + } + if got := strings.Join(paths, "|"); got != tt.want { + t.Errorf("%d.%d: paths hold %s, want %s", tt.v.MajorVersion, tt.v.MinorVersion, got, tt.want) + } + // The rest of the workflow is untouched: the call after the split stays. + acts, _ := get(find(d, "Workflows$Flow"), "Activities") + if n := len(listItems(acts)); n != 2 { + t.Errorf("%d.%d: the main flow holds %d activities, want 2", tt.v.MajorVersion, tt.v.MinorVersion, n) + } + } +} + // settingsWithConstantValue is the project settings unit as `alter settings // constant … in configuration` writes a new override: the value nested in a // Settings$SharedValue (or a Settings$PrivateValue, which holds none). flat is diff --git a/sdk/workflows/workflow.go b/sdk/workflows/workflow.go index dc57904ce..0f6fa8a82 100644 --- a/sdk/workflows/workflow.go +++ b/sdk/workflows/workflow.go @@ -395,6 +395,11 @@ func (a *EndOfParallelSplitPathActivity) ActivityType() string { return "EndOfPa // the path (a jump, or an end-of-workflow), where a marker would be unreachable. // A nil flow — an empty path — gets a flow holding only the marker, which is how // an empty path is stored. +// +// The marker's type is introduced in Mendix 10.21; before it a path is its +// activities alone, and a project holding the marker does not load. The marker +// is added on every version here and dropped below 10.21 on the way to storage +// (canon.LowerToProjectShape), so the builders need not know the version. func EndParallelSplitPath(flow *Flow, newID func() model.ID) *Flow { if flow == nil { flow = &Flow{} From 761fab054f5589e292352f7037bd64d485b53f42 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:56:43 +0000 Subject: [PATCH 24/61] fix(modelsdk): give WidgetReturnType.assignableTo and GroupBox.headerMode their floors Symptom (ako/mxcli#1163): on Mendix 9.0-9.19 every pluggable widget's schema lost the Type of its return types and kept an AssignableTo that version does not have (49 return types across the embedded templates). A return type reaching the floor there was refused with "this Mendix 9.18.0 project cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)". Cause: the vendored generator gave a property the row of the next property in the SDK's versionInfo and left that one out, as it did AggregateFunction. mendixmodelsdk 4.117.0 has WidgetReturnType `type: {}`, `assignableTo: {introduced 9.20.0}`, and pages.GroupBox `caption: {}`, `headerMode: {introduced 8.10.0}`. widgetTypeForVersion drops undeclared keys from the widget schema by the same data, so it dropped the right key. Evidence: Studio Pro stores {EntityProperty, IsList, Type} on 208 of 208 return types in a 9.1.1 app and 326 of 326 in a 9.4.0 app (Marketplace samples), and adds AssignableTo on 57 of 57 in a 9.24.3 one. The group box row changes nothing mxcli writes: the caption is stored as CaptionTemplate, which the table never keyed, and 8.10 is below every supported version. Workflows$CallMicroflowTask.boundaryEvents was already 10.14.0, as in the SDK, and stays. Measured on 10.12.19 with that floor removed: mxcli writes BoundaryEvents [2] on every call and mx convert strips it, so the floor's drop-empty is right. A test now pins it; removing the row fails it. Fix: correct both rows by hand and regenerate modelsdk/version. Only those two rows change in the copy. Measured: the bug-test on fresh 10.24.24 and 11.13.0 apps checks at 0 errors before and after, with identical key sets. 9.24.30 stores Studio Pro 9.24.3's return type keys both times (its mx check stops on a separate pre-existing NullReferenceException in WidgetValue.IsDefault). Control: the new tests fail on the old rows (49 wrong key sets at 9.18; the Type refusal). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...type-and-group-box-rows-misattributed.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-widget-return-type-below-9-20.mdl | 61 +++++++++ .../modelsdk/widget_template_version_test.go | 69 ++++++++++ modelsdk/gen/customwidgets/version.go | 8 +- modelsdk/gen/pages/version.go | 12 +- .../mpr/writer_version_shape_1163_test.go | 128 ++++++++++++++++++ modelsdk/version/metamodel_versions_gen.go | 20 ++- 9 files changed, 294 insertions(+), 8 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json create mode 100644 mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json new file mode 100644 index 000000000..46deef833 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.0–9.19 every pluggable widget's schema lost the Type of its return types and kept an AssignableTo the version does not have (49 return types across the embedded templates); a CustomWidgets$WidgetReturnType reaching the floor there was refused: `this Mendix 9.18.0 project cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)`. Forms$GroupBox had caption at 8.10 and no row for headerMode", "cause": "The vendored generator (modelsdk/gen/*/version.go) gave a property the row of the NEXT property in the SDK's versionInfo and left that one out, as it did Microflows$AggregateAction.aggregateFunction: mendixmodelsdk 4.117.0 has WidgetReturnType `type: {}`, `assignableTo: {introduced 9.20.0}` and pages.GroupBox `caption: {}`, `headerMode: {introduced 8.10.0}`. canon.PropertyDeclared reads the same table, and widgetTypeForVersion uses it to drop undeclared keys from the embedded 11.6 widget schema, so the wrong row dropped the right key", "file": "`modelsdk/gen/customwidgets/version.go`, `modelsdk/gen/pages/version.go` (fixed by hand), `modelsdk/version/metamodel_versions_gen.go` (go generate ./modelsdk/version)", "insight": "**Compare the whole table with the SDK, then read each hit's neighbour**: a text extractor over the SDK's gen/*.js StructureVersionInfo (map Pages$ to Forms$ for storage names; never execute the SDK) found 50 differences; most are SDK data newer than the vendored generator (11.11–11.16 rows it lacks), and five are this misattribution, always 'property gets its neighbour's version, neighbour missing': AggregateAction.aggregateFunction, WidgetReturnType.type, GroupBox.caption, and on Deleted LayoutGridRow.columns (← class, 8.0 — the row propertyfloor.go cites as why generated deletions are not trusted) and DynamicImageViewer.showAsThumbnail (← onClickBehavior, 7.18), plus a never-looked-up SDK-named duplicate row Microflows$AggregateListAction. The Deleted ones and the duplicate change nothing the floor does and were left. Workflows$CallMicroflowTask.boundaryEvents was NOT a mismatch (10.14.0 on both sides): the earlier list's fourth row was a false positive. **Check which BSON key a row is looked up by before predicting its effect**: the floor keys by the lower-cased BSON key, and a group box's caption is stored as CaptionTemplate, so the wrong caption row never applied to anything; headerMode's 8.10 is below every version mxcli writes — the GroupBox fix corrects the data and changes no write. **Studio Pro-authored bytes settle a row faster than a platform run when no mxbuild of that version is cached**: Marketplace samples carry whole apps — Employee Onboarding (9.1.1) stores {EntityProperty,IsList,Type} on 208 of 208 return types, Example project workflow extension 2.0.0 (9.4.0) on 326 of 326, Multilevel Dynamic Workflow (9.24.3) adds AssignableTo on 57 of 57. For boundaryEvents the oracle was a floor-off build on 10.12.19: mxcli then writes `BoundaryEvents: [2]` on every call, mx check passes, mx convert 10.12.19 strips it — the existing floor's drop-empty is what keeps the write clean, and a stated boundary event is refused. Tests: TestTemplateWidgetReturnTypeKeysAcrossMendix9_20 (fails on the old row with 49 wrong key sets at 9.18), TestNormalize_WidgetReturnTypeAcrossMendix9_20, TestPropertyDeclared_RowsTheGeneratorMisattributed, TestNormalize_CallMicroflowBoundaryEventsBelowMendix10_14 (control: removing the row fails it). Seen in passing, not fixed: on 9.24.30 a page with a slider or a data grid 2, written by mxcli, stops mx check (and its update-widgets step) with `NullReferenceException at WidgetValue.IsDefault() (WidgetValue.cs:879)`, with or without these fixes; 10.24.24 and 11.13.0 check the same pages at 0 errors. Bug-test: mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 007339543..ea06bf20b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. - **A workflow with a `parallel split` no longer makes a Mendix 9 or 10.0–10.20 project unloadable** (ako/mxcli#1163) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity". mxcli ends every path with that marker, because from 10.21 the workflow engine runs a path only up to it, but the type itself arrived in 10.21. Before 10.21 Studio Pro stores a path as its activities and nothing more (measured on a Studio Pro 9.4 app). Below 10.21 the marker is no longer written, for `create`, `create or modify` and a path `alter workflow` inserts. Measured on 9.24.30 and 10.12.19: 0 errors. On 9.24, `describe` → `exec` reports the workflow unchanged. 10.21 and later are unchanged. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. +- **Pluggable widgets keep the `Type` of their return types on Mendix 9.0–9.19** (ako/mxcli#1163) — the version data mxcli vendors from the Model SDK had a return type's `Type` introduced in 9.20. That version belongs to `AssignableTo`, which the data left out. A widget schema written to a 9.0–9.19 project therefore lost the `Type` of every return type and kept an `AssignableTo` that version does not have. A return type that reached the property floor was refused with "cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)". Studio Pro 9.1 and 9.4 store `Type` on every return type and no `AssignableTo`. The same mistake gave a group box's caption the 8.10 version of its header mode. That row changed nothing mxcli writes, because the caption is stored under another name and mxcli supports no version that old. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 78e924507..8350ab77b 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -207,6 +207,8 @@ A published OData service is not an older shape of the same document. Before 10. Pluggable widget PropertyTypes change between Mendix versions. A widget template extracted from 11.6 may have more or fewer properties than the widget installed in a 10.24 project. The [widget augmentation system](../internals/widget-templates.md) handles this by reconciling templates against the project's `.mpk` files. +The schema itself also gained keys over time. A key the project's version does not declare is left out of the schema mxcli writes, by the same version data the property floor uses. For example, a return type's `AssignableTo` arrived in 9.20, so a 9.0–9.19 schema stores `EntityProperty`, `IsList` and `Type`, as Studio Pro 9.1 and 9.4 do ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). + ## Widget Template Versions Embedded widget templates are extracted from Mendix 11.6 and cover: diff --git a/mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl b/mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl new file mode 100644 index 000000000..42451968f --- /dev/null +++ b/mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl @@ -0,0 +1,61 @@ +mdl 1; +-- ako/mxcli#1163: two more rows of the version data mxcli vendors from the +-- Model SDK gave a property the version of the property after it, and left +-- that one out — the same generator mistake as AggregateFunction: +-- +-- CustomWidgets$WidgetReturnType Type 9.20.0 (belongs to AssignableTo) +-- Forms$GroupBox Caption 8.10.0 (belongs to HeaderMode) +-- +-- mendixmodelsdk 4.117.0: WidgetReturnType `type: {}`, `assignableTo: +-- {introduced: "9.20.0"}`; pages.GroupBox `caption: {}`, `headerMode: +-- {introduced: "8.10.0"}`. +-- +-- What it changed: a pluggable widget's schema is written with only the keys +-- the project's version declares, by the same data. On 9.0–9.19 every return +-- type in it lost its Type and kept AssignableTo (49 return types across the +-- embedded templates; the slider below has 3). Studio Pro stores +-- {EntityProperty, IsList, Type} on 9.1.1 (208 of 208, the Marketplace's +-- Employee Onboarding app) and 9.4.0 (326 of 326, "Example project workflow +-- extension" 2.0.0), and {AssignableTo, EntityProperty, IsList, Type} on +-- 9.24.3 (57 of 57, Multilevel Dynamic Workflow). The group box row changed +-- nothing mxcli writes: the caption is stored as CaptionTemplate, which the +-- table never keyed, and 8.10 is below every version mxcli supports. +-- +-- Measured, exec then `mxcli docker check` on fresh apps, before and after: +-- 10.24.24 and 11.13.0 report 0 errors and store the same key sets. 9.24.30 +-- stores Studio Pro 9.24.3's return type key set both times, but its mx check +-- stops with a NullReferenceException in WidgetValue.IsDefault on this page, +-- and on a data grid 2 page. That happens with the build before either +-- ako/mxcli#1163 fix as well, so it is a separate defect. Not run on a +-- 9.0–9.19 platform: those mxbuilds are Mono builds (mx.exe), and no Mono +-- runtime is installed. + +create module Probe1163Rt; + +create enumeration Probe1163Rt.Country ( + NL 'Netherlands', + BE 'Belgium' +); + +create persistent entity Probe1163Rt.Customer ( + Name: String(100), + Country: enumeration(Probe1163Rt.Country), + Score: Integer +); + +create page Probe1163Rt.Customer_Edit +( + params: ( $Customer: Probe1163Rt.Customer ), + title: 'Customer', + layout: Atlas_Core.PopupLayout +) +{ + dataview dvCustomer (datasource: $Customer) { + groupbox gbCustomer (caption: 'Customer', HeaderMode: H3, Collapsible: YesExpanded) { + textbox txtName (label: 'Name', attribute: Name) + pluggablewidget 'com.mendix.widget.custom.slider.Slider' sldScore ( + valueAttribute: Score + ) + } + } +}; diff --git a/mdl/backend/modelsdk/widget_template_version_test.go b/mdl/backend/modelsdk/widget_template_version_test.go index ea7789649..7edeea39d 100644 --- a/mdl/backend/modelsdk/widget_template_version_test.go +++ b/mdl/backend/modelsdk/widget_template_version_test.go @@ -184,6 +184,75 @@ func TestTemplateWidgetValueTypeKeysAt10_15(t *testing.T) { } } +// typeKeySets returns the key set of every element of typeName in an encoded +// document. +func typeKeySets(t *testing.T, b []byte, typeName string) [][]string { + t.Helper() + var doc bsonv2.D + if err := bsonv2.Unmarshal(b, &doc); err != nil { + t.Fatal(err) + } + var out [][]string + var walk func(v any) + walk = func(v any) { + switch x := v.(type) { + case bsonv2.D: + var keys []string + match := false + for _, e := range x { + keys = append(keys, e.Key) + if e.Key == "$Type" && e.Value == typeName { + match = true + } + walk(e.Value) + } + if match { + sort.Strings(keys) + out = append(out, keys) + } + case bsonv2.A: + for _, e := range x { + walk(e) + } + } + } + walk(doc) + return out +} + +// CustomWidgets$WidgetReturnType: Type on every version, AssignableTo from +// 9.20.0 (mendixmodelsdk 4.117.0). The vendored generator gave Type the row of +// AssignableTo and left AssignableTo out, so on 9.0–9.19 the template's return +// types lost the Type every one of them holds and kept an AssignableTo the +// version does not have (ako/mxcli#1163). Studio Pro stores {EntityProperty, +// IsList, Type} on 9.1.1 (208 of 208, the Marketplace's Employee Onboarding +// app) and 9.4.0 (326 of 326, "Example project workflow extension" 2.0.0), and +// adds AssignableTo by 9.24.3 (57 of 57, Multilevel Dynamic Workflow 1.0.0). +func TestTemplateWidgetReturnTypeKeysAcrossMendix9_20(t *testing.T) { + for _, tt := range []struct { + pv *types.ProjectVersion + want string + }{ + {v(9, 18, 0), "$ID,$Type,EntityProperty,IsList,Type"}, + {v(9, 20, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, + {v(9, 24, 30), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, + {v(11, 13, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, // control + } { + seen := 0 + for _, id := range mwidgets.ListAvailableTemplates() { + for _, got := range typeKeySets(t, encodeTemplateWidget(t, id, tt.pv), "CustomWidgets$WidgetReturnType") { + seen++ + if strings.Join(got, ",") != tt.want { + t.Errorf("%d.%d %s: WidgetReturnType keys %v, want %s", tt.pv.MajorVersion, tt.pv.MinorVersion, id, got, tt.want) + } + } + } + if seen == 0 { + t.Fatalf("%d.%d: no template holds a WidgetReturnType", tt.pv.MajorVersion, tt.pv.MinorVersion) + } + } +} + func containsKey(s []string, k string) bool { for _, x := range s { if x == k { diff --git a/modelsdk/gen/customwidgets/version.go b/modelsdk/gen/customwidgets/version.go index de213ddd9..101778f48 100644 --- a/modelsdk/gen/customwidgets/version.go +++ b/modelsdk/gen/customwidgets/version.go @@ -48,7 +48,13 @@ var VersionInfos = map[string]version.TypeVersionInfo{ }, "CustomWidgets$WidgetReturnType": { Properties: map[string]version.PropertyVersionInfo{ - "type": {Introduced: "9.20.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave type + // the row of assignableTo, which it then left out. mendixmodelsdk + // 4.117.0 has type with no version and assignableTo introduced + // 9.20.0; Studio Pro 9.1.1 and 9.4.0 store Type and no AssignableTo + // on every return type, Studio Pro 9.24.3 both. + "assignableTo": {Introduced: "9.20.0"}, + "type": {}, }, }, "CustomWidgets$WidgetValue": { diff --git a/modelsdk/gen/pages/version.go b/modelsdk/gen/pages/version.go index 506d0cc1d..eab4e5d9c 100644 --- a/modelsdk/gen/pages/version.go +++ b/modelsdk/gen/pages/version.go @@ -406,9 +406,15 @@ var VersionInfos = map[string]version.TypeVersionInfo{ }, "Forms$GroupBox": { Properties: map[string]version.PropertyVersionInfo{ - "caption": {Introduced: "8.10.0"}, - "widget": {Deleted: "7.15.0"}, - "widgets": {Introduced: "7.15.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave caption + // the row of headerMode, which it then left out. mendixmodelsdk + // 4.117.0 has caption with no version and headerMode introduced + // 8.10.0; Studio Pro 9.1.1, 9.4.0 and 9.24.3 store both (the caption + // as CaptionTemplate). + "caption": {}, + "headerMode": {Introduced: "8.10.0"}, + "widget": {Deleted: "7.15.0"}, + "widgets": {Introduced: "7.15.0"}, }, }, "Forms$Header": { diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index 1542401b5..db8be67f5 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -8,7 +8,9 @@ import ( "go.mongodb.org/mongo-driver/v2/bson" + "github.com/mendixlabs/mxcli/modelsdk/canon" "github.com/mendixlabs/mxcli/modelsdk/mpr/version" + mversion "github.com/mendixlabs/mxcli/modelsdk/version" ) // ako/mxcli#1163: what the 9.24 doctype run found mxcli writing that Mendix 9 @@ -82,6 +84,132 @@ func TestNormalize_ReduceFoldIsRefusedBelowMendix10(t *testing.T) { } } +// The vendored generator gave two more properties the row of their type's next +// property and left that one out, as it did AggregateFunction: mendixmodelsdk +// 4.117.0 has CustomWidgets$WidgetReturnType `type: {}` and `assignableTo: +// {introduced: "9.20.0"}`, and Forms$GroupBox (pages.GroupBox) `caption: {}` +// and `headerMode: {introduced: "8.10.0"}`. The table had type 9.20.0 and +// caption 8.10.0, and no row for assignableTo or headerMode. +// Workflows$CallMicroflowTask.boundaryEvents is 10.14.0 in both, and stays. +func TestPropertyDeclared_RowsTheGeneratorMisattributed(t *testing.T) { + for _, tt := range []struct { + typ, key string + v *version.ProjectVersion + want bool + }{ + // Studio Pro 9.1.1 and 9.4.0 store Type on every return type; 9.24.3 + // adds AssignableTo. + {"CustomWidgets$WidgetReturnType", "Type", pv(9, 0, 0), true}, + {"CustomWidgets$WidgetReturnType", "Type", pv(9, 18, 0), true}, + {"CustomWidgets$WidgetReturnType", "AssignableTo", pv(9, 19, 0), false}, + {"CustomWidgets$WidgetReturnType", "AssignableTo", pv(9, 20, 0), true}, + // The caption is stored as CaptionTemplate, which the table never + // keyed: the wrong row applied to nothing. HeaderMode is 8.10, below + // every version mxcli writes. + {"Forms$GroupBox", "CaptionTemplate", pv(8, 0, 0), true}, + {"Forms$GroupBox", "HeaderMode", pv(8, 9, 0), false}, + {"Forms$GroupBox", "HeaderMode", pv(8, 10, 0), true}, + {"Forms$GroupBox", "HeaderMode", pv(9, 0, 0), true}, + // Unchanged: right in the table already. + {"Workflows$CallMicroflowTask", "BoundaryEvents", pv(10, 13, 0), false}, + {"Workflows$CallMicroflowTask", "BoundaryEvents", pv(10, 14, 0), true}, + } { + v := mversion.Version{Major: tt.v.MajorVersion, Minor: tt.v.MinorVersion, Patch: tt.v.PatchVersion} + if got := canon.PropertyDeclared(tt.typ, tt.key, v); got != tt.want { + t.Errorf("%s.%s declared on %s = %v, want %v", tt.typ, tt.key, v, got, tt.want) + } + } +} + +// pageWithReturnType is a page holding one pluggable widget whose Type has a +// return type as the template carries it: Type set, AssignableTo empty. +func pageWithReturnType() bson.D { + rt := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetReturnType"}, + {Key: "AssignableTo", Value: ""}, {Key: "EntityProperty", Value: ""}, {Key: "IsList", Value: false}, {Key: "Type", Value: "String"}} + vt := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetValueType"}, {Key: "ReturnType", Value: rt}} + widget := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, {Key: "Name", Value: "w"}, + {Key: "Type", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$CustomWidgetType"}, + {Key: "ObjectType", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetObjectType"}, + {Key: "PropertyTypes", Value: bson.A{int32(2), bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetPropertyType"}, + {Key: "ValueType", Value: vt}}}}}}}}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$Page"}, {Key: "Name", Value: "P"}, + {Key: "Widgets", Value: bson.A{int32(2), widget}}} +} + +// Below 9.20 a return type's Type was refused ("this Mendix 9.18.0 project +// cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)") and its +// AssignableTo passed. Now Type is kept and the empty AssignableTo dropped. +func TestNormalize_WidgetReturnTypeAcrossMendix9_20(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want string + }{ + {pv(9, 18, 0), "$ID,$Type,EntityProperty,IsList,Type"}, + {pv(9, 20, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, + {pv(11, 13, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, // control + } { + d, err := normalizeShape(t, tt.v, pageWithReturnType()) + if err != nil { + t.Errorf("%d.%d: a widget with a return type is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + if got := strings.Join(keysOf(find(d, "CustomWidgets$WidgetReturnType")), ","); got != tt.want { + t.Errorf("%d.%d: return type keys %s, want %s", tt.v.MajorVersion, tt.v.MinorVersion, got, tt.want) + } + } +} + +// boundaryEvent is a timer boundary event as mxcli writes one (10.20's type). +func boundaryEvent() bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$InterruptingTimerBoundaryEvent"}, + {Key: "Caption", Value: "Late"}, {Key: "FirstExecutionTime", Value: "addDays([%CurrentDateTime%], 1)"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: bson.A{int32(3)}}}}} +} + +// workflowWithCallBoundaryEvents is a workflow holding one microflow call with +// the given BoundaryEvents list. +func workflowWithCallBoundaryEvents(events bson.A) bson.D { + call := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$CallMicroflowTask"}, + {Key: "BoundaryEvents", Value: events}, {Key: "Caption", Value: "Validate"}, + {Key: "Microflow", Value: "M.Validate"}, {Key: "Name", Value: "call"}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), call}}}}, + {Key: "Name", Value: "WF"}, {Key: "Title", Value: "WF"}} +} + +// Workflows$CallMicroflowTask.BoundaryEvents is 10.14.0 (mendixmodelsdk, and +// the table). The writer gives every microflow call an empty list, so below +// 10.14 the empty list is dropped, not refused: Studio Pro 9.24.3 stores no +// BoundaryEvents on a call (Multilevel Dynamic Workflow 1.0.0), and with the +// row's floor removed mxcli writes `BoundaryEvents: [2]` on 10.12.19, which +// `mx convert` 10.12.19 strips. A boundary event the script states is +// refused there, naming the key. +func TestNormalize_CallMicroflowBoundaryEventsBelowMendix10_14(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + kept bool + }{ + {pv(9, 24, 30), false}, + {pv(10, 13, 0), false}, + {pv(10, 14, 0), true}, + {pv(11, 13, 0), true}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithCallBoundaryEvents(bson.A{int32(2)})) + if err != nil { + t.Errorf("%d.%d: a microflow call is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + if _, ok := get(find(d, "Workflows$CallMicroflowTask"), "BoundaryEvents"); ok != tt.kept { + t.Errorf("%d.%d: BoundaryEvents kept %v, want %v", tt.v.MajorVersion, tt.v.MinorVersion, ok, tt.kept) + } + } + _, err := normalizeShape(t, pv(10, 13, 0), workflowWithCallBoundaryEvents(bson.A{int32(2), boundaryEvent()})) + if err == nil || !strings.Contains(err.Error(), "Workflows$CallMicroflowTask.BoundaryEvents (Mendix 10.14.0+)") { + t.Errorf("a boundary event on a microflow call on 10.13 was not refused naming the key: %v", err) + } +} + // pid is a fresh PersistentId: a non-zero GUID, as the writer mints one (a zero // GUID is an empty value, which the floor would drop rather than refuse). func pid() bson.Binary { diff --git a/modelsdk/version/metamodel_versions_gen.go b/modelsdk/version/metamodel_versions_gen.go index 37591339c..899146f87 100644 --- a/modelsdk/version/metamodel_versions_gen.go +++ b/modelsdk/version/metamodel_versions_gen.go @@ -341,7 +341,13 @@ var metamodelVersions = []map[string]TypeVersionInfo{ }, "CustomWidgets$WidgetReturnType": { Properties: map[string]PropertyVersionInfo{ - "type": {Introduced: "9.20.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave type + // the row of assignableTo, which it then left out. mendixmodelsdk + // 4.117.0 has type with no version and assignableTo introduced + // 9.20.0; Studio Pro 9.1.1 and 9.4.0 store Type and no AssignableTo + // on every return type, Studio Pro 9.24.3 both. + "assignableTo": {Introduced: "9.20.0"}, + "type": {}, }, }, "CustomWidgets$WidgetValue": { @@ -2416,9 +2422,15 @@ var metamodelVersions = []map[string]TypeVersionInfo{ }, "Forms$GroupBox": { Properties: map[string]PropertyVersionInfo{ - "caption": {Introduced: "8.10.0"}, - "widget": {Deleted: "7.15.0"}, - "widgets": {Introduced: "7.15.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave caption + // the row of headerMode, which it then left out. mendixmodelsdk + // 4.117.0 has caption with no version and headerMode introduced + // 8.10.0; Studio Pro 9.1.1, 9.4.0 and 9.24.3 store both (the caption + // as CaptionTemplate). + "caption": {}, + "headerMode": {Introduced: "8.10.0"}, + "widget": {Deleted: "7.15.0"}, + "widgets": {Introduced: "7.15.0"}, }, }, "Forms$Header": { From 4939d516d8d8fe8d52893cf3a8eff5ba42bfd886 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 09:00:59 +0800 Subject: [PATCH 25/61] =?UTF-8?q?fix(git):=20doctor=20judges=20only=20core?= =?UTF-8?q?.autocrlf=20=E2=80=94=20the=20mendix.*=20counters=20are=20Studi?= =?UTF-8?q?o=20Pro's=20own=20(measured=20on=20the=20lab=20repo,=20S0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- cmd/mxcli/cmd_git_doctor.go | 5 +- cmd/mxcli/gitdoctor.go | 59 +++++++++++++++-------- cmd/mxcli/gitdoctor_test.go | 41 +++++++++++++++- docs-site/src/tools/outside-studio-pro.md | 2 +- 4 files changed, 82 insertions(+), 25 deletions(-) diff --git a/cmd/mxcli/cmd_git_doctor.go b/cmd/mxcli/cmd_git_doctor.go index 1e17d23f3..10d5d61b4 100644 --- a/cmd/mxcli/cmd_git_doctor.go +++ b/cmd/mxcli/cmd_git_doctor.go @@ -23,8 +23,9 @@ var gitDoctorCmd = &cobra.Command{ Checked: 1. Local git config: core.autocrlf=False (a checkout must not rewrite line - endings inside .mxunit files), mendix.commits-since-gc and - mendix.lineEndingResetDone (Studio Pro's own bookkeeping). --fix sets them. + endings inside .mxunit files; --fix rewrites any other value), and the + mendix.* keys (Studio Pro's own bookkeeping) backfilled when missing — + a value Studio Pro wrote stays. 2. The remote URL. Studio Pro can only use HTTPS; an SSH remote (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. Other non-HTTPS URLs are reported without being changed. diff --git a/cmd/mxcli/gitdoctor.go b/cmd/mxcli/gitdoctor.go index 48f045728..c0726d4b0 100644 --- a/cmd/mxcli/gitdoctor.go +++ b/cmd/mxcli/gitdoctor.go @@ -31,11 +31,23 @@ import ( var errDoctorIssuesRemain = errors.New("issues remain (re-run with --fix to repair)") -// mxConfigKeys is the local git config Studio Pro writes and expects. -var mxConfigKeys = [][2]string{ - {"core.autocrlf", "False"}, - {"mendix.commits-since-gc", "0"}, - {"mendix.lineEndingResetDone", "True"}, +// mxConfigKey is one entry of the local git config Studio Pro writes. +// fallback is what a missing key is backfilled with. presentValuePasses marks +// the keys whose value is Studio Pro's own bookkeeping — measured on a real +// checkout (the mx-git-lab lab, S0), commits-since-gc counts up between gc +// runs and lineEndingResetDone comes back in any case, so only a MISSING key +// is an issue there. core.autocrlf is judged: true (or input) makes a +// checkout rewrite line endings inside .mxunit files on Windows. +type mxConfigKey struct { + key string + fallback string + presentValuePasses bool +} + +var mxConfigKeys = []mxConfigKey{ + {"core.autocrlf", "False", false}, + {"mendix.commits-since-gc", "0", true}, + {"mendix.lineEndingResetDone", "True", true}, } var ( @@ -52,25 +64,32 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W found, fixed := 0, 0 fmt.Fprintln(out, "Git local config") - for _, kv := range mxConfigKeys { - key, want := kv[0], kv[1] + for _, ck := range mxConfigKeys { + key, want := ck.key, ck.fallback current := gitConfigLocal(root, key) - if strings.EqualFold(current, want) { - fmt.Fprintf(out, " [x] %s = %s\n", key, current) - continue - } - found++ - if current == "" { + switch { + case current == "": + found++ fmt.Fprintf(out, " [ ] %s — missing (expected %s)\n", key, want) - } else { + if fix { + if _, err := gitOut(root, "config", "--local", key, want); err != nil { + return fmt.Errorf("set %s: %w", key, err) + } + fixed++ + fmt.Fprintf(out, " set to %s\n", want) + } + case ck.presentValuePasses || strings.EqualFold(current, "false"): + fmt.Fprintf(out, " [x] %s = %s\n", key, current) + default: + found++ fmt.Fprintf(out, " [ ] %s = %s (expected %s)\n", key, current, want) - } - if fix { - if _, err := gitOut(root, "config", "--local", key, want); err != nil { - return fmt.Errorf("set %s: %w", key, err) + if fix { + if _, err := gitOut(root, "config", "--local", key, want); err != nil { + return fmt.Errorf("set %s: %w", key, err) + } + fixed++ + fmt.Fprintf(out, " set to %s\n", want) } - fixed++ - fmt.Fprintf(out, " set to %s\n", want) } } diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index c0e0393f4..fafe0a9b0 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -58,8 +58,7 @@ func TestGitDoctor_ConfigMissingReported(t *testing.T) { } } -func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { - dir := t.TempDir() +func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { dir := t.TempDir() doctorEnv(t, dir) gitIn(t, dir, "init", "-q", "-b", "main") @@ -87,6 +86,44 @@ func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { } } +// Measured on a real Studio Pro checkout (the mx-git-lab lab, S0): Studio Pro +// writes core.autocrlf=false in lowercase, counts commits-since-gc up between +// garbage collections (2 after two commits) and writes lineendingresetdone in +// lowercase. A healthy Studio Pro checkout must pass untouched. +func TestGitDoctor_StudioProValuesPass(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "--local", "core.autocrlf", "false") + gitIn(t, dir, "config", "--local", "mendix.commits-since-gc", "2") + gitIn(t, dir, "config", "--local", "mendix.lineendingresetdone", "true") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", false, &out); err != nil { + t.Fatalf("a healthy Studio Pro checkout must pass: %v\n%s", err, out.String()) + } + if strings.Contains(out.String(), "mendix.commits-since-gc — ") { + t.Errorf("the gc counter must not be judged:\n%s", out.String()) + } +} + +// core.autocrlf=true is the one config value that damages a Mendix checkout: +// a checkout rewrites line endings inside .mxunit files on Windows. +func TestGitDoctor_AutoCRLFTrueIsFixed(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "--local", "core.autocrlf", "true") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", true, &out); err != nil { + t.Fatalf("--fix must repair autocrlf=true: %v\n%s", err, out.String()) + } + if got := configLocal(t, dir, "core.autocrlf"); !strings.EqualFold(got, "false") { + t.Errorf("core.autocrlf = %q, want false", got) + } +} + // --- doctor: remote URL scheme ---------------------------------------------- func TestGitDoctor_SSHRemoteRewrittenByFix(t *testing.T) { diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 9cf80d53a..6b71c0b19 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -224,7 +224,7 @@ mxcli git doctor --fix --version 10.21.0 | Check | `--fix` | |-------|---------| -| local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` | sets them | +| local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files; any other value is rewritten), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` — the `mendix.*` values are Studio Pro's own bookkeeping and are only backfilled when missing | sets/backs them | | the remote URL: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged | | the remote's `refs/notes/mx_metadata` | reported only — push the notes ref together with the branch | | each commit on the checked-out branch and its mx_metadata note: missing notes make Studio Pro show `(unknown)`; a malformed note it cannot read at all | computes real notes from the model (`-p`), falling back to a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); when no version is known the newest valid note in history supplies it | From 6ddd5e5ed709dea826e3b84ddaf6e32fee96af99 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 09:11:50 +0800 Subject: [PATCH 26/61] fix(git): the staged-files summary lists what the commit holds, not the junk just unstaged (lab finding F4) --- cmd/mxcli/cmd_git_commit.go | 20 +++++++++++++------- cmd/mxcli/gitdoctor_test.go | 27 ++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 8 deletions(-) diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index cbdb52cbe..bf9fadcb7 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -138,16 +138,22 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error fmt.Fprintln(out, "nothing to commit, working tree clean") return nil } - fmt.Fprintf(out, "Staged %d file(s):\n", len(staged)-len(junk)) - kept := staged - if len(kept) > 10 { - kept = kept[:10] + kept := make([]string, 0, len(staged)-len(junk)) + for _, f := range staged { + if !isJunkPath(f) { + kept = append(kept, f) + } } - for _, f := range kept { + fmt.Fprintf(out, "Staged %d file(s):\n", len(kept)) + shown := kept + if len(shown) > 10 { + shown = shown[:10] + } + for _, f := range shown { fmt.Fprintf(out, " %s\n", f) } - if len(staged)-len(junk) > 10 { - fmt.Fprintf(out, " … and %d more\n", len(staged)-len(junk)-10) + if len(kept) > 10 { + fmt.Fprintf(out, " … and %d more\n", len(kept)-10) } if dryRun { diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index fafe0a9b0..2d43138c9 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -577,9 +577,34 @@ func TestGitCommit_DryRunLeavesHashDriftUntouched(t *testing.T) { } } -func TestGitCommit_DryRunWithPushRefused(t *testing.T) { +// The staged-files summary must list what the commit will hold, not the junk +// that was just unstaged (measured on the mx-git-lab lab: the list opened with +// 10 .mxcli/ paths while the commit held 2 model files). +func TestGitCommit_StagedListOmitsJunk(t *testing.T) { dir, mprPath := doctorNoteRepo(t) doctorEnv(t, dir) + units := baseUnits(t) + units[tPage] = tUnit{tFolder, unitDoc(t, "Forms$Page", "Home", bson.E{Key: "Title", Value: "List"})} + writeProject(t, dir, "11.13.0", units) + if err := os.WriteFile(filepath.Join(dir, "project-settings.user.json"), []byte("{}"), 0o644); err != nil { + t.Fatal(err) + } + + var out bytes.Buffer + if err := commitRun(mprPath, "clean list", false, false, &out); err != nil { + t.Fatalf("commitRun: %v\n%s", err, out.String()) + } + section := out.String()[strings.Index(out.String(), "Staged"):] + if strings.Contains(section, "project-settings.user.json") { + t.Errorf("the staged list names junk that was left out:\n%s", section) + } + if !strings.Contains(section, "App.mpr") { + t.Errorf("the staged list must name the model files:\n%s", section) + } +} + +func TestGitCommit_DryRunWithPushRefused(t *testing.T) { dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) var out bytes.Buffer if err := commitRun(mprPath, "both", true, true, &out); err == nil { From 4e961b23b2a81c3be7d902ba892cbeafa18d61b6 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 10:43:10 +0800 Subject: [PATCH 27/61] =?UTF-8?q?feat(git):=20`git=20merge`=20reviews=20co?= =?UTF-8?q?nflicts=20before=20they=20land=20=E2=80=94=20report=20both=20si?= =?UTF-8?q?des,=20then=20--ours/--theirs=20or=20Studio=20Pro=20(lab=20S5b)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude/skills/mendix/teamserver-git/SKILL.md | 23 +- CHANGELOG.md | 1 + cmd/mxcli/cmd_git_commit.go | 11 +- cmd/mxcli/cmd_git_merge.go | 416 ++++++++++++++++++ cmd/mxcli/gitdoctor_test.go | 6 +- cmd/mxcli/gitmerge_test.go | 305 +++++++++++++ docs-site/src/tools/outside-studio-pro.md | 38 ++ 7 files changed, 780 insertions(+), 20 deletions(-) create mode 100644 cmd/mxcli/cmd_git_merge.go create mode 100644 cmd/mxcli/gitmerge_test.go diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index 3b2af2afd..ea51485aa 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -14,9 +14,9 @@ description: "Commit, branch and push a Mendix project in git the way Studio Pro | 42-55 | Branches: create them on the server | | 56-97 | Commit from the agent | | 98-112 | Push the branch and the notes together, and soon | -| 113-125 | Never rewrite or merge model history with git | -| 126-136 | Reading what Studio Pro recorded | -| 137-145 | Checklist | +| 113-132 | Never rewrite or merge model history with git | +| 133-143 | Reading what Studio Pro recorded | +| 144-152 | Checklist | # Working with Git on a Mendix Team Server Project @@ -112,11 +112,18 @@ git push origin refs/notes/mx_metadata ## Never rewrite or merge model history with git -- **No `git merge`, `rebase` or `cherry-pick` of model changes.** The `.mpr` is - a SQLite file; git cannot merge it, and a textual merge of `mprcontents/` - leaves an index that does not match the files. Studio Pro has a model-aware - merge: **Version Control › Merge Changes Here** (*Merge feature branch* or - *Cherry Pick*). Studio Pro's history on Team Server is linear for that reason. +- **No plain `git merge`, `rebase` or `cherry-pick` of model changes.** The + `.mpr` is a SQLite file; git cannot merge it (every two-sided model change + is a binary conflict), and a textual merge of `mprcontents/` leaves an + index that does not match the files. Model-aware merging exists in two + forms: **`mxcli git merge `** from the command line (it hands the + conflict to `mx git-merge`, Studio Pro's own engine, and commits the + resolved model as a normal merge commit with the note), and **Version + Control › Merge Changes Here** in Studio Pro (*Merge feature branch* or + *Cherry Pick*) — which is also where the conflicts mx cannot auto-resolve + go. A conflict `mxcli git merge` cannot resolve aborts the merge cleanly. + Studio Pro's history on Team Server is linear for a reason: do not "fix" + a conflicted model merge by hand-editing files. - **No `--amend` or force-push of a pushed commit.** Before pushing, an amend is fine: the `notes.rewriteRef` Studio Pro puts in `.git/config` carries the note. Re-run `mxcli git note --write` anyway; it is cheap. diff --git a/CHANGELOG.md b/CHANGELOG.md index ea7ab00ce..5a9751d20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the human then re-runs with `--ours` or `--theirs` to take one side of every conflicted model file (deterministic, mx not consulted again), or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index bf9fadcb7..697c6ef84 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -179,16 +179,7 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error return fmt.Errorf("committed %s, but attaching the note failed (%v) — run 'mxcli git note -p %s --write' to attach it", shortCommit(head), err, mprPath) } - if push { - if branch == "" { - return errors.New("detached HEAD; cannot push — check out a branch and push manually") - } - if _, err := gitOutIn(root, nil, "push", "origin", branch, mxMetadataRef); err != nil { - return fmt.Errorf("push %s and %s: %w", branch, mxMetadataRef, err) - } - fmt.Fprintf(out, "Pushed %s and %s to origin.\n", branch, mxMetadataRef) - } - return nil + return pushBranchAndNotes(out, root, branch, push) } // nulFields splits git's -z output (NUL-separated, no quoting) into paths. diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go new file mode 100644 index 000000000..ecdcebc74 --- /dev/null +++ b/cmd/mxcli/cmd_git_merge.go @@ -0,0 +1,416 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strings" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/mendixlabs/mxcli/modelsdk/mpr" + "github.com/spf13/cobra" +) + +// cmd_git_merge.go: `mxcli git merge ` — merge a branch the way Studio +// Pro does. `git merge` runs first; when it stops on the model files (the +// .mpr is SQLite and every .mxunit is BSON, so any two-sided model change is +// a binary conflict), the conflict is handed to `mx git-merge` — the same +// model-aware engine Studio Pro's own merge uses — with the three revisions +// checked out into temporary worktrees. What mx resolves is committed as a +// normal two-parent merge commit with the mx_metadata note; what it reports +// as ManualResolutionRequired aborts the merge with the conflict list and a +// pointer to Studio Pro, leaving nothing behind. + +var gitMergeCmd = &cobra.Command{ + Use: "merge ", + Short: "Merge a branch with model-aware conflict resolution (mx git-merge)", + Long: `Merge into the current branch, resolving model conflicts the way +Studio Pro does. + +'git merge ' runs first. A merge git resolves itself (disjoint files, a +fast-forward) is committed and annotated like 'mxcli git commit' does. When +git stops on the model files — the .mpr and the .mxunit units are binary, so +two sides changing the model always conflict — the three revisions are +checked out into temporary worktrees and 'mx git-merge BASE MINE THEIRS' +(Studio Pro's own model-aware merge, resolved for the project's Mendix +version) decides: + + - everything mx auto-resolves is staged and committed as the merge commit, + with the mx_metadata note attached (this keeps both sides — the usual + best outcome); + - a conflict mx reports as ManualResolutionRequired is never merged + silently: the merge is aborted, and both sides' changes are printed for + review. Then decide: + --ours take our side of every conflicted model file + --theirs take their side + (deterministic; mx is not consulted again), or resolve for real in + Studio Pro (Version Control › Merge Changes Here), which stays the tool + for mixed resolutions; + - a conflict in non-model files (text sources) is left in place for manual + resolution, like plain git. + +The merge is refused while Studio Pro has the project open (.mpr.lock). +MPR v2 projects only. The mx binary is resolved for the project's Mendix +version (PATH, the mxcli cache, or the Studio Pro installation).`, + Example: ` mxcli git merge feature/orders -p app.mpr + mxcli git merge origin/main -p app.mpr --push + mxcli git merge feature/orders -p app.mpr --theirs # after reviewing the report`, + Args: cobra.ExactArgs(1), + RunE: runGitMerge, + SilenceUsage: true, + SilenceErrors: true, +} + +func init() { + gitMergeCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") + _ = gitMergeCmd.MarkFlagRequired("project") + gitMergeCmd.Flags().StringP("message", "m", "", "merge commit message (default: git's)") + gitMergeCmd.Flags().Bool("ours", false, "on a model conflict mx cannot resolve, take our side of every conflicted model file") + gitMergeCmd.Flags().Bool("theirs", false, "on a model conflict mx cannot resolve, take their side of every conflicted model file") + gitMergeCmd.Flags().Bool("push", false, "push the branch and the notes ref to their remote afterwards") + gitCmd.AddCommand(gitMergeCmd) +} + +func runGitMerge(cmd *cobra.Command, args []string) error { + ref := args[0] + message, _ := cmd.Flags().GetString("message") + ours, _ := cmd.Flags().GetBool("ours") + theirs, _ := cmd.Flags().GetBool("theirs") + push, _ := cmd.Flags().GetBool("push") + project, _ := cmd.Flags().GetString("project") + strategy := "" + switch { + case ours && theirs: + return errors.New("--ours and --theirs cannot be combined") + case ours: + strategy = "ours" + case theirs: + strategy = "theirs" + } + return mergeRun(project, ref, message, strategy, push, realMxGitMerge, cmd.OutOrStdout()) +} + +// mxMergeRunner runs `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT +// THEIRS_COMMIT` and returns mx's exit code and output. +type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, error) + +// mergeRun is the command without its wiring. runMx is injected for tests. +// mergeRun is the command without its wiring. runMx is injected for tests. +// +// Conflict handling is a three-step flow a human stays in charge of: +// +// 1. `git merge` runs; a merge it resolves itself is annotated and done. +// 2. A model conflict goes to `mx git-merge` first — what it auto-resolves +// keeps both sides, which is almost always the best outcome — and is +// committed with the note. +// 3. What mx reports as ManualResolutionRequired is never merged silently: +// with no strategy the merge is aborted and both sides' changes are +// printed for review; `--ours`/`--theirs` then take one side wholesale +// (mx is not consulted — the human has decided). +func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRunner, out io.Writer) error { + switch strategy { + case "", "ours", "theirs": + default: + return fmt.Errorf("unknown strategy %q (use --ours or --theirs)", strategy) + } + abs, err := filepath.Abs(mprPath) + if err != nil { + return err + } + if !mpr.AllowWritesWhileStudioProOpen && os.Getenv(mpr.AllowStudioProOpenEnv) != "1" { + if lock, mod := mpr.StudioProLockFile(abs); lock != "" { + return &mpr.StudioProOpenError{MprPath: abs, LockPath: lock, LockTime: mod} + } + } + repo, err := openGitNoteRepo(mprPath) + if err != nil { + return err + } + defer repo.Close() + root := repo.Root + + branch := strings.TrimSpace(gitOutOr(root, "symbolic-ref", "--quiet", "--short", "HEAD")) + if branch == "" { + return errors.New("detached HEAD: check out a branch before merging") + } + + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + if strings.Contains(mergeOut, "Already up to date") { + fmt.Fprintln(out, "Already up to date.") + return nil + } + head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) + return finishMerge(out, repo, head, branch, push) + } + + // A ref that does not resolve fails before any conflict exists. + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } + + conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) + var model, other []string + for _, p := range conflicted { + if isModelPath(repo, p) { + model = append(model, p) + } else { + other = append(other, p) + } + } + if len(other) > 0 { + fmt.Fprintln(out, "Non-model conflicts, resolve by hand, then 'git add' and 'git commit':") + for _, p := range other { + fmt.Fprintf(out, " %s\n", p) + } + return errors.New("merge has non-model conflicts left for manual resolution") + } + + baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) + oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) + + // The human has decided: take one side of every conflicted model file. + // Deterministic, and mx is not consulted — `git checkout --ours/--theirs` + // reads the conflicted stages directly. + if strategy != "" { + side := "--" + strategy + for _, p := range model { + if _, err := gitOutIn(root, nil, "checkout", side, "--", p); err != nil { + return fmt.Errorf("take the %s version of %s: %w", strategy, p, err) + } + } + args := []string{"add", "--"} + args = append(args, model...) + if _, err := gitOutIn(root, nil, args...); err != nil { + return fmt.Errorf("stage the resolved model files: %w", err) + } + return commitTheMerge(root, repo, model, message, branch, push, out) + } + + // mx first: what it auto-resolves keeps both sides, which is almost + // always better than taking one side wholesale. + // Check out the base and theirs revisions; MINE is the working tree. + tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") + if err != nil { + return err + } + baseDir := filepath.Join(tmp, "base") + theirsDir := filepath.Join(tmp, "theirs") + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", baseDir, baseSha); err != nil { + os.RemoveAll(tmp) + return fmt.Errorf("check out the merge base: %w", err) + } + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", theirsDir, theirsSha); err != nil { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + os.RemoveAll(tmp) + return fmt.Errorf("check out %s: %w", ref, err) + } + cleanWorktrees := func() { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", theirsDir) + _ = os.RemoveAll(tmp) + } + + version := projectMendixVersion(mprPath) + fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", + strings.Join(shortNames(model), ", "), version) + code, mxOut, runErr := runMx( + filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), + abs, + filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), + [3]string{baseSha, oursSha, theirsSha}, + ) + if s := strings.TrimSpace(mxOut); s != "" { + fmt.Fprintln(out, s) + } + if runErr != nil { + cleanWorktrees() + return fmt.Errorf("mx git-merge failed: %w", runErr) + } + if code != 0 { + // The review step: show what each side changed, then leave nothing + // behind. The human decides — --ours/--theirs or Studio Pro. + cleanWorktrees() + printConflictReport(root, repo, baseSha, oursSha, theirsSha, out) + if _, abortErr := gitOutIn(root, nil, "merge", "--abort"); abortErr != nil { + return fmt.Errorf("the model merge has conflicts and aborting it failed (%v); resolve in Studio Pro", abortErr) + } + fmt.Fprintln(out, "Merge aborted; the working tree is back to the pre-merge state.") + return errors.New("the model merge needs a decision: re-run with --ours or --theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") + } + + args := []string{"add", "--"} + args = append(args, model...) + if _, err := gitOutIn(root, nil, args...); err != nil { + cleanWorktrees() + return fmt.Errorf("stage the resolved model files: %w", err) + } + if err := commitTheMerge(root, repo, nil, message, branch, push, out); err != nil { + cleanWorktrees() + return err + } + cleanWorktrees() + return nil +} + +// printConflictReport shows what each side changed — the review material a +// human needs before deciding how the conflict is resolved. +func printConflictReport(root string, repo *gitNoteRepo, baseSha, oursSha, theirsSha string, out io.Writer) { + base, err := repo.snapshot(baseSha) + if err != nil { + fmt.Fprintf(out, " (could not read the merge base: %v)\n", err) + return + } + ours, err := repo.snapshot(oursSha) + if err != nil { + fmt.Fprintf(out, " (could not read our side: %v)\n", err) + return + } + theirs, err := repo.snapshot(theirsSha) + if err != nil { + fmt.Fprintf(out, " (could not read their side: %v)\n", err) + return + } + oursChanges := computeModelChanges(base, ours, nil) + theirsChanges := computeModelChanges(base, theirs, nil) + theirsIDs := map[string]bool{} + for _, c := range theirsChanges { + theirsIDs[c.UnitID] = true + } + oursIDs := map[string]bool{} + for _, c := range oursChanges { + oursIDs[c.UnitID] = true + } + fmt.Fprintf(out, "\nBoth sides changed — a decision is needed on:\n") + for _, c := range oursChanges { + if theirsIDs[c.UnitID] { + fmt.Fprintf(out, " %s %s [%s] (unit %s)\n", c.UnitType, c.UnitName, c.Module, shortCommit(c.UnitID)) + } + } + fmt.Fprintf(out, "\nYours (%s):\n", shortCommit(oursSha)) + for _, c := range oursChanges { + fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(theirsIDs, c.UnitID)) + } + fmt.Fprintf(out, "\nTheirs (%s):\n", shortCommit(theirsSha)) + for _, c := range theirsChanges { + fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(oursIDs, c.UnitID)) + } +} + +func oursTheirsMark(other map[string]bool, unitID string) string { + if other[unitID] { + return " ◄ also changed on the other side" + } + return "" +} + +// commitTheMerge commits the resolved merge (MERGE_HEAD is present), writes +// the note and pushes. stagedPaths, when given, are staged first. +func commitTheMerge(root string, repo *gitNoteRepo, stagedPaths []string, message, branch string, push bool, out io.Writer) error { + if stagedPaths != nil { + args := []string{"add", "--"} + args = append(args, stagedPaths...) + if _, err := gitOutIn(root, nil, args...); err != nil { + return fmt.Errorf("stage the resolved model files: %w", err) + } + } + commitArgs := []string{"commit", "--no-edit"} + if message != "" { + commitArgs = []string{"commit", "-m", message} + } + if _, err := gitOutIn(root, nil, commitArgs...); err != nil { + return fmt.Errorf("commit the merge: %w", err) + } + + head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + fmt.Fprintf(out, "Model merge committed as %s.\n", shortCommit(head)) + return finishMerge(out, repo, head, branch, push) +} + +// finishMerge attaches the mx_metadata note to the finished merge and prints +// (or, with push, runs) the branch-and-notes push. +func finishMerge(out io.Writer, repo *gitNoteRepo, head, branch string, push bool) error { + if err := annotateCommits(out, repo, []string{head}, branch, true, false); err != nil { + return fmt.Errorf("committed %s, but attaching the note failed (%v) — run 'mxcli git note -p … --write' to attach it", shortCommit(head), err) + } + return pushBranchAndNotes(out, repo.Root, branch, push) +} + +func pushBranchAndNotes(out io.Writer, root, branch string, push bool) error { + if push { + if branch == "" { + return errors.New("detached HEAD; cannot push — check out a branch and push manually") + } + if _, err := gitOutIn(root, nil, "push", "origin", branch, mxMetadataRef); err != nil { + return fmt.Errorf("push %s and %s: %w", branch, mxMetadataRef, err) + } + fmt.Fprintf(out, "Pushed %s and %s to origin.\n", branch, mxMetadataRef) + return nil + } + if branch != "" { + fmt.Fprintf(out, "Push with: git push origin %s %s\n", branch, mxMetadataRef) + } + return nil +} + +// isModelPath reports whether a repo-relative path is part of the model: +// the .mpr itself or an .mxunit under mprcontents. +func isModelPath(repo *gitNoteRepo, rel string) bool { + f := filepath.ToSlash(rel) + if f == repo.MPRPath { + return true + } + return strings.HasPrefix(f, repo.Contents+"/") && strings.HasSuffix(f, ".mxunit") +} + +// projectMendixVersion reads the project's _ProductVersion for the mx +// resolution; "" on any error (ResolveMxForVersion then falls back to PATH). +func projectMendixVersion(mprPath string) string { + r, err := mpr.Open(mprPath) + if err != nil { + return "" + } + defer r.Close() + v, err := r.GetMendixVersion() + if err != nil { + return "" + } + return v +} + +// realMxGitMerge shells out to `mx git-merge -v …`, resolved for the +// project's Mendix version. +func realMxGitMerge(base, mine, theirs string, shas [3]string) (int, string, error) { + version := projectMendixVersion(mine) + mxPath, err := docker.ResolveMxForVersion("", version) + if err != nil { + return 1, "", fmt.Errorf("no mx for Mendix %s: %w (model conflicts need it)", version, err) + } + cmd := exec.Command(mxPath, "git-merge", "-v", base, mine, theirs, shas[0], shas[1], shas[2]) + b, err := cmd.CombinedOutput() + code := 0 + if err != nil { + var ee *exec.ExitError + if errors.As(err, &ee) { + code = ee.ExitCode() + } else { + return 1, string(b), err + } + } + return code, string(b), nil +} + +func shortNames(paths []string) []string { + out := make([]string, 0, len(paths)) + for _, p := range paths { + out = append(out, filepath.Base(p)) + } + return out +} diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index 2d43138c9..a621e2569 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -58,7 +58,8 @@ func TestGitDoctor_ConfigMissingReported(t *testing.T) { } } -func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { dir := t.TempDir() +func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { + dir := t.TempDir() doctorEnv(t, dir) gitIn(t, dir, "init", "-q", "-b", "main") @@ -603,7 +604,8 @@ func TestGitCommit_StagedListOmitsJunk(t *testing.T) { } } -func TestGitCommit_DryRunWithPushRefused(t *testing.T) { dir, mprPath := doctorNoteRepo(t) +func TestGitCommit_DryRunWithPushRefused(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) doctorEnv(t, dir) var out bytes.Buffer diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go new file mode 100644 index 000000000..a727bf953 --- /dev/null +++ b/cmd/mxcli/gitmerge_test.go @@ -0,0 +1,305 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +func bsonKeyValue(key, value string) bson.E { + return bson.E{Key: key, Value: value} +} + +// `mxcli git merge ` — merge a branch with model-aware resolution: git +// merge first, and on a model-file conflict `mx git-merge` (the same engine +// Studio Pro uses). The mx run is injected as runMx so the orchestration is +// testable without a Mendix installation; the real runner shells out to mx. + +// modelConflictRepo makes a repository whose merge of "side" into "feature" +// conflicts on both model files: each side changed the same DomainModel unit. +func modelConflictRepo(t *testing.T) (string, string) { + t.Helper() + dir, mprPath := doctorNoteRepo(t) + units := baseUnits(t) + units[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} + writeProject(t, dir, "11.13.0", units) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "main side") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + side := baseUnits(t) + side[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", sideKey)} + writeProject(t, dir, "11.13.0", side) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "side change") + gitIn(t, dir, "switch", "-q", "feature") + _, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflicted merge, got %v", err) + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); !strings.Contains(uu, ".mxunit") { + t.Fatalf("fixture: expected unresolved model files, got %q", uu) + } + return dir, mprPath +} + +var ( + mainSideKey = bsonKeyValue("DocA", "main") + sideKey = bsonKeyValue("DocB", "side") +) + +func okRunner(out string) mxMergeRunner { + return func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 0, out, nil + } +} + +func TestGitMerge_CleanMergeTextFileWritesNote(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + gitIn(t, dir, "branch", "docs", "HEAD~1") + gitIn(t, dir, "switch", "-q", "docs") + if err := os.WriteFile(filepath.Join(dir, "NOTES.txt"), []byte("docs\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "NOTES.txt") + gitIn(t, dir, "commit", "-q", "-m", "docs side") + gitIn(t, dir, "switch", "-q", "feature") + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("mx must not run on a merge git resolves itself") + return 0, "", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "docs", "", "", false, noMx, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + var n mxNote + if err := json.Unmarshal([]byte(noteOn(t, dir, "HEAD")), &n); err != nil { + t.Fatalf("merge commit has no valid note: %v", err) + } + if !n.HasModelerVersion || n.BranchName != "feature" { + t.Errorf("note header: %+v", n) + } +} + +func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + + var gotBase, gotMine, gotTheirs string + var gotShas [3]string + runner := func(base, mine, theirs string, shas [3]string) (int, string, error) { + gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas + return 0, "Merge successful.", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", false, runner, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + if !strings.Contains(gotBase, "mx-git-lab") && !strings.Contains(filepath.Base(gotBase), ".mpr") { + // the base path must be an .mpr inside a temporary checkout + if filepath.Ext(gotBase) != ".mpr" { + t.Errorf("runner got base %q, want the base checkout's .mpr", gotBase) + } + } + if filepath.Base(gotMine) != "App.mpr" { + t.Errorf("runner's mine = %q, want the working .mpr", gotMine) + } + if filepath.Base(gotTheirs) != "App.mpr" { + t.Errorf("runner's theirs = %q, want the theirs checkout's .mpr", gotTheirs) + } + if gotShas[0] == "" || gotShas[1] == gotShas[0] || gotShas[2] == gotShas[1] { + t.Errorf("shas must be base/ours/theirs distinct: %v", gotShas) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U") + if strings.TrimSpace(uu) != "" { + t.Errorf("conflicts still unresolved:\n%s", uu) + } + if !strings.Contains(noteOn(t, dir, "HEAD"), `"HasModelerVersion":true`) { + t.Errorf("merge commit has no note:\n%s", noteOn(t, dir, "HEAD")) + } +} + +func TestGitMerge_ModelConflictUnresolvedAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + fails := func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", false, fails, &out) + if err == nil || !strings.Contains(err.Error(), "Studio Pro") { + t.Fatalf("want a Studio Pro referral, got %v", err) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved despite the abort: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U") + if strings.TrimSpace(uu) != "" { + t.Errorf("conflicted files remain:\n%s", uu) + } +} + +func TestGitMerge_NonModelConflictLeftAlone(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + gitIn(t, dir, "checkout", "-q", "-b", "side", "HEAD~1") + if err := os.WriteFile(filepath.Join(dir, "NOTES.txt"), []byte("side\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "NOTES.txt") + gitIn(t, dir, "commit", "-q", "-m", "side notes") + gitIn(t, dir, "checkout", "-q", "feature") + if err := os.WriteFile(filepath.Join(dir, "NOTES.txt"), []byte("main\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "NOTES.txt") + gitIn(t, dir, "commit", "-q", "-m", "main notes") + _, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflict, got %v", err) + } + + calls := 0 + counting := func(base, mine, theirs string, shas [3]string) (int, string, error) { + calls++ + return 0, "", nil + } + var out bytes.Buffer + merr := mergeRun(mprPath, "side", "", "", false, counting, &out) + if merr == nil { + t.Fatal("a non-model conflict must be reported, not silently passed") + } + if calls != 0 { + t.Errorf("mx must not run for non-model conflicts (ran %d times)", calls) + } + uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U") + if !strings.Contains(uu, "NOTES.txt") { + t.Errorf("the conflicted text file must be left for manual resolution:\n%s", uu) + } + gitInE(t, dir, "merge", "--abort") // clean up the fixture +} + +func TestGitMerge_AlreadyUpToDate(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("mx must not run") + return 0, "", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "HEAD~1", "", "", false, noMx, &out); err != nil { + t.Fatalf("mergeRun: %v", err) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if !strings.Contains(out.String(), "up to date") { + t.Errorf("must say so:\n%s", out.String()) + } +} + +// The conflict report is the review material: on ManualResolutionRequired the +// merge is aborted and both sides' changes are printed, so a human can decide +// before anything lands. mx is not even needed for the report. +func TestGitMerge_ConflictReportShowsBothSidesAndAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "--ours") { + t.Fatalf("the report must point at the resolution flags, got %v", err) + } + report := out.String() + for _, want := range []string{"DomainModels$DomainModel", "also changed on the other side", "Yours", "Theirs"} { + if !strings.Contains(report, want) { + t.Errorf("report must contain %q:\n%s", want, report) + } + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// --ours / --theirs take the decided side for every conflicted model file, +// commit the merge and write the note — deterministically, without mx. +func TestGitMerge_SideTakingResolves(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--ours must not consult mx") + return 0, "", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "ours", false, noMx, &out); err != nil { + t.Fatalf("mergeRun --ours: %v\n%s", err, out.String()) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + if !strings.Contains(noteOn(t, dir, "HEAD"), `"HasModelerVersion":true`) { + t.Errorf("merge commit has no note") + } + page := filepath.Join(dir, "mprcontents", tDM[:2], tDM[2:4], tDM+".mxunit") + raw, err := os.ReadFile(page) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(raw, []byte("DocA")) || bytes.Contains(raw, []byte("DocB")) { + t.Error("--ours must leave our side of the unit on disk") + } + + // --theirs flips to the other side, deterministically. + dir2, mprPath2 := modelConflictRepo(t) + theirs := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--theirs must not consult mx") + return 0, "", nil + } + var out2 bytes.Buffer + if err := mergeRun(mprPath2, "side", "", "theirs", false, theirs, &out2); err != nil { + t.Fatalf("mergeRun --theirs: %v\n%s", err, out2.String()) + } + page2 := filepath.Join(dir2, "mprcontents", tDM[:2], tDM[2:4], tDM+".mxunit") + raw2, err := os.ReadFile(page2) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(raw2, []byte("DocB")) || bytes.Contains(raw2, []byte("DocA")) { + t.Error("--theirs must leave their side of the unit on disk") + } +} diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 6b71c0b19..bc6c67299 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -210,6 +210,44 @@ commit the conflict markers. `--dry-run` verifies and reports (including ContentsHash drift, which it does not repair — a dry run writes nothing), stages, reports what it would commit, and unstages again without committing. +## Merging branches: `mxcli git merge` + +`git merge` cannot merge Mendix models: the `.mpr` is SQLite and every +`.mxunit` is BSON, so two sides changing the model always meet a binary +`CONFLICT (content)` — there is nothing for git's textual merge to work on. +Studio Pro's *Merge Changes Here* resolves it with a model-aware merge, and +that engine ships as a command: `mx git-merge BASE MINE THEIRS …`. + +`mxcli git merge ` puts the two together: + +```bash +mxcli git merge feature/orders -p app.mpr +mxcli git merge origin/main -p app.mpr --push +``` + +`git merge ` runs first; a merge git resolves itself (disjoint files, +a fast-forward) is annotated with the mx_metadata note like `mxcli git +commit` does. On a model-file conflict the base and theirs revisions are +checked out into temporary worktrees and `mx git-merge` runs (resolved for +the project's Mendix version): + +- what mx auto-resolves is staged and committed as the normal two-parent + merge commit, with the note attached; +- a conflict mx reports as `ManualResolutionRequired` is never merged + silently: the merge is aborted, and **both sides' changes are printed for + review** — unit by unit, ours and theirs. Then decide: + `--ours` or `--theirs` takes one side of every conflicted model file + (deterministic; mx is not consulted again), or resolve for real in + Studio Pro (*Version Control › Merge Changes Here*), which stays the tool + for mixed resolutions; +- a conflict in non-model files (text sources) is left in place for manual + resolution, like plain git. + +Merging is still never done by plain `git merge` on a Mendix repository: use +`mxcli git merge` or Studio Pro, never a textual merge of `mprcontents/`. +Like every mxcli write, the command is refused while Studio Pro has the +project open. + ## Auditing a repository: `mxcli git doctor` A repository that has known only plain git misses several things Studio Pro From f75997910891b129ac0b245d055101efa20e2223 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 11:25:41 +0800 Subject: [PATCH 28/61] =?UTF-8?q?feat(git):=20per-unit=20conflict=20resolu?= =?UTF-8?q?tion=20=E2=80=94=20property-level=20report=20(mx=20diff)=20and?= =?UTF-8?q?=20a=20plan=20file=20the=20human=20edits=20before=20landing=20(?= =?UTF-8?q?lab=20S5c)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_merge.go | 367 ++++++++++++++++++---- cmd/mxcli/gitmerge_test.go | 192 +++++++++-- docs-site/src/tools/outside-studio-pro.md | 13 +- 4 files changed, 478 insertions(+), 96 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a9751d20..6b6be1f07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the human then re-runs with `--ours` or `--theirs` to take one side of every conflicted model file (deterministic, mx not consulted again), or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the run also writes a plan template (`.merge-plan.json`) beside the project, and the human then re-runs with `--plan ` to resolve **per unit** — each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files (modified-on-both units only; added/deleted/moved goes to Studio Pro) — or with `--ours`/`--theirs` to take one side wholesale, or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index ecdcebc74..8d4679e54 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -3,11 +3,14 @@ package main import ( + "bytes" + "encoding/json" "errors" "fmt" "io" "os" "os/exec" + "path" "path/filepath" "strings" @@ -45,12 +48,18 @@ version) decides: best outcome); - a conflict mx reports as ManualResolutionRequired is never merged silently: the merge is aborted, and both sides' changes are printed for - review. Then decide: - --ours take our side of every conflicted model file - --theirs take their side - (deterministic; mx is not consulted again), or resolve for real in - Studio Pro (Version Control › Merge Changes Here), which stays the tool - for mixed resolutions; + review — at property level, from 'mx diff' (e.g. + "New type > Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: + --plan resolve per unit from the plan template written beside + the .mpr (.merge-plan.json): set "take" to + "ours"/"theirs" per unit; the .mpr index is repaired + from the mixed files. Modified-on-both units only — + added/deleted/moved conflicts go to Studio Pro. + --ours / --theirs take one side of every conflicted model file + (deterministic; mx is not consulted again) + Studio Pro (Version Control › Merge Changes Here) for mixed + within-unit resolutions — still the tool when a decision + must mix both sides inside one unit; - a conflict in non-model files (text sources) is left in place for manual resolution, like plain git. @@ -72,6 +81,7 @@ func init() { gitMergeCmd.Flags().StringP("message", "m", "", "merge commit message (default: git's)") gitMergeCmd.Flags().Bool("ours", false, "on a model conflict mx cannot resolve, take our side of every conflicted model file") gitMergeCmd.Flags().Bool("theirs", false, "on a model conflict mx cannot resolve, take their side of every conflicted model file") + gitMergeCmd.Flags().String("plan", "", "resolve per unit from a plan file written by a previous run (.merge-plan.json)") gitMergeCmd.Flags().Bool("push", false, "push the branch and the notes ref to their remote afterwards") gitCmd.AddCommand(gitMergeCmd) } @@ -81,23 +91,31 @@ func runGitMerge(cmd *cobra.Command, args []string) error { message, _ := cmd.Flags().GetString("message") ours, _ := cmd.Flags().GetBool("ours") theirs, _ := cmd.Flags().GetBool("theirs") + plan, _ := cmd.Flags().GetString("plan") push, _ := cmd.Flags().GetBool("push") project, _ := cmd.Flags().GetString("project") strategy := "" switch { case ours && theirs: return errors.New("--ours and --theirs cannot be combined") + case ours && plan != "", theirs && plan != "": + return errors.New("--plan cannot be combined with --ours/--theirs: the plan is the finer decision") case ours: strategy = "ours" case theirs: strategy = "theirs" } - return mergeRun(project, ref, message, strategy, push, realMxGitMerge, cmd.OutOrStdout()) + return mergeRun(project, ref, message, strategy, plan, push, mxRunners{merge: realMxGitMerge, diff: realMxDiff}, cmd.OutOrStdout()) } -// mxMergeRunner runs `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT -// THEIRS_COMMIT` and returns mx's exit code and output. +// mxRunners are the mx invocations, injected for tests. type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, error) +type mxDiffRunner func(base, mine, outJSON string) error + +type mxRunners struct { + merge mxMergeRunner + diff mxDiffRunner +} // mergeRun is the command without its wiring. runMx is injected for tests. // mergeRun is the command without its wiring. runMx is injected for tests. @@ -112,7 +130,7 @@ type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, // with no strategy the merge is aborted and both sides' changes are // printed for review; `--ours`/`--theirs` then take one side wholesale // (mx is not consulted — the human has decided). -func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRunner, out io.Writer) error { +func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners mxRunners, out io.Writer) error { switch strategy { case "", "ours", "theirs": default: @@ -139,6 +157,10 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu return errors.New("detached HEAD: check out a branch before merging") } + if plan != "" { + return applyPlan(root, repo, abs, mprPath, ref, plan, message, branch, push, out) + } + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) if mergeErr == nil { if strings.Contains(mergeOut, "Already up to date") { @@ -221,7 +243,7 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu version := projectMendixVersion(mprPath) fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", strings.Join(shortNames(model), ", "), version) - code, mxOut, runErr := runMx( + code, mxOut, runErr := runners.merge( filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), abs, filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), @@ -235,15 +257,29 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu return fmt.Errorf("mx git-merge failed: %w", runErr) } if code != 0 { - // The review step: show what each side changed, then leave nothing - // behind. The human decides — --ours/--theirs or Studio Pro. + // The review step: what each side changed, at property level (mx + // diff), plus a per-unit plan template. Then leave nothing behind — + // the human decides. + if runners.diff != nil { + oursJSON := filepath.Join(tmp, "ours-diff.json") + theirsJSON := filepath.Join(tmp, "theirs-diff.json") + d1 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), abs, oursJSON) + d2 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), theirsJSON) + if d1 == nil && d2 == nil { + fmt.Fprintln(out, propertyReport(oursJSON, theirsJSON, conflictedUnitIDs(model))) + } + } + if err := writePlanTemplate(mprPath, baseSha, oursSha, theirsSha, model); err != nil { + fmt.Fprintf(out, " (could not write the plan template: %v)\n", err) + } else { + fmt.Fprintf(out, "Plan template: %s — set \"take\" per unit, re-run with --plan.\n", filepath.Base(mprPath)+".merge-plan.json") + } cleanWorktrees() - printConflictReport(root, repo, baseSha, oursSha, theirsSha, out) if _, abortErr := gitOutIn(root, nil, "merge", "--abort"); abortErr != nil { return fmt.Errorf("the model merge has conflicts and aborting it failed (%v); resolve in Studio Pro", abortErr) } fmt.Fprintln(out, "Merge aborted; the working tree is back to the pre-merge state.") - return errors.New("the model merge needs a decision: re-run with --ours or --theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") + return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, --ours/--theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") } args := []string{"add", "--"} @@ -260,57 +296,6 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu return nil } -// printConflictReport shows what each side changed — the review material a -// human needs before deciding how the conflict is resolved. -func printConflictReport(root string, repo *gitNoteRepo, baseSha, oursSha, theirsSha string, out io.Writer) { - base, err := repo.snapshot(baseSha) - if err != nil { - fmt.Fprintf(out, " (could not read the merge base: %v)\n", err) - return - } - ours, err := repo.snapshot(oursSha) - if err != nil { - fmt.Fprintf(out, " (could not read our side: %v)\n", err) - return - } - theirs, err := repo.snapshot(theirsSha) - if err != nil { - fmt.Fprintf(out, " (could not read their side: %v)\n", err) - return - } - oursChanges := computeModelChanges(base, ours, nil) - theirsChanges := computeModelChanges(base, theirs, nil) - theirsIDs := map[string]bool{} - for _, c := range theirsChanges { - theirsIDs[c.UnitID] = true - } - oursIDs := map[string]bool{} - for _, c := range oursChanges { - oursIDs[c.UnitID] = true - } - fmt.Fprintf(out, "\nBoth sides changed — a decision is needed on:\n") - for _, c := range oursChanges { - if theirsIDs[c.UnitID] { - fmt.Fprintf(out, " %s %s [%s] (unit %s)\n", c.UnitType, c.UnitName, c.Module, shortCommit(c.UnitID)) - } - } - fmt.Fprintf(out, "\nYours (%s):\n", shortCommit(oursSha)) - for _, c := range oursChanges { - fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(theirsIDs, c.UnitID)) - } - fmt.Fprintf(out, "\nTheirs (%s):\n", shortCommit(theirsSha)) - for _, c := range theirsChanges { - fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(oursIDs, c.UnitID)) - } -} - -func oursTheirsMark(other map[string]bool, unitID string) string { - if other[unitID] { - return " ◄ also changed on the other side" - } - return "" -} - // commitTheMerge commits the resolved merge (MERGE_HEAD is present), writes // the note and pushes. stagedPaths, when given, are staged first. func commitTheMerge(root string, repo *gitNoteRepo, stagedPaths []string, message, branch string, push bool, out io.Writer) error { @@ -414,3 +399,253 @@ func shortNames(paths []string) []string { } return out } + +// conflictedUnitIDs extracts the unit ids of conflicted .mxunit paths. +func conflictedUnitIDs(model []string) []string { + var ids []string + for _, p := range model { + if strings.HasSuffix(p, ".mxunit") { + ids = append(ids, strings.TrimSuffix(filepath.Base(p), ".mxunit")) + } + } + return ids +} + +// --- property-level conflict report and the per-unit plan ------------------- + +type mxDiffChange struct { + Change string `json:"change"` + ChangeDescription string `json:"changeDescription"` + Property string `json:"property"` + BaseValue string `json:"baseValue"` + MineValue string `json:"mineValue"` + ChangeDetails []mxDiffChange `json:"changeDetails"` +} + +type mxDiffUnit struct { + ID string `json:"id"` + Type string `json:"type"` + Change string `json:"change"` + ChangeDetails []mxDiffChange `json:"changeDetails"` +} + +type mxDiffFile struct { + UnitDifferences []mxDiffUnit `json:"unitDifferences"` +} + +// propertyReport renders the two mx diff files as one property-level report +// over the conflicted units: what each side changed. +func propertyReport(oursJSON, theirsJSON string, unitIDs []string) string { + want := map[string]bool{} + for _, id := range unitIDs { + want[id] = true + } + lines := map[string][]string{} + for _, f := range []struct{ side, path string }{{"Yours", oursJSON}, {"Theirs", theirsJSON}} { + raw, err := os.ReadFile(f.path) + if err != nil { + lines[f.side] = append(lines[f.side], fmt.Sprintf(" (diff unavailable: %v)", err)) + continue + } + var d mxDiffFile + if err := json.Unmarshal(bytes.TrimPrefix(raw, []byte("\xef\xbb\xbf")), &d); err != nil { + lines[f.side] = append(lines[f.side], fmt.Sprintf(" (diff unreadable: %v)", err)) + continue + } + var side []string + for _, u := range d.UnitDifferences { + if !want[u.ID] { + continue + } + side = append(side, flattenChanges(u.ChangeDetails, " ")...) + } + if len(side) == 0 { + side = append(side, " (no change details)") + } + lines[f.side] = side + } + var b strings.Builder + b.WriteString("\nProperty-level changes on the conflicted units:\n") + for _, side := range []string{"Yours", "Theirs"} { + fmt.Fprintf(&b, "%s:\n", side) + for _, l := range lines[side] { + b.WriteString(l) + b.WriteString("\n") + } + } + return b.String() +} + +// flattenChanges walks mx diff's change tree and yields one line per leaf: +// the description, and for property leaves the value movement. +func flattenChanges(changes []mxDiffChange, indent string) []string { + var out []string + for _, c := range changes { + if c.Property != "" { + out = append(out, fmt.Sprintf("%s%s: %s → %s", indent, c.Property, c.BaseValue, c.MineValue)) + continue + } + desc := c.ChangeDescription + if desc == "" { + desc = c.Change + } + out = append(out, fmt.Sprintf("%s%s", indent, desc)) + out = append(out, flattenChanges(c.ChangeDetails, indent+" ")...) + } + return out +} + +// mergePlanTemplate is the file a ManualResolutionRequired run writes beside +// the .mpr; the human sets "take" per unit and re-runs with --plan. +type mergePlanTemplate struct { + Base string `json:"base"` + Ours string `json:"ours"` + Theirs string `json:"theirs"` + Units []planUnit `json:"units"` +} + +type planUnit struct { + ID string `json:"id"` + Take string `json:"take"` +} + +func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, model []string) error { + tmpl := mergePlanTemplate{Base: baseSha, Ours: oursSha, Theirs: theirsSha} + for _, id := range conflictedUnitIDs(model) { + tmpl.Units = append(tmpl.Units, planUnit{ID: id, Take: "ours"}) + } + b, err := json.MarshalIndent(tmpl, "", " ") + if err != nil { + return err + } + return os.WriteFile(mprPath+".merge-plan.json", b, 0o644) +} + +// applyPlan re-runs the merge (it must conflict the same way again), checks +// the plan against those revisions, resolves per unit, repairs the .mpr index +// from the mixed files and commits — with the note. +func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, message, branch string, push bool, out io.Writer) error { + raw, err := os.ReadFile(planPath) + if err != nil { + return fmt.Errorf("read plan: %w", err) + } + var plan struct { + Base string `json:"base"` + Ours string `json:"ours"` + Theirs string `json:"theirs"` + Units []planUnit `json:"units"` + } + if err := json.Unmarshal(raw, &plan); err != nil { + return fmt.Errorf("plan %s is not valid JSON: %w", planPath, err) + } + + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") + } + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } + baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) + oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) + if plan.Base != baseSha || plan.Ours != oursSha || plan.Theirs != theirsSha { + _, _ = gitOutIn(root, nil, "merge", "--abort") + return errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate") + } + + conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) + var model []string + for _, p := range conflicted { + if isModelPath(repo, p) { + model = append(model, p) + } + } + + // Every planned unit must be modified on both sides; anything else + // (added on one side, deleted, moved) only Studio Pro can decide. + baseSnap, err := repo.snapshot(baseSha) + if err != nil { + return err + } + oursSnap, err := repo.snapshot(oursSha) + if err != nil { + return err + } + theirsSnap, err := repo.snapshot(theirsSha) + if err != nil { + return err + } + oursStatus := changeStatuses(computeModelChanges(baseSnap, oursSnap, nil)) + theirsStatus := changeStatuses(computeModelChanges(baseSnap, theirsSnap, nil)) + take := map[string]string{} + for _, u := range plan.Units { + if u.Take != "ours" && u.Take != "theirs" { + return fmt.Errorf("plan: unit %s has take %q (ours|theirs)", shortCommit(u.ID), u.Take) + } + if oursStatus[u.ID] != "Modified" || theirsStatus[u.ID] != "Modified" { + return fmt.Errorf("plan: unit %s is %s on ours and %s on theirs — only modified-on-both conflicts can be planned; resolve it in Studio Pro", shortCommit(u.ID), statusOrNone(oursStatus[u.ID]), statusOrNone(theirsStatus[u.ID])) + } + take[u.ID] = u.Take + } + + // Ours everywhere, then the planned units from theirs. + for _, p := range model { + if _, err := gitOutIn(root, nil, "checkout", "--ours", "--", p); err != nil { + return fmt.Errorf("take the ours version of %s: %w", p, err) + } + } + for _, id := range conflictedUnitIDs(model) { + if take[id] != "theirs" { + continue + } + p := path.Join(repo.Contents, id[:2], id[2:4], id+".mxunit") + if _, err := gitOutIn(root, nil, "checkout", "--theirs", "--", p); err != nil { + return fmt.Errorf("take the theirs version of %s: %w", p, err) + } + fmt.Fprintf(out, " %s: theirs\n", shortCommit(id)) + } + // The .mpr index is ours; the mixed unit files have different bytes, so + // the hashes are rewritten from the files — files are the truth. + if err := fixHashes(mprPath, true, out); err != nil { + if !errors.Is(err, errHashIssuesRemain) { + return err + } + return errors.New("refusing to commit: the ContentsHash index has MISSING or ORPHAN entries (see above)") + } + + args := []string{"add", "--"} + args = append(args, model...) + if _, err := gitOutIn(root, nil, args...); err != nil { + return fmt.Errorf("stage the resolved model files: %w", err) + } + return commitTheMerge(root, repo, nil, message, branch, push, out) +} + +func changeStatuses(changes []mxModelChange) map[string]string { + out := map[string]string{} + for _, c := range changes { + out[c.UnitID] = c.Status + } + return out +} + +func statusOrNone(s string) string { + if s == "" { + return "unchanged" + } + return s +} + +// realMxDiff shells out to `mx diff BASE MINE OUTPUT` (resolved like git-merge). +func realMxDiff(base, mine, outJSON string) error { + version := projectMendixVersion(mine) + mxPath, err := docker.ResolveMxForVersion("", version) + if err != nil { + return fmt.Errorf("no mx for Mendix %s: %w", version, err) + } + if b, err := exec.Command(mxPath, "diff", base, mine, outJSON).CombinedOutput(); err != nil { + return fmt.Errorf("mx diff: %v: %s", err, strings.TrimSpace(string(b))) + } + return nil +} diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index a727bf953..668feed6d 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -5,6 +5,7 @@ package main import ( "bytes" "encoding/json" + "fmt" "os" "path/filepath" "strings" @@ -49,6 +50,38 @@ func modelConflictRepo(t *testing.T) (string, string) { return dir, mprPath } +// twoUnitConflictRepo is a modelConflictRepo where BOTH the domain model and +// the microflow unit were changed on each side — two conflicted units, so a +// plan can take one unit from ours and the other from theirs. +func twoUnitConflictRepo(t *testing.T) (string, string) { + t.Helper() + dir, mprPath := doctorNoteRepo(t) + units := baseUnits(t) + units[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} + units[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", mainSideKey)} + writeProject(t, dir, "11.13.0", units) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "main side") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + side := baseUnits(t) + side[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", sideKey)} + side[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", sideKey)} + writeProject(t, dir, "11.13.0", side) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "side change") + gitIn(t, dir, "switch", "-q", "feature") + _, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflicted merge, got %v", err) + } + return dir, mprPath +} + +// unitRel is the mprcontents path of a unit id inside a work tree. +func unitRel(id string) string { + return "mprcontents/" + id[:2] + "/" + id[2:4] + "/" + id + ".mxunit" +} + var ( mainSideKey = bsonKeyValue("DocA", "main") sideKey = bsonKeyValue("DocB", "side") @@ -72,12 +105,12 @@ func TestGitMerge_CleanMergeTextFileWritesNote(t *testing.T) { gitIn(t, dir, "commit", "-q", "-m", "docs side") gitIn(t, dir, "switch", "-q", "feature") - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("mx must not run on a merge git resolves itself") return 0, "", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "docs", "", "", false, noMx, &out); err != nil { + if err := mergeRun(mprPath, "docs", "", "", "", false, noMx, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) @@ -99,12 +132,12 @@ func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { var gotBase, gotMine, gotTheirs string var gotShas [3]string - runner := func(base, mine, theirs string, shas [3]string) (int, string, error) { + runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas return 0, "Merge successful.", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "side", "", "", false, runner, &out); err != nil { + if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } if !strings.Contains(gotBase, "mx-git-lab") && !strings.Contains(filepath.Base(gotBase), ".mpr") { @@ -140,11 +173,11 @@ func TestGitMerge_ModelConflictUnresolvedAborts(t *testing.T) { doctorEnv(t, dir) before := commitSubject(t, dir) - fails := func(base, mine, theirs string, shas [3]string) (int, string, error) { + fails := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil - } + }} var out bytes.Buffer - err := mergeRun(mprPath, "side", "", "", false, fails, &out) + err := mergeRun(mprPath, "side", "", "", "", false, fails, &out) if err == nil || !strings.Contains(err.Error(), "Studio Pro") { t.Fatalf("want a Studio Pro referral, got %v", err) } @@ -181,12 +214,12 @@ func TestGitMerge_NonModelConflictLeftAlone(t *testing.T) { } calls := 0 - counting := func(base, mine, theirs string, shas [3]string) (int, string, error) { + counting := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { calls++ return 0, "", nil - } + }} var out bytes.Buffer - merr := mergeRun(mprPath, "side", "", "", false, counting, &out) + merr := mergeRun(mprPath, "side", "", "", "", false, counting, &out) if merr == nil { t.Fatal("a non-model conflict must be reported, not silently passed") } @@ -205,12 +238,12 @@ func TestGitMerge_AlreadyUpToDate(t *testing.T) { doctorEnv(t, dir) before := commitSubject(t, dir) - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("mx must not run") return 0, "", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "HEAD~1", "", "", false, noMx, &out); err != nil { + if err := mergeRun(mprPath, "HEAD~1", "", "", "", false, noMx, &out); err != nil { t.Fatalf("mergeRun: %v", err) } if got := commitSubject(t, dir); got != before { @@ -229,16 +262,16 @@ func TestGitMerge_ConflictReportShowsBothSidesAndAborts(t *testing.T) { doctorEnv(t, dir) before := commitSubject(t, dir) - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil - } + }} var out bytes.Buffer - err := mergeRun(mprPath, "side", "", "", false, noMx, &out) + err := mergeRun(mprPath, "side", "", "", "", false, noMx, &out) if err == nil || !strings.Contains(err.Error(), "--ours") { t.Fatalf("the report must point at the resolution flags, got %v", err) } report := out.String() - for _, want := range []string{"DomainModels$DomainModel", "also changed on the other side", "Yours", "Theirs"} { + for _, want := range []string{"Plan template", "merge-plan.json", "Merge aborted"} { if !strings.Contains(report, want) { t.Errorf("report must contain %q:\n%s", want, report) } @@ -260,12 +293,12 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { dir, mprPath := modelConflictRepo(t) doctorEnv(t, dir) - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("--ours must not consult mx") return 0, "", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "side", "", "ours", false, noMx, &out); err != nil { + if err := mergeRun(mprPath, "side", "", "ours", "", false, noMx, &out); err != nil { t.Fatalf("mergeRun --ours: %v\n%s", err, out.String()) } parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) @@ -286,12 +319,12 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { // --theirs flips to the other side, deterministically. dir2, mprPath2 := modelConflictRepo(t) - theirs := func(base, mine, theirs string, shas [3]string) (int, string, error) { + theirs := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("--theirs must not consult mx") return 0, "", nil - } + }} var out2 bytes.Buffer - if err := mergeRun(mprPath2, "side", "", "theirs", false, theirs, &out2); err != nil { + if err := mergeRun(mprPath2, "side", "", "theirs", "", false, theirs, &out2); err != nil { t.Fatalf("mergeRun --theirs: %v\n%s", err, out2.String()) } page2 := filepath.Join(dir2, "mprcontents", tDM[:2], tDM[2:4], tDM+".mxunit") @@ -303,3 +336,112 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { t.Error("--theirs must leave their side of the unit on disk") } } + +// A plan resolves per unit: tDM from ours, tMf from theirs — one merge commit, +// hashes repaired from the mixed files, note attached. +func TestGitMerge_PlanMixesPerUnit(t *testing.T) { + dir, mprPath := twoUnitConflictRepo(t) + doctorEnv(t, dir) + base := strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")) + ours := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) + theirs := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")) + plan := fmt.Sprintf(`{"base":%q,"ours":%q,"theirs":%q,"units":[{"id":%q,"take":"ours"},{"id":%q,"take":"theirs"}]}`, + base, ours, theirs, tDM, tMf) + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--plan must not consult mx") + return 0, "", nil + }} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out); err != nil { + t.Fatalf("mergeRun --plan: %v\n%s", err, out.String()) + } + assertUnitHas := func(id, key string, want bool) { + t.Helper() + raw, err := os.ReadFile(filepath.Join(dir, unitRel(id))) + if err != nil { + t.Fatal(err) + } + if got := bytes.Contains(raw, []byte(key)); got != want { + t.Errorf("%s contains %q = %v, want %v", id, key, got, want) + } + } + assertUnitHas(tDM, "DocA", true) + assertUnitHas(tDM, "DocB", false) + assertUnitHas(tMf, "DocB", true) + assertUnitHas(tMf, "DocA", false) + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + var out2 bytes.Buffer + if err := fixHashes(mprPath, false, &out2); err != nil { + t.Errorf("hashes drift after the plan merge: %v\n%s", err, out2.String()) + } +} + +// A plan whose revisions no longer match the merge state is stale and refused. +func TestGitMerge_PlanStaleRefused(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + plan := `{"base":"0000000000000000000000000000000000000000","ours":"0000000000000000000000000000000000000001","theirs":"0000000000000000000000000000000000000002","units":[]}` + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + noMx := mxRunners{} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "stale") { + t.Fatalf("want a stale-plan refusal, got %v", err) + } + gitInE(t, dir, "merge", "--abort") // clean up the fixture +} + +// On ManualResolutionRequired the report is property-level (from mx diff) and +// a plan template is written beside the .mpr. +func TestGitMerge_ManualWritesPlanAndPropertyReport(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + fixture := `{"unitDifferences":[{"id":"` + tDM + `","type":"DomainModels$DomainModel","change":"Modified","changeDetails":[{"change":"Modified","changeDescription":"Attribute 'MyFirstModule.CreatedByMxcli.Nickname'","changeDetails":[{"property":"New type > Length","baseValue":"100","mineValue":"150"}]}]}]}` + mx := mxRunners{ + merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil + }, + diff: func(base, mine, outJSON string) error { + return os.WriteFile(outJSON, []byte(fixture), 0o644) + }, + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", "", false, mx, &out) + if err == nil { + t.Fatal("a manual conflict must end with the decision error") + } + report := out.String() + for _, want := range []string{"Length", "100", "150", "Nickname"} { + if !strings.Contains(report, want) { + t.Errorf("report must contain %q:\n%s", want, report) + } + } + planRaw, err := os.ReadFile(mprPath + ".merge-plan.json") + if err != nil { + t.Fatalf("plan template not written: %v", err) + } + var plan struct { + Units []struct { + Id string `json:"id"` + Take string `json:"take"` + } `json:"units"` + } + if err := json.Unmarshal(planRaw, &plan); err != nil { + t.Fatal(err) + } + if len(plan.Units) != 1 || plan.Units[0].Id != tDM || plan.Units[0].Take != "ours" { + t.Errorf("plan template: %+v", plan) + } + gitInE(t, dir, "merge", "--abort") // clean up the fixture +} diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index bc6c67299..fee22c505 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -235,11 +235,16 @@ the project's Mendix version): merge commit, with the note attached; - a conflict mx reports as `ManualResolutionRequired` is never merged silently: the merge is aborted, and **both sides' changes are printed for - review** — unit by unit, ours and theirs. Then decide: + review** — at property level, from `mx diff` ("New type > Length: + 100 → 150" on ours, "→ 200" on theirs). Then decide: + `--plan ` resolves **per unit** from the plan template written beside + the .mpr (`.merge-plan.json`): set `"take"` to `ours`/`theirs` per + unit, and the `.mpr` index is repaired from the mixed files (modified-on- + both units only; added/deleted/moved conflicts go to Studio Pro). `--ours` or `--theirs` takes one side of every conflicted model file - (deterministic; mx is not consulted again), or resolve for real in - Studio Pro (*Version Control › Merge Changes Here*), which stays the tool - for mixed resolutions; + (deterministic; mx is not consulted again). Resolutions that must mix both + sides *inside* one unit stay in Studio Pro (*Version Control › Merge + Changes Here*); - a conflict in non-model files (text sources) is left in place for manual resolution, like plain git. From 61c38e1dd0f7bf3568d6c028e37266b9186b7fbd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:39:26 +0000 Subject: [PATCH 29/61] fix(gitignore): ignore the unit-type index under testdata fixtures too The pattern `.mxcli/unit-types.idx` has a slash in the middle, so git anchors it to the repository root: the index mxcli writes next to a fixture a test opens in place (testdata/testapp-views/.mxcli/) showed up as untracked, though the comment above the line says it is meant to be ignored there. `**/` matches it at any depth: `git check-ignore` did not match the nested copy before, and matches it and the root one after. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 5b15d750c..0ae724663 100644 --- a/.gitignore +++ b/.gitignore @@ -73,4 +73,4 @@ snap-bson go-test-output.txt # MPR v2 unit-type index, written next to any project mxcli opens — including # the testdata/ fixtures tests open in place. -.mxcli/unit-types.idx +**/.mxcli/unit-types.idx From 4335dff9a133c4b2fbd5392b16f05fa84d329541 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:27:59 +0000 Subject: [PATCH 30/61] fix(layout): an error handler's row reserves the lane under its activity Without @position a custom error handler is a row 130px under its activity, and the main line carries on one spacing after the activity however long the row is. When the next activity had a handler too, its row started at its own x, inside the first: the first handler's end event and the second handler's first activity were both stored at 680;330 (ako/mxcli#1768, mendixlabs/mxcli#1672). The lane model from #1154 already made an activity with a handler wait for a reserved lane; the handler row was the occupant that never reserved it. addErrorHandlerFlow now reserves the lane to the row's measured right edge plus the activity gap, and measureStatements does the same and counts the row in its extent, so a branch or loop holding handler rows is measured as built. Without the measurer half, 12 of the new shapes put a merge on an activity. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...onsecutive-error-handler-rows-overlap.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/syntax/features_microflow.go | 13 +- docs-site/src/language/microflow-structure.md | 14 +- .../1768-error-handler-rows-share-a-lane.mdl | 66 ++++ mdl/executor/cmd_microflows_builder_flows.go | 4 + mdl/executor/layout.go | 8 + mdl/executor/layout_handler_lane_1768_test.go | 337 ++++++++++++++++++ mdl/executor/layout_lanes.go | 34 ++ 9 files changed, 468 insertions(+), 10 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json create mode 100644 mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl create mode 100644 mdl/executor/layout_handler_lane_1768_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json new file mode 100644 index 000000000..aff06834a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-29","symptom":"Two consecutive activities with custom error handlers and no @position drew their handler rows on top of each other: the first handler's end event and the second handler's first activity both at 680;330 ($First 360;200, $Body 520;200, $Second 680;200). layout flows --dry-run said 'already laid out' (same builder), MPR008 said nothing (end events not compared), mx check 0 errors. A @position on a later statement also moved every unplaced statement before it onto that statement.","cause":"addErrorHandlerFlow built the row 130px under the activity, running right, but never reserved the lower lane, while clearLowerLane already made the next activity with a handler wait for a reserved lane (the #1154 guard design); measureStatements likewise reserved only for guards and left handler rows out of the extent that fitContainerSize counts, so a loop with a long trailing row was built 110px wider than measured. Case 2: buildFlowGraph's start pre-scan took the first statement WITH an @position instead of the first statement.","file":"mdl/executor/layout_lanes.go (handlerRowWidth, reserveHandlerRow), mdl/executor/layout.go (measureStatements), mdl/executor/cmd_microflows_builder_flows.go (addErrorHandlerFlow), mdl/executor/cmd_microflows_builder_graph.go (start pre-scan), mdl/linter/rules/mpr008_overlapping_activities.go","insight":"The lane model already stated the rule ('an activity with an error handler waits for the lane'); one of the lane's two occupants never registered itself. Look for the occupant that does not reserve before inventing a mechanism. Reserve from the measurer, in builder and measurer alike: the builder-only half (control) left 12 shapes with a merge landing on an activity. Studio Pro reference: all 8 multi-object handlers in PedApp and ako/TestApp are COLUMNS straight under the activity (80-130px per step), none is a row and none has a consecutive pair; a column layout would make the collision impossible but moves every flow with a 2+ statement handler and needs a vertical layout for compound handler bodies, so the lane was kept (2 of 182 example flows move). Two older measurer gaps surfaced by the new invariant test and were left alone: an IF is measured 45px short (the split's centre is counted as its left edge in guardBranchInset and measureIfStatement), and a `join` is measured as an activity plus an end event although it draws nothing, so a handler row ending in joins reserves about twice its drawn width. Check overlaps with the whole node set: MPR008 skipped events, which hid exactly this overlap; on 292 Studio Pro flows the wider set adds 0 findings.","refs":["ako/mxcli#1768","mendixlabs/mxcli#1672"],"test":"TestSecondErrorHandlerRowClearsTheFirst, TestErrorHandlerRowsNeverOverlap, TestLoopAroundHandlerRowsIsMeasuredAsBuilt, TestStatementsBeforeAPlacedOneKeepTheirDerivedPlaces, TestOverlapComparesEvents"} diff --git a/CHANGELOG.md b/CHANGELOG.md index ea06bf20b..1d08a4181 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. +- **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 9cfd46949..ae96ed08c 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -932,11 +932,14 @@ func init() { "right and wraps onto a new row past 2880px; the line joining two rows leaves\n" + "the bottom of one and arrives on top of the next. A guard (if … return; end\n" + "if) drops its branch into the lane below and the main line carries straight\n" + - "on over it. A CASE of four or more branches leaves the split in three groups\n" + - "— top, right, bottom — so its lines do not cross. A statement that carries\n" + - "@position is never moved, and starts the row for what follows it. Prefer no\n" + - "@position at all to a few: hand-placed statements are not measured against\n" + - "what the builder puts around them. (#1154)\n\n" + + "on over it; a custom error handler's row hangs in that lane too, 130px under\n" + + "its activity. A statement that reaches into the lane — a decision, a loop, an\n" + + "activity with a handler of its own — waits for it to clear; a plain activity\n" + + "does not (ako/mxcli#1768). A CASE of four or more branches leaves the split\n" + + "in three groups — top, right, bottom — so its lines do not cross. A\n" + + "statement that carries @position is never moved, and starts the row for what\n" + + "follows it. Prefer no @position at all to a few: hand-placed statements are\n" + + "not measured against what the builder puts around them. (#1154)\n\n" + "@start and @merge position the two nodes that have no statement of their\n" + "own, so each is written on the statement it belongs to. Omit @start and the\n" + "start is placed one spacing unit left of the first activity, on its centre\n" + diff --git a/docs-site/src/language/microflow-structure.md b/docs-site/src/language/microflow-structure.md index d7494c2b5..71eb46801 100644 --- a/docs-site/src/language/microflow-structure.md +++ b/docs-site/src/language/microflow-structure.md @@ -152,11 +152,15 @@ $Order = CREATE Sales.Order (Status = 'New'); Positions are optional. A microflow written without them is laid out by mxcli: the main line runs left to right and wraps onto a new row once it passes two canvas widths (2880 px); a guard — `if … then …; return; end if` — drops its branch into the -lane below while the main line carries on above it; and a `case` of four or more -branches leaves the decision in three groups (top, right, bottom) so its lines do not -cross. A statement that carries `@position` is never moved, and becomes the start of -the row for the statements after it — so either place everything or nothing: a few -hand-placed statements are not measured against what is laid out around them. +lane below while the main line carries on above it; a custom error handler's row hangs +in that same lane, 130 px under its activity; and a `case` of four or more branches +leaves the decision in three groups (top, right, bottom) so its lines do not cross. +Whatever reaches down into the lane — a decision, a loop, an activity with an error +handler of its own — waits for the lane to clear, so two consecutive handlers never +share a row; a plain activity does not wait. A statement that carries `@position` is +never moved, and becomes the start of the row for the statements after it — so either +place everything or nothing: a few hand-placed statements are not measured against +what is laid out around them. To re-arrange a flow that already exists — one drawn in Studio Pro, or one whose positions no longer fit after edits — run `mxcli layout flows`. It applies this diff --git a/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl new file mode 100644 index 000000000..bf28982c1 --- /dev/null +++ b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl @@ -0,0 +1,66 @@ +-- ============================================================================ +-- ako/mxcli#1768 (mendixlabs/mxcli#1672) — the error handlers of two +-- consecutive activities were drawn in one row, the second on the first +-- ============================================================================ +-- +-- Without @position a custom error handler is a row 130px under its activity, +-- running right, and the main line carries on one spacing after the activity +-- whatever the row's length. A second activity with a handler started its row +-- at its own x, inside the first. Reported, from `describe … with handles`: +-- +-- first handler: $FirstFailed 360;330, log 520;330, end event 680;330 +-- $Second 680;200, second handler: $SecondFailed 680;330 <- on the end event +-- +-- `layout flows --dry-run` called the flow already laid out, and MPR008 said +-- nothing, because it did not compare end events. +-- +-- Expected after exec: `describe microflow BugTest1768.SUB_TwoHandlers with +-- handles` keeps $First, its handler row and $Body where they were (360;200, +-- 360..680;330, 520;200) and places $Second at 790;200 with its row at +-- 790..1110;330 — clear of the first row's end, with the gap two activities +-- keep. `lint -r MPR008` reports nothing, and `layout flows --dry-run` calls +-- every flow below already laid out. + +mdl 1; + +create module BugTest1768; + +create non-persistent entity BugTest1768.Result (IsDone: Boolean, Code: Integer); + +create or modify microflow BugTest1768.SUB_TwoHandlers ($Url: String) returns BugTest1768.Result as $Result +begin + $First = call rest service get '{1}' with ({1} = $Url) (Timeout: 30) returns response on error without rollback begin + $FirstFailed = create BugTest1768.Result (Code = 1); + log error node 'BugTest1768' 'first failed'; + return $FirstFailed; + end error; + declare $Body String = $First/Content; + $Second = call rest service get '{1}' with ({1} = $Body) (Timeout: 30) returns response on error without rollback begin + $SecondFailed = create BugTest1768.Result (Code = 2); + log error node 'BugTest1768' 'second failed'; + return $SecondFailed; + end error; + $Done = create BugTest1768.Result (IsDone = true); + return $Done; +end; + +-- One more statement in the first handler: its row then reached 840;330, two +-- objects deep into the second row, and MPR008 reported only the one of the two +-- overlaps that was activity on activity. +create or modify microflow BugTest1768.SUB_TwoHandlersLonger ($Url: String) returns BugTest1768.Result as $Result +begin + $First = call rest service get '{1}' with ({1} = $Url) (Timeout: 30) returns response on error without rollback begin + $FirstFailed = create BugTest1768.Result (Code = 1); + log error node 'BugTest1768' 'first failed'; + log info node 'BugTest1768' 'first failed, details'; + return $FirstFailed; + end error; + declare $Body String = $First/Content; + $Second = call rest service get '{1}' with ({1} = $Body) (Timeout: 30) returns response on error without rollback begin + $SecondFailed = create BugTest1768.Result (Code = 2); + log error node 'BugTest1768' 'second failed'; + return $SecondFailed; + end error; + $Done = create BugTest1768.Result (IsDone = true); + return $Done; +end; diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 00aa4dcb7..f8691d94d 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -1015,6 +1015,10 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in fb.returnEndIDs[id] = true } + // The row now occupies the lane under this activity's line, so the next + // statement that reaches into that lane waits for it (ako/mxcli#1768). + fb.reserveHandlerRow(sourceX, errorBody) + // If the error handler ends with RAISE ERROR or RETURN, it terminates there. // Otherwise, return the last activity ID so caller can create a merge. if errBuilder.endsWithReturn { diff --git a/mdl/executor/layout.go b/mdl/executor/layout.go index 1431cbe1f..55e2ec459 100644 --- a/mdl/executor/layout.go +++ b/mdl/executor/layout.go @@ -98,6 +98,14 @@ func (m *layoutMeasurer) measureStatements(stmts []ast.MicroflowStatement) Bound } else { cursor = left + bounds.Width } + if row := m.handlerRowWidth(stmt); row > 0 { + // The handler's row hangs in the same lane, from the activity's left + // edge, and the main line goes on above it — as reserveHandlerRow + // records it for the builder. + rowRight := left + row + laneBusy = max(laneBusy, rowRight+laneGap) + extent = max(extent, rowRight) + } extent = max(extent, cursor) prev = stmt } diff --git a/mdl/executor/layout_handler_lane_1768_test.go b/mdl/executor/layout_handler_lane_1768_test.go new file mode 100644 index 000000000..11d7a7313 --- /dev/null +++ b/mdl/executor/layout_handler_lane_1768_test.go @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// A custom error handler is drawn as a row under its activity, ErrorHandlerPitch +// below the main line. The main line does not wait for that row — the statement +// after the activity is one pitch further on, however long the row is — so when +// that next statement had a handler too, its row started inside the first one +// (ako/mxcli#1768, mendixlabs/mxcli#1672). Reported, from `describe … with +// handles`: the first handler's end event and the second handler's first +// activity both on 680;330. +const twoHandlersFlow = `create microflow MyModule.SUB_TwoHandlers ($Url: String) returns MyModule.Result as $Result +begin + $First = call rest service get '{1}' with ({1} = $Url) (Timeout: 30) returns response on error without rollback begin + $FirstFailed = create MyModule.Result (Code = 1); + log error node 'MyModule' 'first failed'; + return $FirstFailed; + end error; + declare $Body String = $First/Content; + $Second = call rest service get '{1}' with ({1} = $Body) (Timeout: 30) returns response on error without rollback begin + $SecondFailed = create MyModule.Result (Code = 2); + log error node 'MyModule' 'second failed'; + return $SecondFailed; + end error; + $Done = create MyModule.Result (IsDone = true); + return $Done; +end;` + +// canvasBoxOf is an object's box on its canvas: a Mendix position is the centre. +func canvasBoxOf(o microflows.MicroflowObject) canvasBox { + p, sz := o.GetPosition(), objectSize(o) + return canvasBox{p.X - sz.Width/2, p.Y - sz.Height/2, p.X + sz.Width/2, p.Y + sz.Height/2} +} + +func canvasLabel(o microflows.MicroflowObject) string { + p := o.GetPosition() + return fmt.Sprintf("%s %d;%d", typeShort(o), p.X, p.Y) +} + +// canvasOverlaps lists every pair of objects that share space on the same canvas. +// A loop's children are positioned in the loop's own frame, so each loop body is +// its own canvas, checked separately; notes are left out. +func canvasOverlaps(objects []microflows.MicroflowObject) []string { + var out []string + var plane []microflows.MicroflowObject + for _, o := range objects { + if _, ok := o.(*microflows.Annotation); ok { + continue + } + plane = append(plane, o) + if loop, ok := o.(*microflows.LoopedActivity); ok && loop.ObjectCollection != nil { + for _, inner := range canvasOverlaps(loop.ObjectCollection.Objects) { + out = append(out, "in loop: "+inner) + } + } + } + for i := range plane { + for j := i + 1; j < len(plane); j++ { + a, b := canvasBoxOf(plane[i]), canvasBoxOf(plane[j]) + if a.left < b.right && b.left < a.right && a.top < b.bottom && b.top < a.bottom { + out = append(out, canvasLabel(plane[i])+" overlaps "+canvasLabel(plane[j])) + } + } + } + return out +} + +func TestSecondErrorHandlerRowClearsTheFirst(t *testing.T) { + oc := buildLayout(t, twoHandlersFlow) + if overlaps := canvasOverlaps(oc.Objects); len(overlaps) > 0 { + t.Errorf("objects share a place:\n %s", strings.Join(overlaps, "\n ")) + } + + got := map[string]bool{} + for _, o := range oc.Objects { + got[canvasLabel(o)] = true + } + // Everything up to the second handler stays where it was: the first activity, + // its row, and the plain activity after it, which does not reach into the lane + // and so does not wait for it. + for _, want := range []string{ + "ActionActivity 360;200", // $First + "ActionActivity 360;330", // $FirstFailed + "ActionActivity 520;330", // log error 'first failed' + "EndEvent 680;330", // return $FirstFailed + "ActionActivity 520;200", // $Body + } { + if !got[want] { + t.Errorf("missing %s — a position that did not overlap anything moved", want) + } + } + // $Second reaches into the lane with its own row, so it waits for the first row + // to clear: that row ends at 690 (the end event's right edge), plus the 40px gap + // two activities keep, plus half an activity. The literal is the measurement, + // not the builder's constants, so it cannot agree with whatever they say. + for _, want := range []string{ + "ActionActivity 790;200", // $Second + "ActionActivity 790;330", // $SecondFailed, straight under its activity + } { + if !got[want] { + t.Errorf("missing %s", want) + } + } +} + +// handlerRuns are runs of activities whose error handler rows share a lane. The +// builder places a handler row the same way wherever the run is — on the main +// line, in a loop body, in either branch of a decision — so each is checked in +// all of them. +var handlerRuns = []struct { + name, body string + // rejoins: the handlers fall through to the main line instead of ending. + rejoins bool +}{ + {name: "two handlers with a plain activity between", body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + return; + end error; + log info node 'M' 'between'; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + return; + end error; + log info node 'M' 'done';`}, + {name: "the first handler longer", body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + log error node 'M' 'a3'; + return; + end error; + log info node 'M' 'between'; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + return; + end error; + log info node 'M' 'done';`}, + {name: "three handlers back to back", body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + return; + end error; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + return; + end error; + retrieve $C from M.Item on error without rollback begin + log error node 'M' 'c1'; + return; + end error; + log info node 'M' 'done';`}, + {name: "handlers that rejoin the main line", rejoins: true, body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + end error; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + end error; + log info node 'M' 'done';`}, +} + +// lanePartners put a handler row next to the lane's other occupants: a guard's +// branch, a decision's ELSE row, a handler row inside a decision's branch. +// +// They are checked where the builder places the statement after the decision +// itself — the main line and a loop body. Inside a branch the enclosing merge is +// placed from the measured width of that decision, and the measurer counts a +// split's x as its left edge, so every decision is measured 45px short whether +// or not a handler is involved: `if … else … end if; log …` in a THEN branch puts +// the closing merge 5px into that log. That is its own defect, not this one. +var lanePartners = []struct{ name, body string }{ + {"a guard after a handler", ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + return; + end error; + if $Flag then + log info node 'M' 'refused'; + return; + end if; + log info node 'M' 'done';`}, + {"a decision after a handler", ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + return; + end error; + if $Flag then + log info node 'M' 'yes'; + else + log info node 'M' 'no'; + end if; + log info node 'M' 'done';`}, + {"a handler after a guard", ` + if $Flag then + log info node 'M' 'refused'; + log info node 'M' 'refused, details'; + return; + end if; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + return; + end error; + log info node 'M' 'done';`}, + {"a handler row in a branch, a handler after the decision", ` + if $Flag then + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + log error node 'M' 'a3'; + return; + end error; + else + log info node 'M' 'no'; + end if; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + return; + end error; + log info node 'M' 'done';`}, +} + +type shapeContext struct { + name string + wrap func(body string) string +} + +var ( + onMainLine = shapeContext{"main line", func(b string) string { return b }} + inLoopBody = shapeContext{"loop body", func(b string) string { + return "loop $It in $Items begin\n" + b + "\nend loop;\nlog info node 'M' 'after';" + }} + inThenBranch = shapeContext{"then branch", func(b string) string { + return "if $Other then\n" + b + "\nelse\n log info node 'M' 'else';\nend if;\nlog info node 'M' 'after';" + }} + inElseBranch = shapeContext{"else branch", func(b string) string { + return "if $Other then\n log info node 'M' 'then';\nelse\n" + b + "\nend if;\nlog info node 'M' 'after';" + }} + inThenWithoutElse = shapeContext{"then branch without else", func(b string) string { + return "if $Other then\n" + b + "\nend if;\nlog info node 'M' 'after';" + }} +) + +func handlerShapeFlow(body string) string { + return "create microflow M.F ($Flag: Boolean, $Other: Boolean, $Items: list of M.Item) begin\n" + body + "\nend;" +} + +// assertNoSharedPlace builds a flow and checks that no two of its objects share a +// place on any canvas. The builder places each statement and the measurer sizes +// the branch or loop it sits in; when the two disagree about where a statement +// went, a merge lands on it, which this catches too. +func assertNoSharedPlace(t *testing.T, src string, checkMergeDirection bool) { + t.Helper() + oc := buildLayout(t, src) + if overlaps := canvasOverlaps(oc.Objects); len(overlaps) > 0 { + t.Errorf("objects share a place:\n %s", strings.Join(overlaps, "\n ")) + } + if checkMergeDirection { + assertLayoutInvariants(t, t.Name(), oc) + } +} + +func TestErrorHandlerRowsNeverOverlap(t *testing.T) { + for _, run := range handlerRuns { + for _, ctx := range []shapeContext{onMainLine, inLoopBody, inThenBranch, inElseBranch, inThenWithoutElse} { + t.Run(run.name+" / "+ctx.name, func(t *testing.T) { + // A handler that falls through rejoins at a merge placed between its + // activity and the next one, left of a row of two or more: a + // backward edge, but not a shared place, and older than this test. + assertNoSharedPlace(t, handlerShapeFlow(ctx.wrap(run.body)), !run.rejoins) + }) + } + } + for _, shape := range lanePartners { + for _, ctx := range []shapeContext{onMainLine, inLoopBody} { + t.Run(shape.name+" / "+ctx.name, func(t *testing.T) { + assertNoSharedPlace(t, handlerShapeFlow(ctx.wrap(shape.body)), true) + }) + } + } +} + +// The measurer sizes a container before its body exists; the builder sizes the +// box from what it built, handler rows included. A loop around each run must come +// out exactly as wide as it was measured, or an IF holding the loop places its +// merge inside the box (the mendixlabs/mxcli#1598 class) — and once a handler row +// makes the next one wait, the measurer has to know about the wait as well as the +// builder. +func TestLoopAroundHandlerRowsIsMeasuredAsBuilt(t *testing.T) { + for _, run := range handlerRuns { + t.Run(run.name, func(t *testing.T) { + prog, errs := visitor.Build(handlerShapeFlow("loop $It in $Items begin\n" + run.body + "\nend loop;")) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + var loopStmt *ast.LoopStmt + for _, s := range mf.Body { + if l, ok := s.(*ast.LoopStmt); ok { + loopStmt = l + } + } + measured := (&layoutMeasurer{}).measureStatement(loopStmt) + + fb := &flowBuilder{posX: 200, posY: 200, spacing: HorizontalSpacing} + oc := fb.buildFlowGraph(mf.Body, mf.ReturnType) + var built model.Size + for _, o := range oc.Objects { + if l, ok := o.(*microflows.LoopedActivity); ok { + built = l.Size + } + } + if built.Width != measured.Width { + t.Errorf("loop built %dpx wide, measured %dpx", built.Width, measured.Width) + } + }) + } +} diff --git a/mdl/executor/layout_lanes.go b/mdl/executor/layout_lanes.go index 827efc98b..50edbdd59 100644 --- a/mdl/executor/layout_lanes.go +++ b/mdl/executor/layout_lanes.go @@ -13,6 +13,9 @@ // remembers how far it is taken. Only an element that reaches down into that lane — // another decision, a loop box, an activity with an error handler under it — has to // wait for it to clear; a plain activity does not. +// +// An error handler's row is the lane's other occupant: it hangs ErrorHandlerPitch +// under its activity and the main line carries on over it the same way. package executor import "github.com/mendixlabs/mxcli/mdl/ast" @@ -50,6 +53,37 @@ func reachesBelowMainLine(stmt ast.MicroflowStatement, bounds Bounds) bool { return len(getErrorHandlerBody(stmt)) > 0 } +// handlerRowWidth is how wide the row of a statement's custom error handler is, +// measured from the activity's left edge, or 0 when it has no handler body. The +// row's first element is centred under the activity (addErrorHandlerFlow), so +// the row starts where the activity does and runs right like a branch, end +// event included. +func (m *layoutMeasurer) handlerRowWidth(stmt ast.MicroflowStatement) int { + body := getErrorHandlerBody(stmt) + if len(body) == 0 { + return 0 + } + return m.measureBranch(body).Width +} + +// reserveHandlerRow marks the lane under the activity at activityX taken by its +// error handler's row (ako/mxcli#1768, mendixlabs/mxcli#1672). +// +// The row is laid out ErrorHandlerPitch under its activity and runs right, and the +// main line goes on one pitch after the activity regardless — which is right for a +// plain activity, whose row is above the handler's. But the next activity with a +// handler of its own started its row at its own x, inside this one: two consecutive +// REST calls each with a three-element handler put the first handler's end event +// and the second handler's first activity on the same point. clearLowerLane already +// made such a statement wait for a reserved lane; the row just never reserved it. +func (fb *flowBuilder) reserveHandlerRow(activityX int, body []ast.MicroflowStatement) { + if fb.measurer == nil || len(body) == 0 { + return + } + rowRight := activityX - ActivityWidth/2 + fb.measurer.measureBranch(body).Width + fb.reserveLowerLane(fb.posY, rowRight+laneGap) +} + // leftHalfOf is the distance from a statement's x to its left edge, as gapBetween // counts it. func leftHalfOf(stmt ast.MicroflowStatement) int { From 34d37878ca2922a62d23bfd9c0d4f254a2723063 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:28:19 +0000 Subject: [PATCH 31/61] fix(layout): a later @position no longer moves the statements before it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The start event goes one spacing left of the first statement when that statement carries @position. The pre-scan took the first statement that had one, wherever it was, and every unplaced statement before it follows the start: `declare $A …; @position(840, 200) declare $B …` stored both at 840;200 (ako/mxcli#1768 case 2, mendixlabs/mxcli#1672). Only the first statement moves the start now, as alter's patch (patchDiff.startEvent, declared[0]) and the syntax help already assumed. The help and the language reference now say that statements before a placed one keep their derived places. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- CHANGELOG.md | 1 + cmd/mxcli/syntax/features_microflow.go | 6 +- docs-site/src/language/microflow-structure.md | 8 ++- .../1768-error-handler-rows-share-a-lane.mdl | 16 ++++- mdl/executor/cmd_microflows_builder_graph.go | 19 ++++-- .../layout_position_later_1768_test.go | 68 +++++++++++++++++++ 6 files changed, 106 insertions(+), 12 deletions(-) create mode 100644 mdl/executor/layout_position_later_1768_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d08a4181..8812b9df0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. +- **A `@position` on a later statement no longer moves the statements before it onto it** (ako/mxcli#1768, mendixlabs/mxcli#1672) — `declare $A String = $Url; @position(840, 200) declare $B String = $A;` stored both declares at 840;200, and MPR008 reported the overlap. The start event is placed one spacing left of the first statement when that statement carries `@position`, but the builder took the first statement that had one, wherever it was, and the unplaced statements before it follow the start. Only the first statement moves the start now, as `alter`'s patch and the syntax help already assumed: the flow above starts at 200;200 with `$A` at 360;200 and `$B` at 840;200. The syntax help and the language reference now say what happens to the statements before a placed one. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index ae96ed08c..505b2bcde 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -938,8 +938,10 @@ func init() { "does not (ako/mxcli#1768). A CASE of four or more branches leaves the split\n" + "in three groups — top, right, bottom — so its lines do not cross. A\n" + "statement that carries @position is never moved, and starts the row for what\n" + - "follows it. Prefer no @position at all to a few: hand-placed statements are\n" + - "not measured against what the builder puts around them. (#1154)\n\n" + + "follows it; the statements BEFORE it keep the places the builder derives for\n" + + "them, and only a @position on the first statement moves the start event.\n" + + "Prefer no @position at all to a few: hand-placed statements are not measured\n" + + "against what the builder puts around them. (#1154)\n\n" + "@start and @merge position the two nodes that have no statement of their\n" + "own, so each is written on the statement it belongs to. Omit @start and the\n" + "start is placed one spacing unit left of the first activity, on its centre\n" + diff --git a/docs-site/src/language/microflow-structure.md b/docs-site/src/language/microflow-structure.md index 71eb46801..1eca953c2 100644 --- a/docs-site/src/language/microflow-structure.md +++ b/docs-site/src/language/microflow-structure.md @@ -158,9 +158,11 @@ leaves the decision in three groups (top, right, bottom) so its lines do not cro Whatever reaches down into the lane — a decision, a loop, an activity with an error handler of its own — waits for the lane to clear, so two consecutive handlers never share a row; a plain activity does not wait. A statement that carries `@position` is -never moved, and becomes the start of the row for the statements after it — so either -place everything or nothing: a few hand-placed statements are not measured against -what is laid out around them. +never moved, and becomes the start of the row for the statements after it; the +statements before it keep the places the builder derives for them, and only a +`@position` on the first statement moves the start event. So either place everything +or nothing: a few hand-placed statements are not measured against what is laid out +around them. To re-arrange a flow that already exists — one drawn in Studio Pro, or one whose positions no longer fit after edits — run `mxcli layout flows`. It applies this diff --git a/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl index bf28982c1..5d064f002 100644 --- a/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl +++ b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl @@ -19,7 +19,13 @@ -- 360..680;330, 520;200) and places $Second at 790;200 with its row at -- 790..1110;330 — clear of the first row's end, with the gap two activities -- keep. `lint -r MPR008` reports nothing, and `layout flows --dry-run` calls --- every flow below already laid out. +-- every flow below already laid out except SUB_PositionedLater (layout flows +-- drops its @position). +-- +-- SUB_PositionedLater is the issue's second case: a @position on a later +-- statement used to move the start event to it and every unplaced statement +-- before it along with the start, so `declare $A` and `declare $B` were both +-- stored at 840;200. Expected now: start 200;200, $A 360;200, $B 840;200. mdl 1; @@ -64,3 +70,11 @@ begin $Done = create BugTest1768.Result (IsDone = true); return $Done; end; + +create or modify microflow BugTest1768.SUB_PositionedLater ($Url: String) returns String as $B +begin + declare $A String = $Url; + @position(840, 200) + declare $B String = $A; + return $B; +end; diff --git a/mdl/executor/cmd_microflows_builder_graph.go b/mdl/executor/cmd_microflows_builder_graph.go index 1a4ca5122..b8d681a81 100644 --- a/mdl/executor/cmd_microflows_builder_graph.go +++ b/mdl/executor/cmd_microflows_builder_graph.go @@ -36,15 +36,22 @@ func (fb *flowBuilder) buildFlowGraph(stmts []ast.MicroflowStatement, returns *a // Set baseY for branch restoration (this is the center line) fb.baseY = fb.posY - // Pre-scan: if the first statement carries an @position annotation, shift the - // StartEvent to be one spacing unit to the left of that position so it doesn't - // end up behind activities that use explicit coordinates. - for _, stmt := range stmts { - if ann := getStatementAnnotations(stmt); ann != nil && ann.Position != nil { + // If the first statement carries an @position annotation, shift the StartEvent + // to be one spacing unit to the left of that position so it doesn't end up + // behind activities that use explicit coordinates. + // + // The first statement only. This used to take the first statement that had an + // @position, wherever it was, so a position on a later statement moved the + // start — and every unplaced statement before it, which follows the start — to + // that statement's place: `declare $A …; @position(840, 200) declare $B …` + // stored both at 840;200 (ako/mxcli#1768). Statements before a placed one keep + // the places the builder derives for them, as alter's patch already assumed + // (patchDiff.startEvent reads declared[0]). + if len(stmts) > 0 { + if ann := getStatementAnnotations(stmts[0]); ann != nil && ann.Position != nil { fb.posX = ann.Position.X - fb.spacing fb.posY = ann.Position.Y fb.baseY = fb.posY - break } } diff --git a/mdl/executor/layout_position_later_1768_test.go b/mdl/executor/layout_position_later_1768_test.go new file mode 100644 index 000000000..1e3e9f7d5 --- /dev/null +++ b/mdl/executor/layout_position_later_1768_test.go @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// A statement placed with @position starts the row for what follows it, and the +// start event sits one spacing left of the first statement when THAT one is +// placed. The builder found "the first statement with an @position" instead of +// looking at the first statement, so a position written on a later statement +// moved the start event to it — and every unplaced statement before it along +// with the start, onto the placed one (ako/mxcli#1768 case 2, mendixlabs/mxcli#1672). +// Reported: `describe … with handles` printed @position(840, 200) for both +// `declare $A` and `declare $B`, and MPR008 reported the overlap. +func TestStatementsBeforeAPlacedOneKeepTheirDerivedPlaces(t *testing.T) { + oc := buildLayout(t, `create microflow MyModule.SUB_PositionedLater ($Url: String) returns String as $B +begin + declare $A String = $Url; + @position(840, 200) + declare $B String = $A; + return $B; +end;`) + if overlaps := canvasOverlaps(oc.Objects); len(overlaps) > 0 { + t.Errorf("objects share a place:\n %s", strings.Join(overlaps, "\n ")) + } + got := map[string]bool{} + for _, o := range oc.Objects { + got[canvasLabel(o)] = true + } + for _, want := range []string{ + "StartEvent 200;200", // where a flow with no placed first statement starts + "ActionActivity 360;200", // declare $A: one spacing after the start + "ActionActivity 840;200", // declare $B: where the script put it + "EndEvent 1000;200", // one spacing after $B + } { + if !got[want] { + t.Errorf("missing %s", want) + } + } +} + +// Control for the rule above: a placed FIRST statement still carries the start +// event with it, one spacing to its left on its centre line. +func TestPlacedFirstStatementStillCarriesTheStartEvent(t *testing.T) { + oc := buildLayout(t, `create microflow MyModule.SUB_PositionedFirst ($Url: String) returns String as $B +begin + @position(840, 300) + declare $A String = $Url; + declare $B String = $A; + return $B; +end;`) + got := map[string]bool{} + for _, o := range oc.Objects { + got[canvasLabel(o)] = true + } + for _, want := range []string{ + "StartEvent 680;300", + "ActionActivity 840;300", + "ActionActivity 1000;300", + } { + if !got[want] { + t.Errorf("missing %s", want) + } + } +} From 49dec833056678d54f07c5290d653f43363ef1cd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:28:31 +0000 Subject: [PATCH 32/61] fix(lint): MPR008 compares events and type splits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The overlap rule built its canvas from activities, loops, decisions and merges only, so an activity lying on an error handler's end event was never reported — the shape ako/mxcli#1768 reported, two handler rows on top of each other, passed MPR008 until an activity happened to land on an activity. Start, end, error, break and continue events and inheritance splits are on the canvas too, each judged with its stored size; an event that stores none is taken as Studio Pro's 20x20 rather than as an activity. On the 292 Studio Pro-authored flows of PedApp and ako/TestApp the wider set adds no finding. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- CHANGELOG.md | 1 + .../rules/mpr008_overlapping_activities.go | 26 +++++++++ .../mpr008_overlapping_activities_test.go | 57 +++++++++++++++++++ 3 files changed, 84 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8812b9df0..eafed1b56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. - **A `@position` on a later statement no longer moves the statements before it onto it** (ako/mxcli#1768, mendixlabs/mxcli#1672) — `declare $A String = $Url; @position(840, 200) declare $B String = $A;` stored both declares at 840;200, and MPR008 reported the overlap. The start event is placed one spacing left of the first statement when that statement carries `@position`, but the builder took the first statement that had one, wherever it was, and the unplaced statements before it follow the start. Only the first statement moves the start now, as `alter`'s patch and the syntax help already assumed: the flow above starts at 200;200 with `$A` at 360;200 and `$B` at 840;200. The syntax help and the language reference now say what happens to the statements before a placed one. +- **MPR008 reports an activity drawn on an event** (ako/mxcli#1768, mendixlabs/mxcli#1672) — the overlap rule compared activities, loops, decisions and merges only, so an error handler's activity lying on another handler's end event at 680;330 passed, and with one more statement in that handler it reported one of the two overlaps, the one that happened to be activity on activity. Start, end, error, break and continue events and type splits are compared too, each with its stored size (an event that stores none is taken as Studio Pro's 20×20, not as an activity). On the 292 flows of the Studio Pro-authored PedApp and ako/TestApp fixtures, and on 182 flows built from the `02*` and `06` doctype examples, the wider rule adds no finding. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/mdl/linter/rules/mpr008_overlapping_activities.go b/mdl/linter/rules/mpr008_overlapping_activities.go index 7b0f7152c..3e3da4ddd 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities.go +++ b/mdl/linter/rules/mpr008_overlapping_activities.go @@ -15,6 +15,11 @@ import ( const activityBoxWidth = 120 const activityBoxHeight = 60 +// eventSize is the side of an event's box (start, end, error, break, continue) +// for one that stores no size: Studio Pro stores 20x20 on every event it draws +// (all 712 in ako/TestApp and PedApp), and so does mxcli. +const eventSize = 20 + // OverlappingActivitiesRule flags microflow activities whose canvas positions overlap. // // The most common cause is writing multiple MDL statements after a single @position @@ -124,6 +129,11 @@ func boxesOverlap(a, b actInfo) bool { func newActInfo(obj microflows.MicroflowObject, caption string) actInfo { p := obj.GetPosition() w, h := activityBoxWidth, activityBoxHeight + switch obj.(type) { + case *microflows.StartEvent, *microflows.EndEvent, *microflows.ErrorEvent, + *microflows.BreakEvent, *microflows.ContinueEvent: + w, h = eventSize, eventSize + } if s, ok := obj.(interface{ GetSize() model.Size }); ok { if sz := s.GetSize(); sz.Width > 0 && sz.Height > 0 { w, h = sz.Width, sz.Height @@ -170,8 +180,24 @@ func overlapPlanes(objects []microflows.MicroflowObject) [][]actInfo { } case *microflows.ExclusiveSplit: plane = append(plane, newActInfo(act, act.Caption)) + case *microflows.InheritanceSplit: + plane = append(plane, newActInfo(act, act.Caption)) case *microflows.ExclusiveMerge: plane = append(plane, newActInfo(act, "(merge)")) + // Events are nodes on the canvas too. Leaving them out hid an + // activity drawn on a handler's end event: two error handler rows + // laid on top of each other were reported only when an activity + // happened to land on an activity (ako/mxcli#1768). + case *microflows.StartEvent: + plane = append(plane, newActInfo(act, "(start)")) + case *microflows.EndEvent: + plane = append(plane, newActInfo(act, "(end)")) + case *microflows.ErrorEvent: + plane = append(plane, newActInfo(act, "(error)")) + case *microflows.BreakEvent: + plane = append(plane, newActInfo(act, "(break)")) + case *microflows.ContinueEvent: + plane = append(plane, newActInfo(act, "(continue)")) } } planes = append(planes, plane) diff --git a/mdl/linter/rules/mpr008_overlapping_activities_test.go b/mdl/linter/rules/mpr008_overlapping_activities_test.go index 20df3a146..3d9751308 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities_test.go +++ b/mdl/linter/rules/mpr008_overlapping_activities_test.go @@ -147,3 +147,60 @@ func TestOverlapUsesStoredSizes(t *testing.T) { }) } } + +// ako/mxcli#1768: two error handler rows laid on top of each other put the first +// handler's end event and the second handler's first activity both on 680;330. +// MPR008 compared activities, splits and merges only, so it reported nothing — +// and with one more statement in the first handler it reported one of the two +// overlaps, the one that happened to be activity on activity. Events are nodes +// on the canvas and are compared too, with their own 20x20 box. +func TestOverlapComparesEvents(t *testing.T) { + act := func(x, y int) *microflows.ActionActivity { + a := µflows.ActionActivity{} + a.Position = model.Point{X: x, Y: y} + a.Size = model.Size{Width: 120, Height: 60} + return a + } + end := func(x, y int, sized bool) *microflows.EndEvent { + e := µflows.EndEvent{} + e.Position = model.Point{X: x, Y: y} + if sized { + e.Size = model.Size{Width: 20, Height: 20} + } + return e + } + for _, tc := range []struct { + name string + objs []microflows.MicroflowObject + want bool + }{ + {"end event on an activity (the report)", []microflows.MicroflowObject{end(680, 330, true), act(680, 330)}, true}, + {"end event one pitch after an activity", []microflows.MicroflowObject{act(520, 330), end(680, 330, true)}, false}, + // An event that stores no size is an event, not an activity: judged as a + // 120x60 box it would overlap an activity 80px away. + {"unsized end event 80px from an activity", []microflows.MicroflowObject{act(600, 200), end(680, 200, false)}, false}, + {"unsized end event inside an activity", []microflows.MicroflowObject{act(600, 200), end(640, 200, false)}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + p := overlapPlanes(tc.objs)[0] + if len(p) != 2 { + t.Fatalf("got %d nodes on the canvas, want 2 — the event was left out", len(p)) + } + if got := boxesOverlap(p[0], p[1]); got != tc.want { + t.Errorf("overlap = %v, want %v (%+v, %+v)", got, tc.want, p[0], p[1]) + } + }) + } + + // Every kind of event, and a type split, is on the canvas. + nodes := []microflows.MicroflowObject{ + µflows.StartEvent{}, µflows.EndEvent{}, µflows.ErrorEvent{}, + µflows.BreakEvent{}, µflows.ContinueEvent{}, µflows.InheritanceSplit{}, + } + for i, n := range nodes { + n.SetPosition(model.Point{X: 100 + 200*i, Y: 200}) + } + if got := len(overlapPlanes(nodes)[0]); got != len(nodes) { + t.Errorf("%d of %d nodes are on the canvas", got, len(nodes)) + } +} From 83f22ae7cacaa6b7ec9f209439c234c14261b28d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:16:47 +0000 Subject: [PATCH 33/61] fix(executor): every nested flow body keeps the splice's member guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom (ako/mxcli#1769, mendixlabs/mxcli#1673, Mendix 11.12.4): `alter microflow … { replace loop … with begin … end; }` wrote `$X = find $List by Code = …` in the loop's body as Microflows$FindByExpression with the bare text `Code = …` when the fragment did not know $List's entity. check --references and exec reported nothing; mx check failed CE0117. The same find inserted on the main line was refused (ako/mxcli#891). Cause: confirmed. #891's qualifiedMembersOnly is set on the fragment's builder, but the loop body, the while body and the error-handler body are each built by a fresh flowBuilder whose fields the site listed by hand, and none listed the flag. So every member-bearing statement nested in a fragment (find, filter, sort, aggregate, change, at any depth) was written bare. `replace body of loop` went through the same loop builder and had the same hole. Fix: one construction helper, flowBuilder.nestedBuilder, for all three sites; it inherits the flag with the rest of the shared state. The error handler still sets its return type and label table itself, as before. TestNestedBuilder_InheritsSharedState classifies every flowBuilder field as inherited or the body's own and checks the helper against it by reflection, so the next field cannot be left out silently. With this alone the reported alter is refused, with nothing written, because the fragment does not type a list retrieved over an association; the next commit types it. Control: with the flag left out of nestedBuilder, the nested-guard test reports every statement in every nested shape written instead of refused, the PedApp test reports exec writing the replaced loop and replace body, and the structural test names the missing field. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- mdl/executor/cmd_microflows_builder.go | 54 ++++++ .../cmd_microflows_builder_control.go | 70 +------ mdl/executor/cmd_microflows_builder_flows.go | 43 ++--- .../cmd_microflows_builder_nested_test.go | 172 ++++++++++++++++++ ...ow_splice_nested_guard_1769_pedapp_test.go | 124 +++++++++++++ mdl/executor/flow_splice_nested_guard_test.go | 131 +++++++++++++ 6 files changed, 502 insertions(+), 92 deletions(-) create mode 100644 mdl/executor/cmd_microflows_builder_nested_test.go create mode 100644 mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go create mode 100644 mdl/executor/flow_splice_nested_guard_test.go diff --git a/mdl/executor/cmd_microflows_builder.go b/mdl/executor/cmd_microflows_builder.go index 4862606c1..8533b4d69 100644 --- a/mdl/executor/cmd_microflows_builder.go +++ b/mdl/executor/cmd_microflows_builder.go @@ -200,6 +200,60 @@ type flowBuilder struct { joinScopes map[string]flowBuilderVariableState } +// nestedBuilder returns the builder for a body nested in this one — a loop's, +// a while loop's, an error handler's — whose first activity is centred at +// (x, y). The body has its own position, graph and pending state, and +// inherits what says which flow is being built and how its statements are +// written: the variable scope, the backend and project lookups, the language, +// the notes, and the guards. Every nested body is built here, so a field is +// inherited by all of them or by none — bar what the one site that needs it +// sets (an error handler's return type and label table) — and +// TestNestedBuilder_InheritsSharedState fails until a new field is put on one +// side or the other. +// +// ako/mxcli#1769: each site listed the fields it copied, and none listed +// qualifiedMembersOnly, so a fragment spliced into a stored flow refused an +// unqualifiable member on its main line and wrote it in every body nested in +// it — `find $L by Code = …` in a replaced loop became a find by the invalid +// expression `Code = …` (CE0117). +func (fb *flowBuilder) nestedBuilder(x, y int) *flowBuilder { + return &flowBuilder{ + posX: x, + posY: y, + baseY: y, + spacing: HorizontalSpacing, + + varTypes: fb.varTypes, // one variable scope + declaredVars: fb.declaredVars, // (fixes a nil map panic) + textLang: fb.textLang, + measurer: fb.measurer, + backend: fb.backend, + quiet: fb.quiet, + hierarchy: fb.hierarchy, + restServices: fb.restServices, // for parameter classification + isNanoflow: fb.isNanoflow, + self: fb.self, + scriptFlows: fb.scriptFlows, + // The note registry, so a note declared outside the body and + // referenced on an activity in it attaches to the SAME Annotation + // rather than being refused. The describer emits exactly that (its + // label state is shared across the loop overlay), so a builder that + // refused it would reject its own DESCRIBE output. + // + // Across a loop boundary the result — Annotation in the parent + // collection, both AnnotationFlows in the parent collection, one + // destination inside the loop — was measured at 0 errors on mxbuild + // 11.13. mxbuild is the only oracle available here; Studio Pro is + // stricter in general, though this is the same split mxcli already + // ships for an ordinary loop-body note, whose flow is hoisted to the + // parent. (#1077) + annotationsByLabel: fb.annotationsByLabel, + storedNotes: fb.storedNotes, + // A body of a spliced fragment is spliced too (ako/mxcli#1769). + qualifiedMembersOnly: fb.qualifiedMembersOnly, + } +} + type flowBuilderVariableState struct { varTypes map[string]string declaredVars map[string]string diff --git a/mdl/executor/cmd_microflows_builder_control.go b/mdl/executor/cmd_microflows_builder_control.go index c5d9e66a7..73f65951e 100644 --- a/mdl/executor/cmd_microflows_builder_control.go +++ b/mdl/executor/cmd_microflows_builder_control.go @@ -711,38 +711,11 @@ func (fb *flowBuilder) addLoopStatement(s *ast.LoopStmt) model.ID { } } - // Build nested ObjectCollection for loop body - loopBuilder := &flowBuilder{ - textLang: fb.textLang, - posX: innerStartX, - posY: innerStartY, - baseY: innerStartY, - spacing: HorizontalSpacing, - varTypes: fb.varTypes, // Share variable scope - declaredVars: fb.declaredVars, // Share declared vars (fixes nil map panic) - measurer: fb.measurer, // Share measurer - backend: fb.backend, // Share backend - quiet: fb.quiet, - hierarchy: fb.hierarchy, // Share hierarchy - restServices: fb.restServices, // Share REST services for parameter classification - isNanoflow: fb.isNanoflow, - self: fb.self, - scriptFlows: fb.scriptFlows, - // Share the note registry, so a note declared outside the loop and - // referenced on a body activity attaches to the SAME Annotation rather - // than being refused. The describer emits exactly that (its label state - // is shared across the loop overlay), so a builder that refused it would - // reject its own DESCRIBE output. - // - // The resulting shape — Annotation in the parent collection, both - // AnnotationFlows in the parent collection, one destination inside the - // loop — was measured at 0 errors on mxbuild 11.13. mxbuild is the only - // oracle available here; Studio Pro is stricter in general, though this - // is the same split mxcli already ships for an ordinary loop-body note, - // whose flow is hoisted to the parent a few lines below. (#1077) - annotationsByLabel: fb.annotationsByLabel, - storedNotes: fb.storedNotes, - } + // Build nested ObjectCollection for loop body. It shares the variable + // scope, the note registry and the guards (nestedBuilder), and not the + // label table: the LoopedActivity owns a collection a sequence flow may + // not leave. + loopBuilder := fb.nestedBuilder(innerStartX, innerStartY) // Process loop body statements and connect them with flows. // pendingCase carries the deferred case value a merge-less split leaves for @@ -1092,37 +1065,8 @@ func (fb *flowBuilder) addWhileStatement(s *ast.WhileStmt) model.ID { loopLeftX = loopCenterX - loopWidth/2 } - loopBuilder := &flowBuilder{ - textLang: fb.textLang, - posX: innerStartX, - posY: innerStartY, - baseY: innerStartY, - spacing: HorizontalSpacing, - varTypes: fb.varTypes, - declaredVars: fb.declaredVars, - measurer: fb.measurer, - backend: fb.backend, - quiet: fb.quiet, - hierarchy: fb.hierarchy, - restServices: fb.restServices, - isNanoflow: fb.isNanoflow, - self: fb.self, - scriptFlows: fb.scriptFlows, - // Share the note registry, so a note declared outside the loop and - // referenced on a body activity attaches to the SAME Annotation rather - // than being refused. The describer emits exactly that (its label state - // is shared across the loop overlay), so a builder that refused it would - // reject its own DESCRIBE output. - // - // The resulting shape — Annotation in the parent collection, both - // AnnotationFlows in the parent collection, one destination inside the - // loop — was measured at 0 errors on mxbuild 11.13. mxbuild is the only - // oracle available here; Studio Pro is stricter in general, though this - // is the same split mxcli already ships for an ordinary loop-body note, - // whose flow is hoisted to the parent a few lines below. (#1077) - annotationsByLabel: fb.annotationsByLabel, - storedNotes: fb.storedNotes, - } + // A WHILE body is a loop body: same sharing as addLoopStatement's. + loopBuilder := fb.nestedBuilder(innerStartX, innerStartY) // Body bookkeeping is addLoopStatement's, verbatim: a WHILE body is a loop // body, so the same merge-less split needs the same handling. pendingCase diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index f8691d94d..2b8c23a15 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -867,35 +867,20 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in errorX := sourceX // Build error handler activities - errBuilder := &flowBuilder{ - textLang: fb.textLang, - posX: errorX, - posY: errorY, - baseY: errorY, - spacing: HorizontalSpacing, - returnType: fb.returnType, - varTypes: fb.varTypes, - declaredVars: fb.declaredVars, - measurer: fb.measurer, - backend: fb.backend, - quiet: fb.quiet, - hierarchy: fb.hierarchy, - restServices: fb.restServices, - isNanoflow: fb.isNanoflow, - self: fb.self, - scriptFlows: fb.scriptFlows, - // A handler's activities are merged into the PARENT's object collection - // below, so a note declared outside the handler and referenced inside it - // (or the reverse) lands in one collection — sharing the registry is - // sound here in a way it is not across a loop boundary (#1077). - annotationsByLabel: fb.annotationsByLabel, - storedNotes: fb.storedNotes, - // Same collection, so the same label table: `join recovered` inside a - // handler must find the `merge recovered` declared on the main path. - // This is the one rejoin MDL could not spell before, and the reason the - // registry is a shared pointer rather than per-builder state. - labelReg: fb.labels(), - } + // A handler's activities are merged into the PARENT's object collection + // below, so a note declared outside the handler and referenced inside it + // (or the reverse) lands in one collection — sharing the note registry + // (nestedBuilder) is sound here in a way it is not across a loop boundary + // (#1077). + errBuilder := fb.nestedBuilder(errorX, errorY) + // The handler keeps the flow's return type, which its pending-handler + // routing and the undeclared-`set` advice read; a loop body never had it. + errBuilder.returnType = fb.returnType + // Same collection, so the same label table: `join recovered` inside a + // handler must find the `merge recovered` declared on the main path. + // This is the one rejoin MDL could not spell before, and the reason the + // registry is a shared pointer rather than per-builder state. + errBuilder.labelReg = fb.labels() var lastErrID model.ID var lastErrCase string diff --git a/mdl/executor/cmd_microflows_builder_nested_test.go b/mdl/executor/cmd_microflows_builder_nested_test.go new file mode 100644 index 000000000..460f13947 --- /dev/null +++ b/mdl/executor/cmd_microflows_builder_nested_test.go @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "reflect" + "sort" + "testing" + "unsafe" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" +) + +// nestedBuilderInherits classifies every field of flowBuilder: true when the +// builder of a nested body (a loop's, a while loop's, an error handler's) +// inherits it from its parent — nestedBuilder copies it — and false when the +// body keeps its own. A field added to flowBuilder fails +// TestNestedBuilder_InheritsSharedState until it is put here, which is the +// point: ako/mxcli#1769 was a field (qualifiedMembersOnly) that three +// hand-written copies each left out. +var nestedBuilderInherits = map[string]bool{ + // Which flow is built, and how its statements are written. + "varTypes": true, + "declaredVars": true, + "textLang": true, + "measurer": true, + "backend": true, + "quiet": true, + "hierarchy": true, + "restServices": true, + "isNanoflow": true, + "self": true, + "scriptFlows": true, + "annotationsByLabel": true, + "storedNotes": true, + "qualifiedMembersOnly": true, + + // Where the body is drawn: nestedBuilder's arguments, and the body's own + // lanes and rows (a loop box is sized to fit, so it never wraps). + "posX": false, "posY": false, "baseY": false, "spacing": false, + "lowerLane": false, "allowWrap": false, "row": false, + + // What the body builds, merged into the parent by the site. + "objects": false, "flows": false, "annotationFlows": false, "errors": false, + "statedNotes": false, "curveByOrigin": false, "rejoinMergeAt": false, + "returnEndIDs": false, "fallThroughEndID": false, "endsWithReturn": false, + "lastReturnEndID": false, "returnValue": false, + + // Per-statement pending state. + "nextConnectionPoint": false, "incomingRedirect": false, "nextFlowCase": false, + "nextFlowAnchor": false, "atBodyTail": false, "pendingAnnotations": false, + "previousStmtAnchor": false, "manualLoopBackTarget": false, "pendingJoin": false, + "emptyErrorHandlerFrom": false, "errorHandlerTailFrom": false, "errorHandlerSource": false, + "errorHandlerSkipVar": false, "errorHandlerTailCase": false, "errorHandlerTailAnchor": false, + "errorHandlerTailIsSource": false, "errorHandlerReturnValue": false, "pendingErrorHandlers": false, + + // Set by the one site that shares it: an error handler's body is in its + // parent's collection and on its path (addErrorHandlerFlow); a loop's is + // in a collection a sequence flow may not leave. + "returnType": false, "labelReg": false, + + // The whole flow's: the stored start event's position, and what + // buildFlowGraph collects from the body it builds. + "startPosition": false, "listInputVariables": false, "objectInputVariables": false, + "generatedVars": false, + + // Caches, filled on first use. + "assocCatalog": false, "microflowsCache": false, "microflowsCacheLoaded": false, + "nanoflowsCache": false, "nanoflowsCacheLoaded": false, + + // validateFlowBody's, which builds no nested builder. + "duplicateNamesOwnedElsewhere": false, "checkAssocShapes": false, + "assocObjectVars": false, "joinScopes": false, +} + +func TestNestedBuilder_InheritsSharedState(t *testing.T) { + typ := reflect.TypeOf(flowBuilder{}) + fields := map[string]bool{} + for i := 0; i < typ.NumField(); i++ { + name := typ.Field(i).Name + fields[name] = true + if _, ok := nestedBuilderInherits[name]; !ok { + t.Errorf("flowBuilder.%s is not classified: say in nestedBuilderInherits whether a nested body "+ + "inherits it (and copy it in nestedBuilder) or keeps its own", name) + } + } + var stale []string + for name := range nestedBuilderInherits { + if !fields[name] { + stale = append(stale, name) + } + } + sort.Strings(stale) + for _, name := range stale { + t.Errorf("nestedBuilderInherits names %s, which flowBuilder no longer has", name) + } + + parent := &flowBuilder{} + pv := reflect.ValueOf(parent).Elem() + for i := 0; i < pv.NumField(); i++ { + setNonZero(t, typ.Field(i).Name, settable(pv.Field(i))) + } + child := parent.nestedBuilder(11, 22) + cv := reflect.ValueOf(child).Elem() + placed := map[string]int64{"posX": 11, "posY": 22, "baseY": 22, "spacing": HorizontalSpacing} + for i := 0; i < cv.NumField(); i++ { + name := typ.Field(i).Name + p, c := settable(pv.Field(i)), settable(cv.Field(i)) + switch { + case placed[name] != 0: + if c.Int() != placed[name] { + t.Errorf("nestedBuilder(11, 22).%s = %d, want %d", name, c.Int(), placed[name]) + } + case nestedBuilderInherits[name]: + if !sameValue(p, c) { + t.Errorf("a nested body does not inherit flowBuilder.%s", name) + } + case !c.IsZero(): + t.Errorf("a nested body inherits flowBuilder.%s, classified as its own", name) + } + } +} + +// settable returns v — a field of an addressable struct, exported or not — +// as a value that can be read and set. +func settable(v reflect.Value) reflect.Value { + return reflect.NewAt(v.Type(), unsafe.Pointer(v.UnsafeAddr())).Elem() +} + +// setNonZero gives v a value distinguishable from its zero value and, for a +// reference, from any other instance of its type. +func setNonZero(t *testing.T, name string, v reflect.Value) { + t.Helper() + switch v.Kind() { + case reflect.Bool: + v.SetBool(true) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + v.SetInt(7) + case reflect.String: + v.SetString("x") + case reflect.Map: + v.Set(reflect.MakeMap(v.Type())) + case reflect.Slice: + v.Set(reflect.MakeSlice(v.Type(), 1, 1)) + case reflect.Pointer: + v.Set(reflect.New(v.Type().Elem())) + case reflect.Interface: + mb := reflect.ValueOf(&mock.MockBackend{}) + if !mb.Type().AssignableTo(v.Type()) { + t.Fatalf("flowBuilder.%s: no test value for %s", name, v.Type()) + } + v.Set(mb) + case reflect.Struct: + for i := 0; i < v.NumField(); i++ { + setNonZero(t, name, settable(v.Field(i))) + } + default: + t.Fatalf("flowBuilder.%s: no test value for kind %s", name, v.Kind()) + } +} + +// sameValue reports whether b is a's value — for a reference, the same +// instance, not an equal one. +func sameValue(a, b reflect.Value) bool { + switch a.Kind() { + case reflect.Map, reflect.Slice, reflect.Pointer: + return a.Pointer() == b.Pointer() + case reflect.Interface: + return !a.IsNil() && !b.IsNil() && a.Elem().Pointer() == b.Elem().Pointer() + } + return reflect.DeepEqual(a.Interface(), b.Interface()) +} diff --git a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go new file mode 100644 index 000000000..4cc62bd1f --- /dev/null +++ b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#1769 (mendixlabs/mxcli#1673) on the Studio Pro-authored PedApp, +// through the real codec: the reported alter, which replaces a loop with its +// own text, wrote the find in its body as Microflows$FindByExpression `Code = +// $Lookup/Code` — check and exec silent, CE0117 from mx check — where `create +// microflow` writes Microflows$Find on MyModule.OrderLine.Code. + +const matchLinesSetup = `mdl 1; +create module MyModule; +create persistent entity MyModule.Order (Code: String(50)); +create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); +create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; +create non-persistent entity MyModule.Lookup (Code: String(50)); +create microflow MyModule.SUB_MatchLines ($Order: MyModule.Order, $Lookups: List of MyModule.Lookup) returns Integer as $Found +begin + declare $Found Integer = 0; + retrieve $Lines from $Order/MyModule.OrderLine_Order; + $Open = filter $Lines by Status = 'open'; + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + return $Found; +end;` + +// matchLinesFindIn returns, for each way an alter can write a find, the +// statement that writes one in list: the reported replace of the loop with +// its own text, a replace of its body, and plain inserts in the stored body +// and on the main line, which the guard of ako/mxcli#891 always covered. +func matchLinesFindIn(list string) map[string]string { + find := `$Line = find ` + list + ` by Code = $Lookup/Code;` + return map[string]string{ + "replace loop": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + replace loop $Lookup in $Lookups with begin + loop $Lookup in $Lookups + begin + ` + find + ` + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + end; +};`, + "replace body of loop": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + replace body of loop $Lookup in $Lookups with begin + ` + find + ` + if $Line != empty then + set $Found = $Found + 1; + end if; + end; +};`, + "insert in the loop body": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + insert after $Line begin $Again = find ` + list + ` by Code = $Lookup/Code; end; +};`, + "insert on the main line": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + insert after $Open begin $First = find ` + list + ` by Code = 'x'; end; +};`, + } +} + +func openMatchLinesFixture(t *testing.T) (*Executor, *bytes.Buffer, string) { + t.Helper() + exec, out, dir := openPedAppCopy(t) + if err := agreeExec(t, exec, matchLinesSetup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + return exec, out, dir +} + +// TestAlterMicroflow_FindOverUntypedListIsRefused: over a list whose entity +// the fragment does not know — the stored flow does not type a list +// operation's output — every alter is refused and writes nothing. On the main +// line and in a stored loop body it always was (ako/mxcli#891); in a loop the +// fragment brings, it was written as a find by expression. Check does not +// pass it silently: like the main-line refusal, it names the change as not +// predicted with exec's reason. +func TestAlterMicroflow_FindOverUntypedListIsRefused(t *testing.T) { + const want = `cannot qualify member "Code"` + for name, src := range matchLinesFindIn("$Open") { + t.Run(name, func(t *testing.T) { + exec, _, dir := openMatchLinesFixture(t) + before := projectFiles(t, dir) + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], src) + } + r := exec.CheckFlowVerdictsReport(prog) + var said []string + for _, u := range r.Unpredicted { + said = append(said, u.Reason) + } + for _, v := range r.Violations { + said = append(said, v.Message) + } + if !strings.Contains(strings.Join(said, "\n"), want) { + t.Errorf("check must name the refusal %q, said: %v", want, said) + } + if err := exec.ExecuteProgram(prog); err == nil || !strings.Contains(err.Error(), want) { + t.Errorf("exec must refuse with %q, got: %v", want, err) + } + if changed := diffProjectFiles(before, projectFiles(t, dir)); len(changed) != 0 { + t.Errorf("a refused alter wrote %v", changed) + } + }) + } +} diff --git a/mdl/executor/flow_splice_nested_guard_test.go b/mdl/executor/flow_splice_nested_guard_test.go new file mode 100644 index 000000000..f43add3be --- /dev/null +++ b/mdl/executor/flow_splice_nested_guard_test.go @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1769 (mendixlabs/mxcli#1673): a fragment spliced into a stored +// flow refuses a member it cannot qualify (qualifiedMembersOnly, #885) — but +// only on its own main line. Every body nested in it (a loop's, a while +// loop's, an error handler's) was built by a fresh flowBuilder that did not +// inherit the flag, so `find $L by Code = …` in a replaced loop over a list +// whose entity the fragment does not know was written as a find by the +// expression `Code = …`: check and exec silent, CE0117 from mx check. + +// nestedShapes wraps one statement in each body a fragment can nest it in. +// %s is the statement; $Cs is a typed list to iterate, $C a typed object. +var nestedShapes = map[string]string{ + "loop body": `loop $It in $Cs + begin + %s + end loop;`, + "while body": `while $C != empty + begin + %s + end while;`, + "error handler": `$H = call microflow Mod.GetItem() on error without rollback begin + %s + end error;`, + "loop in a loop": `loop $It in $Cs + begin + loop $Jt in $Cs + begin + %s + end loop; + end loop;`, + "error handler in a loop": `loop $It in $Cs + begin + $H = call microflow Mod.GetItem() on error without rollback begin + %s + end error; + end loop;`, + "while in an error handler": `$H = call microflow Mod.GetItem() on error without rollback begin + while $C != empty + begin + %s + end while; + end error;`, + "if in a loop": `loop $It in $Cs + begin + if $It != empty then + %s + end if; + end loop;`, +} + +// memberStatements are the statements that name a member of their +// variable's entity. %[1]s is a list variable, %[2]s an object variable. +var memberStatements = map[string]string{ + "find": `$F = find %[1]s by Name = 'x';`, + "filter": `$G = filter %[1]s by Name = 'x';`, + "sort": `$S = sort %[1]s by Name asc;`, + "aggregate": `$M = maximum %[1]s by Name;`, + "change": `change %[2]s (Name = 'x');`, +} + +func nestedFragmentBody(t *testing.T, shape, stmt string) []ast.MicroflowStatement { + t.Helper() + src := "mdl 1;\ncreate microflow Mod.Fragment ()\nbegin\n " + + strings.ReplaceAll(nestedShapes[shape], "%s", stmt) + "\nend;" + return parsedFlowBody(t, src) +} + +// TestSpliceFragment_NestedBodiesRefuseUnqualifiedMembers: every member a +// nested body names over a variable the fragment cannot type is refused, as +// it is on the fragment's main line. Control: over the typed $Cs / $C the +// same fragments build, and the find is a find by attribute. +func TestSpliceFragment_NestedBodiesRefuseUnqualifiedMembers(t *testing.T) { + ctx, a := spliceFixture(t) + for shape := range nestedShapes { + for name, stmt := range memberStatements { + untyped := strings.ReplaceAll(strings.ReplaceAll(stmt, "%[1]s", "$A"), "%[2]s", "$A") + _, err := a.buildFragment(ctx, nestedFragmentBody(t, shape, untyped)) + if err == nil || !strings.Contains(err.Error(), `cannot qualify member "Name"`) { + t.Errorf("%s in the %s over an untyped variable: want the refusal, got %v", name, shape, err) + } + + typed := strings.ReplaceAll(strings.ReplaceAll(stmt, "%[1]s", "$Cs"), "%[2]s", "$C") + frag, err := a.buildFragment(ctx, nestedFragmentBody(t, shape, typed)) + if err != nil { + t.Errorf("control: %s in the %s over a typed variable: %v", name, shape, err) + continue + } + if name != "find" { + continue + } + ops := fragmentListOperations(frag.Objects) + if len(ops) != 1 { + t.Errorf("control: find in the %s: want one list operation, got %d", shape, len(ops)) + continue + } + if f, ok := ops[0].(*microflows.FindByAttributeOperation); !ok || f.Attribute != "Mod.Item.Name" { + t.Errorf("control: find in the %s wrote %#v, want a find by attribute Mod.Item.Name", shape, ops[0]) + } + } + } +} + +// fragmentListOperations returns the list operations among objs, loop bodies +// included. +func fragmentListOperations(objs []microflows.MicroflowObject) []microflows.ListOperation { + var out []microflows.ListOperation + for _, obj := range objs { + switch o := obj.(type) { + case *microflows.ActionActivity: + if lo, ok := o.Action.(*microflows.ListOperationAction); ok { + out = append(out, lo.Operation) + } + case *microflows.LoopedActivity: + if o.ObjectCollection != nil { + out = append(out, fragmentListOperations(o.ObjectCollection.Objects)...) + } + } + } + return out +} From 8d3089e06efbdea83985bb96259f8d5f0270a907 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:37:53 +0000 Subject: [PATCH 34/61] fix(executor): type a stored retrieve over an association for a fragment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom (ako/mxcli#1769, mendixlabs/mxcli#1673): after the previous commit, the reported alter — a replaced loop finding by member in $Lines, retrieved over an association before the loop — is refused with "cannot qualify member \"Code\"" where `create microflow` writes Microflows$Find on MyModule.OrderLine.Code. Cause: alterFlowContext.storedVariables, the variables a fragment starts from, typed parameters, declares, creates, database retrieves and calls, and left a retrieve over an association "Unknown": which end it yields depends on the end it starts from. A stored loop's iterator was not typed either, so a member of it in a fragment inserted in the loop body was refused too. Fix: storedVariables types both from the variable they start from, to a fixed point, so the order the stored flow lists them in does not matter. The retrieve's type is flowBuilder.associationRetrieveType: lookupAssociation, entityIsSubtypeOf and associationRetrieveYieldsList, the answer the full build and check --references already take. The parent is the FROM end and the child the TO end, so from the TO end (Order) a reference yields the list of FROM objects (OrderLine), from the FROM end the one TO object; a reference set is a list either way, an owner-Both reference one object either way. A self-reference, an unknown association, or a name two loops or retrieves give different origins stays untyped, and its member is refused rather than guessed. Controls: with storedVariables reverted, the association-typing tests and the PedApp by-attribute tests fail with the refusal; with the ambiguity guard off, the shared iterator name and the retrieve from it are typed. Measured on Mendix 11.13.0 (scratch app, real mx): before, FindByExpression "Code = $Lookup/Code" and CE0117; the previous commit alone, refused with exit 1 and the stored flow byte-identical; with both, Microflows$Find on MyModule.OrderLine.Code, mx check 0 errors, describe → exec "Unchanged microflow". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...d-over-association-list-by-expression.json | 1 + CHANGELOG.md | 1 + ...-alter-loop-find-over-association-list.mdl | 61 ++++++ mdl/executor/cmd_alter_flow.go | 84 +++++++- .../cmd_microflows_builder_actions.go | 45 ++++ .../flow_splice_assoc_retrieve_test.go | 201 ++++++++++++++++++ ...ow_splice_nested_guard_1769_pedapp_test.go | 110 ++++++++++ 7 files changed, 501 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json create mode 100644 mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl create mode 100644 mdl/executor/flow_splice_assoc_retrieve_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json new file mode 100644 index 000000000..266f7ebcb --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-29", "symptom": "ako/mxcli#1769 (mendixlabs/mxcli#1673, 11.12.4): `alter microflow … { replace loop $Lookup in $Lookups with begin loop … begin $Line = find $Lines by Code = $Lookup/Code; … end loop; end; }`, $Lines retrieved over an association before the loop, stored the find as Microflows$FindByExpression `Code = $Lookup/Code`. check --references and exec reported nothing (\"Altered microflow\"), describe printed `find $Lines where Code = …`, mx check failed CE0117. create microflow writes Microflows$Find on MyModule.OrderLine.Code; the same find inserted on the main line was refused (#891's guard). `replace body of loop` had the same hole.", "cause": "Two. (1) #891's flowBuilder.qualifiedMembersOnly is set on the fragment's builder, but the loop, while and error-handler bodies are built by fresh flowBuilders whose fields each site listed by hand, and none listed the flag, so every member a nested body names (find/filter by member, sort, aggregate, change) at any depth was written bare. (2) alterFlowContext.storedVariables, the variables a fragment starts from, typed parameters, declares, creates, database retrieves and calls but left a retrieve over an association 'Unknown' (direction was the stated reason), so the member could not be qualified at all. Stored loop iterators were not typed either.", "file": "mdl/executor/cmd_microflows_builder.go, mdl/executor/cmd_microflows_builder_control.go, mdl/executor/cmd_microflows_builder_flows.go, mdl/executor/cmd_alter_flow.go, mdl/executor/cmd_microflows_builder_actions.go", "fix": "flowBuilder.nestedBuilder is the one constructor for loop, while and error-handler bodies and inherits qualifiedMembersOnly with the rest of the shared state (the handler still adds returnType and labelReg). storedVariables types a retrieve over an association from its start variable through flowBuilder.associationRetrieveType (lookupAssociation + entityIsSubtypeOf + associationRetrieveYieldsList, the answer the full build and check take: from the TO end the list of the FROM entity, from the FROM end the TO object, a reference set a list either way, owner Both one object) and a loop's iterator from its list, to a fixed point so candidate order does not matter. A self-reference, an unknown association, or a name two loops or retrieves give different origins stays untyped and its member is refused.", "test": "mdl/executor/cmd_microflows_builder_nested_test.go TestNestedBuilder_InheritsSharedState (every flowBuilder field classified inherited or own, checked against nestedBuilder by reflection); flow_splice_nested_guard_test.go TestSpliceFragment_NestedBodiesRefuseUnqualifiedMembers (find/filter/sort/aggregate/change in loop, while, handler, loop in loop, handler in loop, while in handler, if in loop; typed control); flow_splice_assoc_retrieve_test.go (both directions, reference set, one-to-one, chain out of order, iterator, self-reference, ambiguous iterator); flow_splice_nested_guard_1769_pedapp_test.go (PedApp, real codec: replace loop, replace body, insert in body, insert on main line all write Microflows$Find; over an untyped list all four refused with nothing written; describe->exec Unchanged). Controls: flag left out of nestedBuilder -> every nested shape written and the structural test names the field; storedVariables reverted -> the reported shape refused; ambiguity guard off -> $ItOrder typed. Platform 11.13.0: unfixed CE0117 (1 error); fix (1) alone refused, exit 1, stored flow byte-identical; fixed Microflows$Find on MyModule.OrderLine.Code, mx check 0 errors, describe->exec Unchanged.", "insight": "A guard set on a builder is only as good as the places that construct builders: grep `&flowBuilder{` before trusting a flowBuilder flag, and where a field is copied by hand at several sites, replace the copies with one constructor and a test that makes every field be classified. The reported statement was not the blast radius: every member-writing statement in every nested body, and `replace body of`, had it. Separately, an unqualifiable member is the symptom of an untyped variable; read what the fragment's starting scope types (storedVariables) before adding another refusal. Check reports a fragment-builder refusal as 'not predicted' with the reason, not MDL090, as it does #891's main-line refusal.", "ce": ["CE0117"], "refs": ["ako/mxcli#1769", "mendixlabs/mxcli#1673", "ako/mxcli#891", "ako/mxcli#885"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index eafed1b56..6390f9622 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. - **A `@position` on a later statement no longer moves the statements before it onto it** (ako/mxcli#1768, mendixlabs/mxcli#1672) — `declare $A String = $Url; @position(840, 200) declare $B String = $A;` stored both declares at 840;200, and MPR008 reported the overlap. The start event is placed one spacing left of the first statement when that statement carries `@position`, but the builder took the first statement that had one, wherever it was, and the unplaced statements before it follow the start. Only the first statement moves the start now, as `alter`'s patch and the syntax help already assumed: the flow above starts at 200;200 with `$A` at 360;200 and `$B` at 840;200. The syntax help and the language reference now say what happens to the statements before a placed one. - **MPR008 reports an activity drawn on an event** (ako/mxcli#1768, mendixlabs/mxcli#1672) — the overlap rule compared activities, loops, decisions and merges only, so an error handler's activity lying on another handler's end event at 680;330 passed, and with one more statement in that handler it reported one of the two overlaps, the one that happened to be activity on activity. Start, end, error, break and continue events and type splits are compared too, each with its stored size (an event that stores none is taken as Studio Pro's 20×20, not as an activity). On the 292 flows of the Studio Pro-authored PedApp and ako/TestApp fixtures, and on 182 flows built from the `02*` and `06` doctype examples, the wider rule adds no finding. +- **A find in a loop an `alter microflow` writes is stored as a find by attribute, not as a find by an invalid expression** (ako/mxcli#1769, mendixlabs/mxcli#1673) — `alter microflow … { replace loop … with begin … end; }` whose loop body held `$Line = find $Lines by Code = $Lookup/Code;`, with `$Lines` retrieved over an association before the loop, stored `Microflows$FindByExpression` with the bare text `Code = $Lookup/Code`: `check --references` and `exec` reported nothing, `describe` printed `find $Lines where …`, and mx check failed CE0117 "Error(s) in expression." `create microflow` stores the same find as `Microflows$Find` on `MyModule.OrderLine.Code`. Two causes. The guard that refuses a member a spliced fragment cannot qualify (ako/mxcli#891) applied on the fragment's main line only, not in a loop, while loop or error handler inside it, and so did not cover `replace body of loop` either: there any find, filter, sort, aggregate or change member was written bare. And the fragment did not know what a stored retrieve over an association holds, nor a stored loop's iterator. Both are now typed as `create microflow` types them: from the association's TO end a retrieve yields the list of its FROM entity, from the FROM end the one TO object. So the reported alter writes `Microflows$Find`, and what still cannot be typed (a self-reference, a list operation's output) is refused with nothing written, wherever the statement sits. Measured on Mendix 11.13.0: CE0117 before; `Microflows$Find` on `MyModule.OrderLine.Code`, mx check 0 errors, and describe → exec "Unchanged" after. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl b/mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl new file mode 100644 index 000000000..3c21e20a6 --- /dev/null +++ b/mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl @@ -0,0 +1,61 @@ +mdl 1; +-- ako/mxcli#1769 (mendixlabs/mxcli#1673) — a find in a loop an alter writes +-- +-- `alter microflow … { replace loop … with begin … end; }` whose loop body +-- finds by member in a list retrieved over an association before the loop +-- (`retrieve $Lines from $Order/MyModule.OrderLine_Order`) stored the find as +-- Microflows$FindByExpression with the bare text `Code = $Lookup/Code`: check +-- --references and exec reported nothing, and mx check failed CE0117 "Error(s) +-- in expression." `create microflow` stores the same find as Microflows$Find +-- on MyModule.OrderLine.Code. +-- +-- Two causes. The member guard a spliced fragment has (ako/mxcli#891) was not +-- inherited by the builder of a body nested in it — a loop, a while loop, an +-- error handler — so it wrote a member it could not qualify where the main +-- line refused it. And the fragment did not know what a stored retrieve over +-- an association holds, so it could not qualify the member. A retrieve from +-- the TO end of a reference (Order here) yields the list of the FROM entity +-- (OrderLine); from the FROM end, the one TO object. +-- +-- Run on any project: +-- mxcli exec mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl -p app.mpr +-- mxcli bson dump -p app.mpr --type microflow --object MyModule.SUB_MatchLines +-- Expected: "Altered microflow MyModule.SUB_MatchLines"; the dump shows +-- "$Type": "Microflows$Find" with "Attribute": "MyModule.OrderLine.Code" and +-- "Expression": "$Lookup/Code"; mx check reports no error for the flow, and +-- exec of `describe microflow MyModule.SUB_MatchLines` reports Unchanged. +-- Before the fix: Microflows$FindByExpression "Code = $Lookup/Code" and CE0117 +-- (measured on 11.13.0). + +create module MyModule; +create persistent entity MyModule.Order (Code: String(50)); +create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); +create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; +create non-persistent entity MyModule.Lookup (Code: String(50)); + +create microflow MyModule.SUB_MatchLines ($Order: MyModule.Order, $Lookups: List of MyModule.Lookup) returns Integer as $Found +begin + declare $Found Integer = 0; + retrieve $Lines from $Order/MyModule.OrderLine_Order; + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + return $Found; +end; + +-- The loop replaced with its own text. +alter microflow MyModule.SUB_MatchLines { + replace loop $Lookup in $Lookups with begin + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + end; +}; diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go index 30d6c192e..1205acecf 100644 --- a/mdl/executor/cmd_alter_flow.go +++ b/mdl/executor/cmd_alter_flow.go @@ -987,7 +987,18 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared for _, p := range a.mf.Parameters { add(p.Name, p.Type) } + // What is typed from another variable — a retrieve over an association + // from its start, a loop's iterator from its list — is typed once the + // rest are, below. + deps := storedDependents{} + fb := &flowBuilder{backend: ctx.Backend} for _, c := range a.cands { + if loop, ok := c.Object.(*microflows.LoopedActivity); ok { + if src, ok := loop.LoopSource.(*microflows.IterableList); ok { + deps.add(src.VariableName, storedDependent{from: src.ListVariableName}) + } + continue + } act, ok := c.Object.(*microflows.ActionActivity) if !ok || c.OutputVariable == "" { continue @@ -1005,7 +1016,13 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared } case *microflows.RetrieveAction: // A database retrieve names its entity; one over an association - // depends on which side it starts from and stays untyped here. + // yields an end of it, which depends on the end it starts from + // and is typed below (ako/mxcli#1769). + if src, ok := x.Source.(*microflows.AssociationRetrieveSource); ok { + declared[c.OutputVariable] = "Unknown" + deps.add(c.OutputVariable, storedDependent{from: src.StartVariable, assoc: src.AssociationQualifiedName}) + break + } src, ok := x.Source.(*microflows.DatabaseRetrieveSource) entity := "" if ok { @@ -1026,7 +1043,7 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared // A call's result has the called microflow's return type. var rt microflows.DataType if x.MicroflowCall != nil { - rt = (&flowBuilder{backend: ctx.Backend}).lookupMicroflowReturnType(x.MicroflowCall.Microflow) + rt = fb.lookupMicroflowReturnType(x.MicroflowCall.Microflow) } switch rt.(type) { case *microflows.ObjectType, *microflows.ListType: @@ -1038,9 +1055,72 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared declared[c.OutputVariable] = "Unknown" } } + deps.typeAll(fb, varTypes, declared) return varTypes, declared } +// storedDependent is a stored variable whose type follows from another's: the +// output of a retrieve over assoc from the variable from, or — assoc empty — +// the iterator of a loop over the list from. +type storedDependent struct { + from, assoc string +} + +// storedDependents are the stored flow's dependent variables by name. A name +// two of them give different origins is nil: which one a fragment means +// depends on where it goes (two loops can share an iterator name), and a guess +// would qualify a member with the wrong entity. It stays untyped and the +// member is refused. +type storedDependents map[string]*storedDependent + +func (d storedDependents) add(name string, dep storedDependent) { + if name == "" { + return + } + if old, seen := d[name]; seen { + if old == nil || *old != dep { + d[name] = nil + } + return + } + d[name] = &dep +} + +// typeAll types every dependent whose origin is typed — a parameter, a +// database retrieve, another dependent — whatever order the stored flow lists +// them in, until a pass types nothing new. A name the rest of the stored flow +// already types keeps that type. +func (d storedDependents) typeAll(fb *flowBuilder, varTypes, declared map[string]string) { + for progress := true; progress; { + progress = false + for name, dep := range d { + if dep == nil { + continue + } + if _, typed := varTypes[name]; typed { + continue + } + from, ok := varTypes[dep.from] + if !ok { + continue + } + t := "" + if dep.assoc == "" { + if elem, isList := strings.CutPrefix(from, "List of "); isList { + t = elem + } + } else { + t = fb.associationRetrieveType(from, dep.assoc) + } + if t != "" { + varTypes[name] = t + delete(declared, name) + progress = true + } + } + } +} + // systemVariables are in scope everywhere they exist at all; the platform // reports a misuse (a $latestError outside an error handler) itself. var systemVariables = map[string]bool{ diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index cbdf6f816..55dd0c3c8 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -2833,6 +2833,51 @@ func (fb *flowBuilder) lookupAssociation(moduleName, assocName string) *assocLoo return nil } +// associationRetrieveType is what `retrieve $X from $Start/assocQN` binds when +// $Start holds startType ("Module.Entity"): the list or the object at the +// other end, by associationRetrieveYieldsList — the answer addRetrieveAction +// and check --references take. Mind the inversion: an association's parent is +// its FROM end and its child the TO end, so a retrieve from the TO end yields +// the FROM entity. "" when it cannot be established: the association or the +// start unknown, the start a list, or the start no end of it — or, over a +// reference, both ends (a self-reference), where which end it starts from is +// not known. +func (fb *flowBuilder) associationRetrieveType(startType, assocQN string) string { + if startType == "" || strings.HasPrefix(startType, "List of ") { + return "" + } + module, name, ok := strings.Cut(assocQN, ".") + if !ok || module == "" || name == "" { + return "" + } + info := fb.lookupAssociation(module, name) + if info == nil || info.parentEntityQN == "" || info.childEntityQN == "" { + return "" + } + refSet := info.Type == domainmodel.AssociationTypeReferenceSet + fromTo := fb.entityIsSubtypeOf(startType, info.childEntityQN) + fromFrom := fb.entityIsSubtypeOf(startType, info.parentEntityQN) + switch { + case fromTo && fromFrom: + // Either end: a reference set to itself is a list of it whichever + // end it starts from, and nothing else is known. + if refSet && info.parentEntityQN == info.childEntityQN { + return "List of " + info.parentEntityQN + } + return "" + case !fromTo && !fromFrom: + return "" + } + other := info.childEntityQN + if fromTo { + other = info.parentEntityQN + } + if associationRetrieveYieldsList(refSet, info.Owner == domainmodel.AssociationOwnerBoth, fromTo) { + return "List of " + other + } + return other +} + // qualifyIteratorAttributes rewrites a bare attribute name in a FILTER/FIND // predicate into `$currentObject/`. // diff --git a/mdl/executor/flow_splice_assoc_retrieve_test.go b/mdl/executor/flow_splice_assoc_retrieve_test.go new file mode 100644 index 000000000..be5dd7aad --- /dev/null +++ b/mdl/executor/flow_splice_assoc_retrieve_test.go @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1769 (mendixlabs/mxcli#1673): a fragment spliced into a stored +// flow is built knowing the variables the stored flow declares +// (storedVariables), and a retrieve over an association was left untyped +// there, though a database retrieve and a list parameter were typed. A find by +// member in a list retrieved over an association could not be qualified, so +// the fragment refused it — or, in a loop the fragment brought, wrote it as a +// find by the invalid expression `Code = …` (CE0117). + +// orderFixture is the reported domain model: OrderLine_Order runs FROM +// OrderLine TO Order (ParentPointer = OrderLine, ChildPointer = Order), so a +// retrieve over it from an Order yields the list of its lines and one from an +// OrderLine yields its one order. Order_Tag is a reference set FROM Order TO +// Tag; Order_Invoice a one-to-one (owner Both) reference FROM Order TO +// Invoice; Order_Parent a reference from Order to itself. +func orderFixture(t *testing.T) (*ExecContext, *alterFlowContext) { + t.Helper() + mod := mkModule("Mod") + attr := func(name string) *domainmodel.Attribute { + return &domainmodel.Attribute{BaseElement: model.BaseElement{ID: nextID("attr")}, Name: name} + } + order := mkEntity(mod.ID, "Order") + order.Attributes = []*domainmodel.Attribute{attr("Code")} + line := mkEntity(mod.ID, "OrderLine") + line.Attributes = []*domainmodel.Attribute{attr("Code"), attr("Status")} + lookup := mkEntity(mod.ID, "Lookup") + lookup.Attributes = []*domainmodel.Attribute{attr("Code")} + tag := mkEntity(mod.ID, "Tag") + tag.Attributes = []*domainmodel.Attribute{attr("Label")} + invoice := mkEntity(mod.ID, "Invoice") + invoice.Attributes = []*domainmodel.Attribute{attr("Number")} + dm := mkDomainModel(mod.ID, order, line, lookup, tag, invoice) + tags := mkAssociation(dm.ID, "Order_Tag", order.ID, tag.ID) + tags.Type = domainmodel.AssociationTypeReferenceSet + oneToOne := mkAssociation(dm.ID, "Order_Invoice", order.ID, invoice.ID) + oneToOne.Owner = domainmodel.AssociationOwnerBoth + dm.Associations = []*domainmodel.Association{ + mkAssociation(dm.ID, "OrderLine_Order", line.ID, order.ID), + tags, + oneToOne, + mkAssociation(dm.ID, "Order_Parent", order.ID, order.ID), + } + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetModuleByNameFunc: func(name string) (*model.Module, error) { + if name == "Mod" { + return mod, nil + } + return nil, nil + }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + obj := func(e string) microflows.DataType { return µflows.ObjectType{EntityQualifiedName: "Mod." + e} } + over := func(out, start, assoc string) mfmutator.Candidate { + return mfmutator.Candidate{OutputVariable: out, Object: µflows.ActionActivity{Action: µflows.RetrieveAction{ + Source: µflows.AssociationRetrieveSource{StartVariable: start, AssociationQualifiedName: "Mod." + assoc}}}} + } + loop := func(iterator, list string) mfmutator.Candidate { + return mfmutator.Candidate{Object: µflows.LoopedActivity{ + LoopSource: µflows.IterableList{VariableName: iterator, ListVariableName: list}}} + } + a := &alterFlowContext{ + stmt: &ast.AlterFlowStmt{}, + mf: µflows.Microflow{Parameters: []*microflows.MicroflowParameter{ + {Name: "Order", Type: obj("Order")}, + {Name: "Line", Type: obj("OrderLine")}, + {Name: "Tag", Type: obj("Tag")}, + {Name: "Invoice", Type: obj("Invoice")}, + {Name: "Lookups", Type: µflows.ListType{EntityQualifiedName: "Mod.Lookup"}}, + }}, + entityNames: map[model.ID]string{order.ID: "Mod.Order", line.ID: "Mod.OrderLine"}, + cands: []mfmutator.Candidate{ + over("Lines", "Order", "OrderLine_Order"), // from the TO end: the lines + over("TheOrder", "Line", "OrderLine_Order"), // from the FROM end: the order + over("Tags", "Order", "Order_Tag"), // reference set, forward + over("TagOrders", "Tag", "Order_Tag"), // reference set, reverse + over("TheInvoice", "Order", "Order_Invoice"), // one-to-one, forward + over("InvoiceOrder", "Invoice", "Order_Invoice"), // one-to-one, reverse + // Starts from a variable a later candidate types: the order of the + // candidates is not the order a variable is typed in. + over("SiblingLines", "LineOrder", "OrderLine_Order"), + over("LineOrder", "Line", "OrderLine_Order"), + over("Parent", "Order", "Order_Parent"), // self-reference: which end is unknown + over("Nowhere", "Order", "Order_Missing"), // no such association + over("Unstarted", "NoSuchVariable", "OrderLine_Order"), + // A loop's iterator is an element of its list, and a retrieve in + // the body can start from it. + loop("EachLine", "Lines"), + over("EachOrder", "EachLine", "OrderLine_Order"), + // Two loops whose iterators share a name over different lists: + // which one a fragment means depends on where it goes. + loop("It", "Lines"), + loop("It", "Tags"), + over("ItOrder", "It", "OrderLine_Order"), + }, + } + return ctx, a +} + +// TestStoredVariables_AssociationRetrieveIsTyped: the output of a stored +// retrieve over an association has the type the association gives it from +// the end it starts at — a list of the FROM entity from the TO end, the one +// TO object from the FROM end, a list either way over a reference set, one +// object either way over a one-to-one. What cannot be established stays +// untyped, so a member of it is refused, not guessed. +func TestStoredVariables_AssociationRetrieveIsTyped(t *testing.T) { + ctx, a := orderFixture(t) + varTypes, declared := a.storedVariables(ctx) + want := map[string]string{ + "Lines": "List of Mod.OrderLine", + "TheOrder": "Mod.Order", + "Tags": "List of Mod.Tag", + "TagOrders": "List of Mod.Order", + "TheInvoice": "Mod.Invoice", + "InvoiceOrder": "Mod.Order", + "SiblingLines": "List of Mod.OrderLine", + "LineOrder": "Mod.Order", + "EachLine": "Mod.OrderLine", + "EachOrder": "Mod.Order", + } + for v, w := range want { + if got := varTypes[v]; got != w { + t.Errorf("$%s: typed %q, want %q", v, got, w) + } + } + for _, v := range []string{"Parent", "Nowhere", "Unstarted", "ItOrder"} { + if got, ok := varTypes[v]; ok { + t.Errorf("$%s: typed %q, want it left untyped", v, got) + } + if declared[v] != "Unknown" { + t.Errorf("$%s: declared %q, want Unknown", v, declared[v]) + } + } + if got, ok := varTypes["It"]; ok { + t.Errorf("$It, the iterator of two loops over different lists: typed %q, want it left untyped", got) + } +} + +// TestSpliceFragment_ReplacedLoopFindsOverAssociationList is the reported +// alter: a replaced loop whose body finds in a list retrieved over an +// association before the loop is written as a find by attribute, the way +// `create microflow` writes it — not as a find by the expression `Code = …`. +func TestSpliceFragment_ReplacedLoopFindsOverAssociationList(t *testing.T) { + ctx, a := orderFixture(t) + frag, err := a.buildFragment(ctx, parsedFlowBody(t, `mdl 1; +create microflow Mod.Fragment () +begin + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + change $TheOrder (Code = $Lookup/Code); + end loop; +end;`)) + if err != nil { + t.Fatalf("the replaced loop: %v", err) + } + ops := fragmentListOperations(frag.Objects) + if len(ops) != 1 { + t.Fatalf("want one list operation, got %d", len(ops)) + } + f, ok := ops[0].(*microflows.FindByAttributeOperation) + if !ok || f.Attribute != "Mod.OrderLine.Code" || f.Expression != "$Lookup/Code" { + t.Fatalf("the find was written as %#v, want a find by attribute Mod.OrderLine.Code = $Lookup/Code", ops[0]) + } + var changed []string + for _, obj := range frag.Objects { + loop, ok := obj.(*microflows.LoopedActivity) + if !ok { + continue + } + for _, inner := range loop.ObjectCollection.Objects { + if act, ok := inner.(*microflows.ActionActivity); ok { + if ch, ok := act.Action.(*microflows.ChangeObjectAction); ok { + for _, m := range ch.Changes { + changed = append(changed, m.AttributeQualifiedName) + } + } + } + } + } + if strings.Join(changed, ",") != "Mod.Order.Code" { + t.Errorf("the change over the forward retrieve wrote %v, want [Mod.Order.Code]", changed) + } +} diff --git a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go index 4cc62bd1f..11680b697 100644 --- a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go +++ b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go @@ -4,10 +4,12 @@ package executor import ( "bytes" + "context" "strings" "testing" "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" ) // ako/mxcli#1769 (mendixlabs/mxcli#1673) on the Studio Pro-authored PedApp, @@ -122,3 +124,111 @@ func TestAlterMicroflow_FindOverUntypedListIsRefused(t *testing.T) { }) } } + +// matchLinesFlow returns the stored MyModule.SUB_MatchLines. +func matchLinesFlow(t *testing.T, exec *Executor) *microflows.Microflow { + t.Helper() + ctx := exec.newExecContext(context.Background()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + all, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range all { + if m.Name == "SUB_MatchLines" && h.GetModuleName(h.FindModuleID(m.ContainerID)) == "MyModule" { + mf, err := ctx.Backend.GetMicroflow(m.ID) + if err != nil { + t.Fatal(err) + } + return mf + } + } + t.Fatal("MyModule.SUB_MatchLines not found") + return nil +} + +// TestAlterMicroflow_FindOverAssociationListIsByAttribute: every alter that +// writes `find $Lines by Code = …`, where $Lines is retrieved over an +// association before it, writes Microflows$Find on MyModule.OrderLine.Code, +// as `create microflow` does; check predicts it and exec writes it. +func TestAlterMicroflow_FindOverAssociationListIsByAttribute(t *testing.T) { + for name, src := range matchLinesFindIn("$Lines") { + t.Run(name, func(t *testing.T) { + exec, _, _ := openMatchLinesFixture(t) + predictedAgree(t, exec, src, "") + var finds int + for _, op := range fragmentListOperations(matchLinesFlow(t, exec).ObjectCollection.Objects) { + switch f := op.(type) { + case *microflows.FindOperation: + t.Errorf("a find over $%s was stored by the expression %q", f.ListVariable, f.Expression) + case *microflows.FindByAttributeOperation: + finds++ + if f.Attribute != "MyModule.OrderLine.Code" { + t.Errorf("a find over $%s is on %q, want MyModule.OrderLine.Code", f.ListVariable, f.Attribute) + } + } + } + if finds == 0 { + t.Errorf("no find by attribute was stored") + } + }) + } +} + +// The reported alter writes what describe prints back unchanged: exec of the +// describe output reports Unchanged. +func TestAlterMicroflow_ReplacedLoopDescribeRoundTrips(t *testing.T) { + exec, out, _ := openMatchLinesFixture(t) + predictedAgree(t, exec, matchLinesFindIn("$Lines")["replace loop"], "") + var desc bytes.Buffer + saved := exec.output + exec.output = &desc + err := afRun(t, exec, "describe microflow MyModule.SUB_MatchLines;") + exec.output = saved + if err != nil { + t.Fatal(err) + } + if !strings.Contains(desc.String(), "find $Lines by Code = $Lookup/Code;") { + t.Errorf("describe does not print the find by attribute:\n%s", desc.String()) + } + out.Reset() + if err := afRun(t, exec, "mdl 1;\n"+desc.String()); err != nil { + t.Fatalf("exec of the describe output: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "Unchanged") { + t.Errorf("exec of the describe output must report Unchanged, got:\n%s", out.String()) + } +} + +// A stored loop's iterator is typed from its list, so a member of it in a +// fragment inserted in the loop body is qualified; it was refused. +func TestAlterMicroflow_ChangeOnStoredIteratorIsQualified(t *testing.T) { + exec, _, _ := openMatchLinesFixture(t) + predictedAgree(t, exec, `mdl 1; +alter microflow MyModule.SUB_MatchLines { + insert after $Line begin change $Lookup (Code = 'x'); end; +};`, "") + var changed []string + var walk func(objs []microflows.MicroflowObject) + walk = func(objs []microflows.MicroflowObject) { + for _, obj := range objs { + switch o := obj.(type) { + case *microflows.ActionActivity: + if ch, ok := o.Action.(*microflows.ChangeObjectAction); ok { + for _, m := range ch.Changes { + changed = append(changed, m.AttributeQualifiedName) + } + } + case *microflows.LoopedActivity: + walk(o.ObjectCollection.Objects) + } + } + } + walk(matchLinesFlow(t, exec).ObjectCollection.Objects) + if strings.Join(changed, ",") != "MyModule.Lookup.Code" { + t.Errorf("the change was stored on %v, want [MyModule.Lookup.Code]", changed) + } +} From 47ec0e02fac3dfe0737ce3a2c6e18d91adc6b85d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:04:49 +0000 Subject: [PATCH 35/61] fix(canon): store a user task's targeting as UserSource below Mendix 11.2 Symptom (ako/mxcli#1771): on Mendix 10.12 to 11.1 every user task mxcli wrote failed mx check with CE1859 "XPath constraint is required". That included tasks targeted by a microflow, and multi-user tasks. exec and mxcli check were clean. `mx convert -p` 10.24.24 then stored an empty XPathBasedUserSource in place of each targeting, so the targeting was silently lost. Cause: userTaskToGen writes UserTargeting, which Mendix introduced in 11.2.0 (mendixmodelsdk: UserTaskActivity.userSource deleted and userTargeting introduced in 11.2.0). It is declared on the abstract UserTaskActivity, and the property floor looks up the concrete type only, so the key went into every version. 10.24 ignores it and fills the required UserSource with its default. Older shape, measured: two Studio Pro-authored Marketplace samples, Employee Onboarding (9.1.1, XPath) and Multilevel Dynamic Workflow (9.24.3, microflow), converted 9.24.30 -> 10.24.24 -> 11.13.0. The targeting keeps its $ID and value byte for byte. Only its $Type and the key holding it change: XPathBasedUserSource <-> XPathUserTargeting, MicroflowBasedUserSource <-> MicroflowUserTargeting, and EmptyUserSource -> NoUserTargeting (this fix's 10.24 output converted up). Fix: canon.LowerToProjectShape renames the targeting to the user source below 11.2, in its place, for every writer. Group targeting is 11.2 and has no older form, so it is refused, naming the task. Measured on fresh apps, exec of the bug-test then mx check: - 10.24.24.119653: before CE1859 x3, after 0 errors. mx convert 10.24.24 changes no key and no value. describe -> exec "Unchanged workflow". - 11.13.0: 0 errors, and the key sets match the unfixed build. Control: stubbing the lowering fails TestNormalize_UserTaskTargetingBelowMendix11_2 24 times, with UserTargeting still on the task. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-10-29-user-task-targeting-below-11-2.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1771-user-task-targeting-below-11-2.mdl | 69 +++++ .../modelsdk/workflow_usertask_shape_test.go | 90 ++++++ modelsdk/canon/versionshape.go | 11 +- modelsdk/canon/versionshape_usertask.go | 93 +++++++ .../mpr/writer_version_shape_1771_test.go | 256 ++++++++++++++++++ 8 files changed, 522 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json create mode 100644 mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl create mode 100644 mdl/backend/modelsdk/workflow_usertask_shape_test.go create mode 100644 modelsdk/canon/versionshape_usertask.go create mode 100644 modelsdk/mpr/writer_version_shape_1771_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json new file mode 100644 index 000000000..fd5df2999 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 10.12 to 11.1 (measured on 10.12.19 and 10.24.24) every user task mxcli wrote failed mx check with CE1859 \"XPath constraint is required\". That included tasks targeted by a microflow, and multi-user tasks. `mxcli check` and `exec` were clean, and `mx convert -p` 10.24.24 rewrote each task with an empty XPathBasedUserSource, so the targeting was silently lost", "cause": "userTaskToGen writes the targeting as UserTargeting (XPathUserTargeting / MicroflowUserTargeting / NoUserTargeting), the 11.2 property. Before 11.2 a user task stores UserSource (XPathBasedUserSource / MicroflowBasedUserSource / EmptyUserSource) (mendixmodelsdk: UserTaskActivity.userSource deleted, userTargeting introduced 11.2.0). Both are declared on the abstract Workflows$UserTaskActivity, and the floor looks up the concrete type only, so it never saw them. 10.24 loads the task and ignores the unknown key, then fills the required UserSource with its default, an XPath source with no constraint", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTargeting, userSourceOf), hooked in `modelsdk/canon/versionshape.go` (LowerToProjectShape: userSource below 11.2)", "insight": "**Read a version's ERROR as a read of the default, not of what was written.** CE1859 named the XPath constraint even on microflow-targeted tasks. 10.24 had dropped the whole key and read the required property's default, an XPath source. So 'XPath constraint is required' on a task with no XPath points at the property holding it, not the constraint. **The convert chain is the oracle for a renamed property**: Marketplace samples (Employee Onboarding 117785, 9.1.1 XPath; Multilevel Dynamic Workflow 221741, 9.24.3 microflow) converted 9.1.1 → 9.24.30 → 10.24.24 → 11.13.0 keep the targeting's $ID and value byte for byte and change only $Type and the holding key. So the lowering is a rename. 9.1.1 does not convert straight to 10.24 ('version not supported'), so go through 9.24.30 first. Converting this fix's 10.24 output up to 11.13 turns EmptyUserSource into NoUserTargeting, which confirms the third pair. Workflow Commons (2.6.0, 3.7.0, 3.8.0) holds no workflow at all. Group targeting has no older form and is refused naming the task. Measured: bug-test on a fresh 10.24.24 app, before CE1859 x3 and after 0 errors. `mx convert -p` 10.24.24 changes no key and no value. describe → exec prints 'Unchanged workflow'. 11.13.0 gives 0 errors with the same key sets before and after. Control: stubbing the lowering fails TestNormalize_UserTaskTargetingBelowMendix11_2 24 times with UserTargeting still on the task. Tests: TestNormalize_UserTaskTargetingBelowMendix11_2, TestNormalize_RefusesGroupTargetingBelowMendix11_2, TestWorkflowRead_UserTaskTargetingInTheMendix10Shape; mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl", "refs": ["#1771"], "ce": ["CE1859"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 6390f9622..eab978aac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 8350ab77b..0af20a6fa 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -187,6 +187,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | | The end of a parallel split path (`Workflows$EndOfParallelSplitPathActivity`, a type introduced in 10.21) | 10.21 | Not stored: a path holds its activities and nothing more (Studio Pro 9.4) | +| A user task's targeting (`targeting users xpath […]`, `targeting users microflow …`, none) | 11.2 | A `UserSource` (`XPathBasedUserSource`, `MicroflowBasedUserSource`, `EmptyUserSource`) holding the same value, instead of a `UserTargeting` ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)) | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | | A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | @@ -197,6 +198,7 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a data view fed by another widget below 10.0; - a private constant value in a configuration below 10.9; - a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; +- a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl new file mode 100644 index 000000000..c23a49cc1 --- /dev/null +++ b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl @@ -0,0 +1,69 @@ +mdl 1; +-- ako/mxcli#1771: on Mendix 10.12 to 11.1 every user task mxcli wrote failed +-- mx check with CE1859 "XPath constraint is required". That included tasks +-- targeted by a microflow. `exec` and `mxcli check` were clean. +-- +-- mxcli stored the targeting as UserTargeting, the 11.2 property. Before 11.2 a +-- user task stores it as UserSource. UserTargeting is declared on the abstract +-- UserTaskActivity, which the property floor does not look up, so the key went +-- into every version. The older versions read the task as having no user +-- source. `mx convert -p` 10.24.24 then dropped the targeting and stored an +-- empty XPathBasedUserSource in its place, so the targeting was silently lost. +-- +-- The older shape, measured with `mx convert -p`. The sources are Studio +-- Pro-authored Marketplace samples: Employee Onboarding (9.1.1, XPath +-- targeting) and Multilevel Dynamic Workflow (9.24.3, microflow targeting). +-- Each was converted 9.24.30 → 10.24.24 → 11.13.0. The value is carried byte +-- for byte, and only its holder changes: +-- UserSource: XPathBasedUserSource ↔ UserTargeting: XPathUserTargeting +-- UserSource: MicroflowBasedUserSource ↔ UserTargeting: MicroflowUserTargeting +-- A task with no targeting is an EmptyUserSource (9.10+), and from 11.2 a +-- NoUserTargeting. Group targeting is 11.2. Before 11.2 it is refused with +-- MDL-VERSION01 by check -p, and at write time. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`): exec, then +-- `mxcli docker check`. +-- 10.24.24.119653 before: CE1859 on every task. After: 0 errors. +-- `mx convert -p` 10.24.24 changes no key and no value. +-- describe → exec prints "Unchanged workflow". +-- 11.13.0 0 errors before and after, with the same key sets. + +create module Probe1771Tg; + +create persistent entity Probe1771Tg.Request ( Title: String(100) ); + +create microflow Probe1771Tg.ACT_GetUsers ($Workflow: System.Workflow, $Context: Probe1771Tg.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create page Probe1771Tg.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771Tg.WF_Targeting + parameter $WorkflowContext: Probe1771Tg.Request + display 'Targeting' +begin + user task ByXPath 'By XPath' + page Probe1771Tg.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Done' { }; + user task ByMicroflow 'By microflow' + page Probe1771Tg.TaskPage + targeting users microflow Probe1771Tg.ACT_GetUsers + outcomes 'Done' { }; + user task Untargeted 'No targeting' + page Probe1771Tg.TaskPage + outcomes 'Done' { }; + multi user task MultiByXPath 'Multi by XPath' + page Probe1771Tg.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Approve' { } 'Reject' { }; +end workflow; diff --git a/mdl/backend/modelsdk/workflow_usertask_shape_test.go b/mdl/backend/modelsdk/workflow_usertask_shape_test.go new file mode 100644 index 000000000..34d2ce89f --- /dev/null +++ b/mdl/backend/modelsdk/workflow_usertask_shape_test.go @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "reflect" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/modelsdk/canon" + "github.com/mendixlabs/mxcli/modelsdk/codec" + genWf "github.com/mendixlabs/mxcli/modelsdk/gen/workflows" + "github.com/mendixlabs/mxcli/modelsdk/version" + "github.com/mendixlabs/mxcli/sdk/workflows" +) + +// ako/mxcli#1771: below 11.2 a user task's targeting is stored as UserSource, +// the shape canon.LowerToProjectShape writes on every unit write. describe +// reads what is stored, so a task written to such a project must read back as +// the targeting the statement gave it — or describe → exec drops it. + +// storedUserTask writes act as a workflow for a project of version pv — the +// encoder, then the lowering and the floor every unit write passes — and reads +// the stored document back. +func storedUserTask(t *testing.T, act *workflows.UserTask, pv *types.ProjectVersion) *workflows.UserTask { + t.Helper() + wf := &workflows.Workflow{ + Name: "WF", + Parameter: &workflows.WorkflowParameter{EntityRef: "M.Ctx"}, + Flow: &workflows.Flow{Activities: []workflows.WorkflowActivity{act}}, + } + raw, err := encodeWorkflow(wf, pv) + if err != nil { + t.Fatalf("encode: %v", err) + } + v := version.Version{Major: pv.MajorVersion, Minor: pv.MinorVersion, Patch: pv.PatchVersion} + if raw, err = canon.LowerToProjectShape(raw, &v); err != nil { + t.Fatalf("%s: lower: %v", v, err) + } + if raw, err = canon.StripUndeclaredProperties(raw, &v); err != nil { + t.Fatalf("%s: floor: %v", v, err) + } + el, err := codec.NewDecoder(codec.DefaultRegistry).Decode(raw) + if err != nil { + t.Fatalf("decode: %v", err) + } + f, _ := el.(*genWf.Workflow).Flow().(*genWf.Flow) + back := workflowFlowFromGen(f) + if len(back.Activities) != 1 { + t.Fatalf("%s: read %d activities, want 1", v, len(back.Activities)) + } + ut, ok := back.Activities[0].(*workflows.UserTask) + if !ok { + t.Fatalf("%s: read back %T, want a user task", v, back.Activities[0]) + } + return ut +} + +func TestWorkflowRead_UserTaskTargetingInTheMendix10Shape(t *testing.T) { + for _, tc := range []struct { + name string + source workflows.UserSource + }{ + {"users microflow", &workflows.MicroflowBasedUserSource{Microflow: "M.ACT_Users"}}, + {"users xpath", &workflows.XPathBasedUserSource{XPath: "[Name = 'admin']"}}, + {"none", &workflows.NoUserSource{}}, + } { + for _, pv := range []*types.ProjectVersion{ + {MajorVersion: 10, MinorVersion: 12, PatchVersion: 19}, + {MajorVersion: 10, MinorVersion: 24, PatchVersion: 24}, + {MajorVersion: 11, MinorVersion: 1}, + {MajorVersion: 11, MinorVersion: 13}, // control: the 11.2 shape + } { + for _, multi := range []bool{false, true} { + ut := &workflows.UserTask{Page: "M.TaskPage", IsMulti: multi, UserSource: tc.source} + ut.Name, ut.Caption = "Review", "Review" + ut.Outcomes = []*workflows.UserTaskOutcome{{Value: "Approve"}, {Value: "Reject"}} + + got := storedUserTask(t, ut, pv) + if got.IsMulti != multi { + t.Errorf("%s %d.%d multi=%v: read back multi=%v", tc.name, pv.MajorVersion, pv.MinorVersion, multi, got.IsMulti) + } + if !reflect.DeepEqual(got.UserSource, tc.source) { + t.Errorf("%s %d.%d multi=%v: targeting read back as %#v, want %#v", + tc.name, pv.MajorVersion, pv.MinorVersion, multi, got.UserSource, tc.source) + } + } + } + } +} diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 7ac90cb7b..4b51ae071 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -59,6 +59,9 @@ import ( // Marketplace's "Example project workflow extension" 2.0.0), and a project // holding the marker does not load (TypeCacheUnknownTypeException). The // floor checks property versions, never a type's own. +// - Workflows$ user task targeting: UserTargeting from 11.2.0, declared on +// the abstract UserTaskActivity; before, UserSource (ako/mxcli#1771). See +// lowerUserTargeting. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -79,9 +82,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), unmarkedPaths: pv.Compare(endOfParallelSplitPathSince) < 0, + userSource: !PropertyDeclared("Workflows$UserTaskActivity", "UserTargeting", *pv), } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths { + !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths && !l.userSource { return contents, nil } var doc bson.D @@ -103,6 +107,7 @@ type shapeLowering struct { flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool flatConstant, variableArgs, unmarkedPaths bool + userSource bool changed bool } @@ -145,6 +150,10 @@ func (l *shapeLowering) walk(v any) (any, error) { if x, err = l.lowerConstantValue(x); err != nil { return nil, err } + case l.userSource && userTaskTypes[typ]: + if x, err = l.lowerUserTargeting(x); err != nil { + return nil, err + } } if l.noFullObjects { if x, err = l.dropForceFullObjects(x, typ); err != nil { diff --git a/modelsdk/canon/versionshape_usertask.go b/modelsdk/canon/versionshape_usertask.go new file mode 100644 index 000000000..8ce65d694 --- /dev/null +++ b/modelsdk/canon/versionshape_usertask.go @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// The workflow user task's older shapes (ako/mxcli#1771), lowered here rather +// than in the workflow encoder because `alter workflow` inserts and edits user +// tasks through the raw mutator, which never passes the encoder: every unit +// write passes LowerToProjectShape. + +// userTaskTypes are the $Types a user task is stored as, in any version. +var userTaskTypes = map[string]bool{ + "Workflows$SingleUserTaskActivity": true, + "Workflows$MultiUserTaskActivity": true, + "Workflows$UserTask": true, +} + +// userSourceOf maps a user targeting (11.2+) to the user source an older +// version stores the same fact in. Measured with `mx convert -p` on +// Studio Pro-authored workflows (Employee Onboarding 9.1.1, Multilevel Dynamic +// Workflow 9.24.3) taken 9.24.30 → 10.24.24 → 11.13.0: the targeting keeps its +// $ID and its one value byte for byte, only its $Type and the key holding it +// change. Group targeting has no user source: it is 11.2, with the groups it +// targets. +var userSourceOf = map[string]string{ + "Workflows$XPathUserTargeting": "Workflows$XPathBasedUserSource", + "Workflows$MicroflowUserTargeting": "Workflows$MicroflowBasedUserSource", + "Workflows$NoUserTargeting": "Workflows$EmptyUserSource", +} + +// emptyUserSourceSince is the version that introduced Workflows$EmptyUserSource, +// a user task with no targeting (mendixmodelsdk 4.117.0: introduced "9.10.0"). +var emptyUserSourceSince = version.Version{Major: 9, Minor: 10} + +// lowerUserTargeting replaces a user task's UserTargeting with the UserSource a +// version before 11.2 stores, in its place. mxcli writes UserTargeting on every +// version, and the floor never saw it: it is declared on the abstract +// UserTaskActivity. 10.12.19 and 10.24.24 then read the task as having no user +// source and report CE1859 "XPath constraint is required" on it, whatever its +// targeting was, and `mx convert` stores an empty XPathBasedUserSource in its +// place — the targeting silently gone. Group targeting is refused, naming the +// task. +func (l *shapeLowering) lowerUserTargeting(task bson.D) (bson.D, error) { + i := shapeIndex(task, "UserTargeting") + if i < 0 { + return task, nil + } + targeting, _ := task[i].Value.(bson.D) + var source bson.D + if targeting != nil { + typ, _ := shapeString(targeting, "$Type") + name, _ := shapeString(task, "Name") + sourceType, ok := userSourceOf[typ] + switch { + case !ok: + return nil, fmt.Errorf("user task %q targets %s, which Mendix introduced in 11.2 (this project is %s): "+ + "a user task there targets users, by XPath or by a microflow", name, targetingKind(typ), l.pv) + case sourceType == "Workflows$EmptyUserSource" && l.pv.Compare(emptyUserSourceSince) < 0: + return nil, fmt.Errorf("user task %q has no targeting, which Mendix introduced in 9.10 (this project is %s): "+ + "give it `targeting users xpath […]` or `targeting users microflow …`", name, l.pv) + } + source = append(bson.D{}, targeting...) + source[shapeIndex(source, "$Type")].Value = sourceType + } + task[i] = bson.E{Key: "UserSource", Value: source} + l.changed = true + // A stored UserSource beside it (an older document an edit gave a + // UserTargeting) is the fact the edit replaced. + for j := range task { + if j != i && task[j].Key == "UserSource" { + return append(append(bson.D{}, task[:j]...), task[j+1:]...), nil + } + } + return task, nil +} + +// targetingKind names a targeting $Type as the statement that writes it does. +func targetingKind(typ string) string { + switch typ { + case "Workflows$XPathGroupTargeting": + return "groups by XPath (`targeting groups xpath`)" + case "Workflows$MicroflowGroupTargeting": + return "groups by a microflow (`targeting groups microflow`)" + } + return "with a " + typ +} diff --git a/modelsdk/mpr/writer_version_shape_1771_test.go b/modelsdk/mpr/writer_version_shape_1771_test.go new file mode 100644 index 000000000..1d1ccd49e --- /dev/null +++ b/modelsdk/mpr/writer_version_shape_1771_test.go @@ -0,0 +1,256 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "fmt" + "sort" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// ako/mxcli#1771: a workflow user task as mxcli builds it is the 11.2 shape, +// and Mendix stored a user task differently before two version boundaries. +// Measured with `mx convert -p` on Studio Pro-authored projects (the +// Marketplace's Employee Onboarding, 9.1.1, XPath targeting; Multilevel Dynamic +// Workflow, 9.24.3, microflow targeting), converted 9.24.30 → 10.24.24 → 11.13.0: +// +// 9.x–10.11 Workflows$UserTask UserSource + UserTaskCompletion +// 10.12–11.1 Workflows$SingleUserTaskActivity UserSource +// 11.2+ Workflows$SingleUserTaskActivity UserTargeting +// +// The targeting value is carried byte-identical across both conversions; only +// its holder changes: XPathBasedUserSource ↔ XPathUserTargeting and +// MicroflowBasedUserSource ↔ MicroflowUserTargeting, each {$ID, $Type, value}. + +var eidSeq byte + +// eid is a distinct, non-zero element $ID, so a test can tell elements apart +// and follow a pointer. +func eid() bson.Binary { + eidSeq++ + return bson.Binary{Subtype: 0, Data: []byte{0x17, 0x71, eidSeq, 3, 4, 5, 0x46, 7, 0x88, 9, 10, 11, 12, 13, 14, eidSeq}} +} + +func xpathUserTargeting(xpath string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$XPathUserTargeting"}, {Key: "XPathConstraint", Value: xpath}} +} + +func microflowUserTargeting(mf string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$MicroflowUserTargeting"}, {Key: "Microflow", Value: mf}} +} + +func noUserTargeting() bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$NoUserTargeting"}} +} + +func stringTemplate(text string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Microflows$StringTemplate"}, + {Key: "Parameters", Value: bson.A{int32(2)}}, {Key: "Text", Value: text}} +} + +func userTaskOutcome(value string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$UserTaskOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: bson.A{int32(3)}}}}, + {Key: "PersistentId", Value: pid()}, {Key: "Value", Value: value}} +} + +// userTaskDoc is a user task as userTaskToGen builds it (below 11.1, where the +// encoder leaves out RelativeMiddlePoint and Size): the keys it writes, in its +// order, with the codec's BoundaryEvents and Annotation defaults last. multi +// adds AwaitAllUsers, a consensus CompletionCriteria falling back to the first +// outcome, and TargetUserInput, as a `multi user task` with no rule. +func userTaskDoc(name string, multi bool, targeting bson.D) bson.D { + typ := "Workflows$SingleUserTaskActivity" + if multi { + typ = "Workflows$MultiUserTaskActivity" + } + approve, reject := userTaskOutcome("Approve"), userTaskOutcome("Reject") + d := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typ}, + {Key: "AutoAssignSingleTargetUser", Value: false}} + if multi { + d = append(d, bson.E{Key: "AwaitAllUsers", Value: false}) + } + d = append(d, bson.E{Key: "Caption", Value: name}) + if multi { + approveID, _ := get(approve, "$ID") + d = append(d, bson.E{Key: "CompletionCriteria", Value: bson.D{{Key: "$ID", Value: eid()}, + {Key: "$Type", Value: "Workflows$ConsensusCompletionCriteria"}, {Key: "FallbackOutcomePointer", Value: approveID}}}) + } + d = append(d, + bson.E{Key: "DueDate", Value: ""}, + bson.E{Key: "Name", Value: name}, + bson.E{Key: "OnCreatedEvent", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$NoEvent"}}}, + bson.E{Key: "Outcomes", Value: bson.A{int32(3), approve, reject}}, + bson.E{Key: "PersistentId", Value: pid()}, + bson.E{Key: "TaskDescription", Value: stringTemplate("")}, + bson.E{Key: "TaskName", Value: stringTemplate(name)}, + bson.E{Key: "TaskPage", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$PageReference"}, {Key: "Page", Value: "M.TaskPage"}}}) + if multi { + d = append(d, bson.E{Key: "TargetUserInput", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$AllUserInput"}}}) + } + return append(d, + bson.E{Key: "UserTargeting", Value: targeting}, + bson.E{Key: "BoundaryEvents", Value: bson.A{int32(2)}}, + bson.E{Key: "Annotation", Value: nil}) +} + +// workflowWithUserTasks wraps tasks in a workflow unit. +func workflowWithUserTasks(tasks ...bson.D) bson.D { + acts := bson.A{int32(3)} + for _, t := range tasks { + acts = append(acts, t) + } + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: acts}}}, + {Key: "Name", Value: "WF"}, + {Key: "Title", Value: "WF"}} +} + +// findNamed returns the first document in v named name. +func findNamed(v any, name string) bson.D { + switch x := v.(type) { + case bson.D: + if n, _ := get(x, "Name"); n == name { + return x + } + for _, e := range x { + if d := findNamed(e.Value, name); d != nil { + return d + } + } + case bson.A: + for _, el := range x { + if d := findNamed(el, name); d != nil { + return d + } + } + } + return nil +} + +// sameID reports whether two stored $IDs (bson.Binary, not comparable with ==) +// are the same. +func sameID(a, b any) bool { + x, ok1 := a.(bson.Binary) + y, ok2 := b.(bson.Binary) + return ok1 && ok2 && bytes.Equal(x.Data, y.Data) +} + +func vlabel(v *version.ProjectVersion) string { + return fmt.Sprintf("%d.%d.%d", v.MajorVersion, v.MinorVersion, v.PatchVersion) +} + +func sortedKeys(d bson.D) string { + ks := keysOf(d) + sort.Strings(ks) + return strings.Join(ks, ",") +} + +// User targeting is UserTargeting from 11.2 and UserSource before it +// (mendixmodelsdk: UserTaskActivity.userSource deleted, userTargeting +// introduced 11.2.0). UserTargeting is declared on the abstract +// UserTaskActivity, which the floor does not look up, so it was written to +// every version: on 10.12.19 and 10.24.24 the task then had no UserSource, and +// mx check reported CE1859 "XPath constraint is required" on every user task — +// the microflow-targeted ones too — while `mx convert` 10.24.24 dropped the +// targeting and stored an empty XPathBasedUserSource in its place. +func TestNormalize_UserTaskTargetingBelowMendix11_2(t *testing.T) { + kinds := []struct { + name string + targeting func() bson.D + sourceType string + valueKey string + value string + }{ + {"users xpath", func() bson.D { return xpathUserTargeting("[Name = 'admin']") }, "Workflows$XPathBasedUserSource", "XPathConstraint", "[Name = 'admin']"}, + {"users microflow", func() bson.D { return microflowUserTargeting("M.ACT_GetUsers") }, "Workflows$MicroflowBasedUserSource", "Microflow", "M.ACT_GetUsers"}, + {"none", noUserTargeting, "Workflows$EmptyUserSource", "", ""}, + } + for _, tt := range []struct { + v *version.ProjectVersion + lowered bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 12, 19), true}, + {pv(10, 24, 24), true}, + {pv(11, 1, 0), true}, + {pv(11, 2, 0), false}, + {pv(11, 13, 0), false}, // control + } { + for _, k := range kinds { + for _, multi := range []bool{false, true} { + targeting := k.targeting() + wantID, _ := get(targeting, "$ID") + d, err := normalizeShape(t, tt.v, workflowWithUserTasks(userTaskDoc("Task", multi, targeting))) + label := vlabel(tt.v) + " " + k.name + if multi { + label += " (multi)" + } + if err != nil { + t.Errorf("%s: refused: %v", label, err) + continue + } + task := findNamed(d, "Task") + _, hasTargeting := get(task, "UserTargeting") + srcAny, hasSource := get(task, "UserSource") + if !tt.lowered { + if !hasTargeting || hasSource { + t.Errorf("%s: keys %v, want UserTargeting and no UserSource", label, keysOf(task)) + } + continue + } + if hasTargeting || !hasSource { + t.Errorf("%s: keys %v, want UserSource and no UserTargeting", label, keysOf(task)) + continue + } + src, _ := srcAny.(bson.D) + if typ, _ := get(src, "$Type"); typ != k.sourceType { + t.Errorf("%s: UserSource is %v, want %s", label, typ, k.sourceType) + } + if id, _ := get(src, "$ID"); !sameID(id, wantID) { + t.Errorf("%s: UserSource $ID %v, want the targeting's own %v", label, id, wantID) + } + wantKeys := "$ID,$Type" + if k.valueKey != "" { + wantKeys += "," + k.valueKey + if v, _ := get(src, k.valueKey); v != k.value { + t.Errorf("%s: %s = %q, want %q", label, k.valueKey, v, k.value) + } + } + if got := strings.Join(keysOf(src), ","); got != wantKeys { + t.Errorf("%s: UserSource keys %s, want %s", label, got, wantKeys) + } + } + } + } +} + +// Group targeting is 11.2 (XPathGroupTargeting, MicroflowGroupTargeting, and +// the workflow groups they target): before, a user task targets users only. +// Refused naming the task, not written as a targeting the version reads as +// "XPath constraint is required". +func TestNormalize_RefusesGroupTargetingBelowMendix11_2(t *testing.T) { + group := func(typ, key, value string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typ}, {Key: key, Value: value}} + } + for _, g := range []bson.D{ + group("Workflows$XPathGroupTargeting", "XPathConstraint", "[Name = 'Reviewers']"), + group("Workflows$MicroflowGroupTargeting", "Microflow", "M.ACT_GetGroups"), + } { + for _, v := range []*version.ProjectVersion{pv(10, 24, 24), pv(11, 1, 0)} { + _, err := normalizeShape(t, v, workflowWithUserTasks(userTaskDoc("ByGroup", false, g))) + if err == nil || !strings.Contains(err.Error(), "ByGroup") || !strings.Contains(err.Error(), "11.2") { + t.Errorf("%s: group targeting on %s was not refused naming the task and 11.2: %v", g[1].Value, vlabel(v), err) + } + } + if _, err := normalizeShape(t, pv(11, 2, 0), workflowWithUserTasks(userTaskDoc("ByGroup", false, g))); err != nil { + t.Errorf("%s: refused on 11.2: %v", g[1].Value, err) + } + } +} From f9fcc87b498d35ed2c303fa5c6ef394f5662f1dd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:29:20 +0000 Subject: [PATCH 36/61] fix(canon): write a user task as a Workflows$UserTask below Mendix 10.12 Symptom (ako/mxcli#1771): on Mendix 9.x a workflow with a user task made the project unloadable. exec reported "Created workflow", then mx check 9.24.30 died with "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". Cause: mxcli writes Workflows$SingleUserTaskActivity and MultiUserTaskActivity. Mendix introduced both types in 10.12.0 and deleted Workflows$UserTask there (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared a written $Type with the project's version. Older shape, measured: Studio Pro 9.24.3 (the Marketplace's Multilevel Dynamic Workflow) stores {$ID, $Type, Annotation, AutoAssignSingleTargetUser, Caption, DueDate, Name, OnCreatedEvent, Outcomes, TaskDescription, TaskName, TaskPage, UserSource, UserTaskCompletion}. The completion (9.22) says which kind of task it is. A single-user task has a bare SingleInputCompletion. A multi-user task has a MultiInputCompletion holding its CompletionCriteria, its TargetUserInput and, from 10.2, AwaitAllUsers. Fix: canon.LowerToProjectShape rewrites the task as that UserTask below 10.12. The multi-user settings move into the completion, and the outcomes stay on the task, so the rule's outcome pointers still resolve. The completion's $ID is derived from the task's. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, majority and threshold below 10.0, a microflow rule below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. An empty BoundaryEvents list is dropped below 10.14; it is declared on the abstract UserTaskActivity, which the floor never saw. describe reads a 9.x multi-user task's completion, and the create-or-modify guard now sees its rule and participants too, so a rewrite no longer resets them silently. Measured on fresh apps, exec of the bug-test then mx check: - 9.24.30.51827: before TypeCacheUnknownTypeException, after 0 errors. mx convert 9.24.30 changes no key and no value. describe -> exec "Unchanged workflow". After mx convert 10.24.24, the user tasks are identical in keys and values to mxcli's direct 10.24 output. - 10.24.24 and 11.13.0: 0 errors. Control: stubbing lowerUserTaskType fails the three normalize tests on 9.24 with the 10.12 types in place. The read test and the guard test failed before their fixes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...2026-10-29-user-task-type-below-10-12.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../bug-tests/1771-user-task-below-10-12.mdl | 80 ++++++ mdl/backend/modelsdk/workflow_read.go | 8 + .../modelsdk/workflow_usertask_shape_test.go | 54 ++++ mdl/executor/validate_workflow_rewrite.go | 14 +- .../validate_workflow_rewrite_1771_test.go | 54 ++++ modelsdk/canon/versionshape.go | 19 +- modelsdk/canon/versionshape_usertask.go | 149 +++++++++++ .../writer_version_shape_1771_type_test.go | 231 ++++++++++++++++++ 11 files changed, 608 insertions(+), 5 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json create mode 100644 mdl-examples/bug-tests/1771-user-task-below-10-12.mdl create mode 100644 mdl/executor/validate_workflow_rewrite_1771_test.go create mode 100644 modelsdk/mpr/writer_version_shape_1771_type_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json new file mode 100644 index 000000000..6421029ff --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 9.x a workflow with a user task made the project unloadable. exec reported 'Created workflow', then mx check 9.24.30 and Studio Pro died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity`. The same happened on 10.0–10.11 by the SDK's version data, which was not measured there", "cause": "userTaskToGen (and the ALTER mutator's inserts, through it) writes Workflows$SingleUserTaskActivity / MultiUserTaskActivity, TYPES introduced in 10.12.0, which also deleted Workflows$UserTask (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared the written $Type with the project's version. Before 10.12 the kind of task is the UserTask's UserTaskCompletion (9.22): SingleInputCompletion, or MultiInputCompletion {CompletionCriteria, TargetUserInput, AwaitAllUsers from 10.2}", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTaskType, dropTaskBoundaryEvents, userTaskActivitySince, completionCriteriaSince), `mdl/backend/modelsdk/workflow_read.go` (UserTask case reads the MultiInputCompletion), `mdl/executor/validate_workflow_rewrite.go` (rawMultiUserTaskSettings sees a 9.x multi-user task)", "insight": "**A type rename can also be a restructure.** Converting MultiUserTaskActivity down is more than a $Type swap. Three of its keys move into a nested MultiInputCompletion, and the outcomes stay on the task, so the CompletionCriteria's outcome pointers still resolve. Studio Pro 9.24.3's key set is the target: Multilevel Dynamic Workflow (Marketplace 221741) stores `$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent,Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion`. **With no Studio Pro sample of a 9.x multi-user task, convert UP as the oracle.** The 9.24 output, after `mx convert -p` 10.24.24, has user tasks identical in keys and values to what mxcli writes on 10.24 directly, rule and participants included. **Every reader of the old shape needs the restructure too.** describe read a 9.x multi-user task as a single one, and the create-or-modify guard looked for the rule only on MultiUserTaskActivity, so a 9.24 rewrite would reset the rule silently. **Give an element the lowering ADDS a derived $ID** (completionID from the task's $ID), so the same task lowers to the same bytes. The boundary events on a user task (10.14, declared on the abstract UserTaskActivity) slipped past the floor the same way: an empty list is dropped below 10.14, and a boundary event is refused. Measured: bug-test on fresh apps. 9.24.30 before TypeCacheUnknownTypeException, after 0 errors. `mx convert -p` 9.24.30 changes no key and no value. describe → exec 'Unchanged workflow'. 10.24.24 and 11.13.0 give 0 errors. The CE2513 'Java 11' after converting up is in an untouched 9.24 app too. Control: stubbing lowerUserTaskType fails TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12, …MultiUserTask… and …RefusesWhatAnOlderUserTaskCannotHold with the 10.12 types on 9.24. The reader and guard tests failed before their fixes. Tests: those three, TestNormalize_UserTaskCompletionIDIsStable, TestWorkflowRead_UserTaskInTheMendix9Shape, TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings; mdl-examples/bug-tests/1771-user-task-below-10-12.mdl", "refs": ["#1771", "#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index eab978aac..f210553a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 0af20a6fa..f5c8c1325 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -187,6 +187,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | | The end of a parallel split path (`Workflows$EndOfParallelSplitPathActivity`, a type introduced in 10.21) | 10.21 | Not stored: a path holds its activities and nothing more (Studio Pro 9.4) | +| A user task (`Workflows$SingleUserTaskActivity` / `MultiUserTaskActivity`, types introduced in 10.12) | 10.12 | A `Workflows$UserTask` whose `UserTaskCompletion` is a `SingleInputCompletion`, or a `MultiInputCompletion` holding the rule, the participants and, from 10.2, `AwaitAllUsers` (Studio Pro 9.24, [ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)) | | A user task's targeting (`targeting users xpath […]`, `targeting users microflow …`, none) | 11.2 | A `UserSource` (`XPathBasedUserSource`, `MicroflowBasedUserSource`, `EmptyUserSource`) holding the same value, instead of a `UserTargeting` ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)) | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | | A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | @@ -199,6 +200,7 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a private constant value in a configuration below 10.9; - a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; - a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); +- a multi-user task below 9.22, `decide by majority` or `decide by threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14 ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl new file mode 100644 index 000000000..bf4fe380e --- /dev/null +++ b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl @@ -0,0 +1,80 @@ +mdl 1; +-- ako/mxcli#1771: on Mendix 9.x a workflow with a user task made the project +-- unloadable. `exec` reported "Created workflow", then mx check and Studio Pro +-- could not load the project at all: +-- +-- TypeCacheUnknownTypeException: The type cache does not contain a type with +-- qualified name Workflows$SingleUserTaskActivity. +-- +-- mxcli writes a user task as a Workflows$SingleUserTaskActivity or +-- MultiUserTaskActivity. Both types arrived in 10.12, which also deleted the +-- one before them (mendixmodelsdk). Before 10.12 a user task is a +-- Workflows$UserTask, and its UserTaskCompletion (9.22) says whether it is a +-- single-user or a multi-user task. Studio Pro 9.24.3 stores `{…, UserSource, +-- UserTaskCompletion: SingleInputCompletion}` (the Marketplace's Multilevel +-- Dynamic Workflow). A multi-user task's rule, participants and, from 10.2, +-- `await all users` sit in a MultiInputCompletion, and its outcomes stay on the +-- task. What the older type cannot hold is refused, naming the task: a rule +-- added later (majority and threshold in 10.0, microflow in 10.3), `await all +-- users` below 10.2, and a boundary event on a user task below 10.14. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`), exec then +-- `mxcli docker check`: +-- 9.24.30.51827 before: TypeCacheUnknownTypeException. After: 0 errors. +-- `mx convert -p` 9.24.30 changes no key and no value. +-- describe → exec prints "Unchanged workflow". Converted up +-- to 10.24.24, the tasks are those mxcli writes on 10.24 +-- directly. +-- 10.24.24, 11.13.0 0 errors, with the same key sets as before the fix. + +create module Probe1771Ut; + +create persistent entity Probe1771Ut.Request ( Title: String(100) ); + +create microflow Probe1771Ut.ACT_GetUsers ($Workflow: System.Workflow, $Context: Probe1771Ut.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create page Probe1771Ut.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771Ut.WF_Tasks + parameter $WorkflowContext: Probe1771Ut.Request + display 'Tasks' +begin + user task Review 'Review' + page Probe1771Ut.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Approve' { } 'Reject' { }; + user task Handle 'Handle' + page Probe1771Ut.TaskPage + targeting users microflow Probe1771Ut.ACT_GetUsers + outcomes 'Done' { }; + multi user task Vote 'Vote' + page Probe1771Ut.TaskPage + targeting users xpath [Name = 'admin'] + participants 2 + decide by veto 'Reject' + outcomes 'Approve' { } 'Reject' { }; + multi user task Agree 'Agree' + page Probe1771Ut.TaskPage + targeting users microflow Probe1771Ut.ACT_GetUsers + outcomes 'Yes' { } 'No' { }; +end workflow; + +-- ALTER inserts a user task through the mutator, outside CREATE's encoder; the +-- lowering runs on every unit write, so it covers both. +alter workflow Probe1771Ut.WF_Tasks { + insert after Handle { user task Confirm 'Confirm' + page Probe1771Ut.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'OK' { }; } +}; diff --git a/mdl/backend/modelsdk/workflow_read.go b/mdl/backend/modelsdk/workflow_read.go index 5da51c1cd..31e79e9f3 100644 --- a/mdl/backend/modelsdk/workflow_read.go +++ b/mdl/backend/modelsdk/workflow_read.go @@ -192,6 +192,14 @@ func workflowActivityFromGen(el element.Element) workflows.WorkflowActivity { t.Page = taskPageName(a.TaskPage()) } t.Outcomes = userTaskOutcomesFromGen(a.OutcomesItems()) + // Below 10.12 a multi-user task is a UserTask whose completion holds + // what MultiUserTaskActivity holds itself from 10.12 (ako/mxcli#1771). + if mc, ok := a.UserTaskCompletion().(*genWf.MultiInputCompletion); ok && mc != nil { + t.IsMulti = true + t.AwaitAllUsers = mc.AwaitAllUsers() + t.CompletionCriteria = completionCriteriaFromGen(mc.CompletionCriteria(), t.Outcomes) + t.TargetUserInput = targetUserInputFromGen(mc.TargetUserInput()) + } return t case *genWf.SingleUserTaskActivity: t := &workflows.UserTask{ diff --git a/mdl/backend/modelsdk/workflow_usertask_shape_test.go b/mdl/backend/modelsdk/workflow_usertask_shape_test.go index 34d2ce89f..21914ac10 100644 --- a/mdl/backend/modelsdk/workflow_usertask_shape_test.go +++ b/mdl/backend/modelsdk/workflow_usertask_shape_test.go @@ -56,6 +56,60 @@ func storedUserTask(t *testing.T, act *workflows.UserTask, pv *types.ProjectVers return ut } +// Below 10.12 a user task is a Workflows$UserTask, and a multi-user task's +// rule, participants and `await all users` live in its UserTaskCompletion (a +// Workflows$MultiInputCompletion). Each reads back as the statement gave it, +// as on 10.12 and later. +func TestWorkflowRead_UserTaskInTheMendix9Shape(t *testing.T) { + outcomes := func() []*workflows.UserTaskOutcome { + return []*workflows.UserTaskOutcome{{Value: "Approve"}, {Value: "Reject"}} + } + for _, tc := range []struct { + name string + pv *types.ProjectVersion + multi bool + cc *workflows.CompletionCriteria + input *workflows.TargetUserInput + await bool + }{ + {"single", &types.ProjectVersion{MajorVersion: 9, MinorVersion: 24, PatchVersion: 30}, false, nil, nil, false}, + {"consensus", &types.ProjectVersion{MajorVersion: 9, MinorVersion: 24, PatchVersion: 30}, true, + &workflows.CompletionCriteria{Kind: "Consensus", FallbackOutcome: "Reject"}, &workflows.TargetUserInput{Kind: "All"}, false}, + {"veto, two participants", &types.ProjectVersion{MajorVersion: 9, MinorVersion: 24, PatchVersion: 30}, true, + &workflows.CompletionCriteria{Kind: "Veto", VetoOutcome: "Reject"}, &workflows.TargetUserInput{Kind: "Absolute", Amount: 2}, false}, + {"majority, await all", &types.ProjectVersion{MajorVersion: 10, MinorVersion: 6}, true, + &workflows.CompletionCriteria{Kind: "Majority", CompletionType: "Absolute", FallbackOutcome: "Approve"}, + &workflows.TargetUserInput{Kind: "Percentage", Percentage: 50}, true}, + {"control: 10.24", &types.ProjectVersion{MajorVersion: 10, MinorVersion: 24, PatchVersion: 24}, true, + &workflows.CompletionCriteria{Kind: "Veto", VetoOutcome: "Reject"}, &workflows.TargetUserInput{Kind: "Absolute", Amount: 2}, true}, + } { + ut := &workflows.UserTask{Page: "M.TaskPage", IsMulti: tc.multi, Outcomes: outcomes(), + UserSource: &workflows.XPathBasedUserSource{XPath: "[Name = 'admin']"}, + CompletionCriteria: tc.cc, TargetUserInput: tc.input, AwaitAllUsers: tc.await} + ut.Name, ut.Caption = "Review", "Review" + + got := storedUserTask(t, ut, tc.pv) + if got.IsMulti != tc.multi { + t.Errorf("%s: read back multi=%v, want %v", tc.name, got.IsMulti, tc.multi) + } + if !reflect.DeepEqual(got.UserSource, ut.UserSource) { + t.Errorf("%s: targeting read back as %#v", tc.name, got.UserSource) + } + if !tc.multi { + continue + } + if got.CompletionCriteria == nil || *got.CompletionCriteria != *tc.cc { + t.Errorf("%s: rule read back as %+v, want %+v", tc.name, got.CompletionCriteria, tc.cc) + } + if got.TargetUserInput == nil || *got.TargetUserInput != *tc.input { + t.Errorf("%s: participants read back as %+v, want %+v", tc.name, got.TargetUserInput, tc.input) + } + if got.AwaitAllUsers != tc.await { + t.Errorf("%s: await all users read back as %v, want %v", tc.name, got.AwaitAllUsers, tc.await) + } + } +} + func TestWorkflowRead_UserTaskTargetingInTheMendix10Shape(t *testing.T) { for _, tc := range []struct { name string diff --git a/mdl/executor/validate_workflow_rewrite.go b/mdl/executor/validate_workflow_rewrite.go index 6cd7d63ee..ad80144ce 100644 --- a/mdl/executor/validate_workflow_rewrite.go +++ b/mdl/executor/validate_workflow_rewrite.go @@ -347,7 +347,19 @@ func rawAwaitsAllUsers(d map[string]any) bool { // resets when the statement omits it. func rawMultiUserTaskSettings(raw map[string]any) (rules, participants, await []string) { walkRawDocs(raw, func(d map[string]any) { - if t, _ := d["$Type"].(string); t != "Workflows$MultiUserTaskActivity" { + switch t, _ := d["$Type"].(string); t { + case "Workflows$MultiUserTaskActivity": + case "Workflows$UserTask": + // Below 10.12 a multi-user task is a UserTask, and its settings live + // on its MultiInputCompletion (ako/mxcli#1771). + c, _ := d["UserTaskCompletion"].(map[string]any) + if ct, _ := c["$Type"].(string); ct != "Workflows$MultiInputCompletion" { + return + } + d = map[string]any{"Name": d["Name"], "Outcomes": d["Outcomes"], + "CompletionCriteria": c["CompletionCriteria"], "TargetUserInput": c["TargetUserInput"], + "AwaitAllUsers": c["AwaitAllUsers"]} + default: return } if r := rawCompletionRule(d); r != "" { diff --git a/mdl/executor/validate_workflow_rewrite_1771_test.go b/mdl/executor/validate_workflow_rewrite_1771_test.go new file mode 100644 index 000000000..e2c653b1b --- /dev/null +++ b/mdl/executor/validate_workflow_rewrite_1771_test.go @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// mendix9MultiUserTaskWorkflow is a workflow as a Mendix 9.x–10.11 project +// stores it (ako/mxcli#1771): a multi-user task is a Workflows$UserTask whose +// UserTaskCompletion, a MultiInputCompletion, holds its rule and participants; +// a single-user task's is a SingleInputCompletion. +func mendix9MultiUserTaskWorkflow() map[string]any { + first := map[string]any{"$ID": "outcome-fast", "$Type": "Workflows$UserTaskOutcome", "Value": "Fast"} + second := map[string]any{"$ID": "outcome-furious", "$Type": "Workflows$UserTaskOutcome", "Value": "Furious"} + return map[string]any{ + "$Type": "Workflows$Workflow", + "Flow": map[string]any{"$Type": "Workflows$Flow", "Activities": []any{ + 3, + map[string]any{"$Type": "Workflows$UserTask", "Name": "userTask1", + "OnCreatedEvent": map[string]any{"$Type": "Workflows$NoEvent"}, + "UserTaskCompletion": map[string]any{"$Type": "Workflows$SingleInputCompletion"}}, + map[string]any{"$Type": "Workflows$UserTask", "Name": "userTask2", + "OnCreatedEvent": map[string]any{"$Type": "Workflows$NoEvent"}, + "Outcomes": []any{3, first, second}, + "UserTaskCompletion": map[string]any{"$Type": "Workflows$MultiInputCompletion", + "CompletionCriteria": map[string]any{"$Type": "Workflows$VetoCompletionCriteria", "VetoOutcomePointer": "outcome-furious"}, + "TargetUserInput": map[string]any{"$Type": "Workflows$AbsoluteAmountUserInput", "Amount": 2}}}, + }}, + } +} + +// The rewrite guard refuses a statement that would reset a stored multi-user +// task's rule or participants. Below 10.12 they are stored on the task's +// UserTaskCompletion, where the guard did not look, so a rewrite on 9.24 reset +// both to the defaults silently. +func TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings(t *testing.T) { + stmt := parseWorkflowStmt(t, wfRewriteTwoTasks) + err := checkNoDroppedWorkflowConstructs(rawWorkflowCtx(t, mendix9MultiUserTaskWorkflow()), "wf1", "M.W", stmt) + if err == nil || !strings.Contains(err.Error(), "veto") { + t.Fatalf("a rewrite dropping a 9.x multi-user task's veto rule must be refused naming it, got %v", err) + } + + // Control: restating the rule and the participants loses nothing. + restated := parseWorkflowStmt(t, `create or modify workflow M.W parameter $C: M.Ctx +begin + user task userTask1 'User Task' page M.P outcomes 'Good' { } 'Bad' { }; + multi user task userTask2 'Multi' page M.P participants 2 decide by veto 'Furious' outcomes 'Fast' { } 'Furious' { }; +end workflow;`) + if err := checkNoDroppedWorkflowConstructs(rawWorkflowCtx(t, mendix9MultiUserTaskWorkflow()), "wf1", "M.W", restated); err != nil { + t.Fatalf("a rewrite that restates the rule and participants must be allowed, got %v", err) + } +} diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 4b51ae071..fbf5097cd 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -62,6 +62,9 @@ import ( // - Workflows$ user task targeting: UserTargeting from 11.2.0, declared on // the abstract UserTaskActivity; before, UserSource (ako/mxcli#1771). See // lowerUserTargeting. +// - Workflows$SingleUserTaskActivity and MultiUserTaskActivity, TYPES +// introduced in 10.12.0; before, a Workflows$UserTask whose +// UserTaskCompletion says which (ako/mxcli#1771). See lowerUserTaskType. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -83,15 +86,19 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), unmarkedPaths: pv.Compare(endOfParallelSplitPathSince) < 0, userSource: !PropertyDeclared("Workflows$UserTaskActivity", "UserTargeting", *pv), + taskBoundaries: !PropertyDeclared("Workflows$UserTaskActivity", "BoundaryEvents", *pv), + userTaskType: pv.Compare(userTaskActivitySince) < 0, } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths && !l.userSource { + !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths && !l.userSource && + !l.taskBoundaries && !l.userTaskType { return contents, nil } var doc bson.D if err := bson.Unmarshal(contents, &doc); err != nil { return nil, err } + l.root = doc out, err := l.walk(doc) if err != nil { return nil, err @@ -107,8 +114,12 @@ type shapeLowering struct { flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool flatConstant, variableArgs, unmarkedPaths bool - userSource bool + userSource, taskBoundaries, userTaskType bool changed bool + // root is the unit, and used the $IDs it holds, for an element the + // lowering adds (see lowerUserTaskType). + root bson.D + used map[string]bool } // endOfParallelSplitPathSince is the version that introduced the type @@ -150,8 +161,8 @@ func (l *shapeLowering) walk(v any) (any, error) { if x, err = l.lowerConstantValue(x); err != nil { return nil, err } - case l.userSource && userTaskTypes[typ]: - if x, err = l.lowerUserTargeting(x); err != nil { + case userTaskTypes[typ]: + if x, err = l.lowerUserTask(x, typ); err != nil { return nil, err } } diff --git a/modelsdk/canon/versionshape_usertask.go b/modelsdk/canon/versionshape_usertask.go index 8ce65d694..d436e4e4b 100644 --- a/modelsdk/canon/versionshape_usertask.go +++ b/modelsdk/canon/versionshape_usertask.go @@ -39,6 +39,155 @@ var userSourceOf = map[string]string{ // a user task with no targeting (mendixmodelsdk 4.117.0: introduced "9.10.0"). var emptyUserSourceSince = version.Version{Major: 9, Minor: 10} +// userTaskActivitySince is the version that introduced the types +// Workflows$SingleUserTaskActivity and MultiUserTaskActivity, and deleted +// Workflows$UserTask (mendixmodelsdk 4.117.0). Written down here because the +// generated version data holds property versions only, never a type's own. +var userTaskActivitySince = version.Version{Major: 10, Minor: 12} + +// completionCriteriaSince holds the multi-user task rules introduced after the +// multi-user task itself (9.22.0, with consensus and veto): mendixmodelsdk +// 4.117.0 StructureVersionInfo. +var completionCriteriaSince = map[string]version.Version{ + "Workflows$MajorityCompletionCriteria": {Major: 10}, + "Workflows$ThresholdCompletionCriteria": {Major: 10}, + "Workflows$MicroflowCompletionCriteria": {Major: 10, Minor: 3}, +} + +// completionRule names a completion criteria $Type as the statement that +// writes it does. +var completionRule = map[string]string{ + "Workflows$MajorityCompletionCriteria": "decide by majority", + "Workflows$ThresholdCompletionCriteria": "decide by threshold", + "Workflows$MicroflowCompletionCriteria": "decide by microflow", +} + +// lowerUserTask writes a user task in the shape the project's version stores: +// its targeting, its boundary events and, below 10.12, its type. +func (l *shapeLowering) lowerUserTask(task bson.D, typ string) (bson.D, error) { + var err error + if l.userSource { + if task, err = l.lowerUserTargeting(task); err != nil { + return nil, err + } + } + if l.taskBoundaries { + if task, err = l.dropTaskBoundaryEvents(task); err != nil { + return nil, err + } + } + if l.userTaskType && typ != "Workflows$UserTask" { + return l.lowerUserTaskType(task, typ) + } + return task, nil +} + +// dropTaskBoundaryEvents removes a user task's BoundaryEvents below 10.14, +// where a user task declares none (mendixmodelsdk: UserTaskActivity +// .boundaryEvents introduced 10.14.0; a Workflows$UserTask has none at all). +// The codec writes an empty list on every task, which is dropped; a boundary +// event is refused, naming the task. The floor never saw the key: it is +// declared on the abstract UserTaskActivity. +func (l *shapeLowering) dropTaskBoundaryEvents(task bson.D) (bson.D, error) { + i := shapeIndex(task, "BoundaryEvents") + if i < 0 { + return task, nil + } + if !isEmptyValue(task[i].Value) { + name, _ := shapeString(task, "Name") + return nil, fmt.Errorf("user task %q has a boundary event, which Mendix introduced on user tasks in 10.14 "+ + "(this project is %s)", name, l.pv) + } + l.changed = true + return append(append(bson.D{}, task[:i]...), task[i+1:]...), nil +} + +// lowerUserTaskType rewrites a Workflows$SingleUserTaskActivity or +// MultiUserTaskActivity as the Workflows$UserTask a version before 10.12 +// stores. Studio Pro 9.24.3 stores `{…, UserSource, UserTaskCompletion}` +// (Multilevel Dynamic Workflow, Marketplace 221741), and the completion says +// which kind of task it is (9.22.0): +// +// - a single-user task: a bare Workflows$SingleInputCompletion; +// - a multi-user task: a Workflows$MultiInputCompletion holding what the +// MultiUserTaskActivity holds itself from 10.12: CompletionCriteria (its +// outcome pointers unchanged, the outcomes stay on the task), +// TargetUserInput and, from 10.2.0, AwaitAllUsers. +// +// A 9.24.30 project holding the 10.12 types does not load +// (TypeCacheUnknownTypeException). What the older type cannot hold is refused, +// naming the task: a multi-user task below 9.22, a rule introduced later +// (majority and threshold 10.0, microflow 10.3), and `await all users` below +// 10.2. The completion's $ID is derived from the task's, so the same task +// lowers to the same bytes. +func (l *shapeLowering) lowerUserTaskType(task bson.D, typ string) (bson.D, error) { + name, _ := shapeString(task, "Name") + hasCompletion := PropertyDeclared("Workflows$UserTask", "UserTaskCompletion", l.pv) + completion := bson.D{{Key: "$ID", Value: l.derivedID(task, "UserTaskCompletion")}} + if typ == "Workflows$MultiUserTaskActivity" { + if !hasCompletion { + return nil, fmt.Errorf("multi user task %q: Mendix introduced multi-user tasks in 9.22 (this project is %s)", + name, l.pv) + } + completion = append(completion, bson.E{Key: "$Type", Value: "Workflows$MultiInputCompletion"}) + var await, criteria, input any + task, await = takeKey(task, "AwaitAllUsers") + task, criteria = takeKey(task, "CompletionCriteria") + task, input = takeKey(task, "TargetUserInput") + if PropertyDeclared("Workflows$MultiInputCompletion", "AwaitAllUsers", l.pv) { + completion = append(completion, bson.E{Key: "AwaitAllUsers", Value: await == true}) + } else if await == true { + return nil, fmt.Errorf("multi user task %q waits for all users, which Mendix introduced in 10.2 (this project is %s)", + name, l.pv) + } + criteriaDoc, _ := criteria.(bson.D) + ct, _ := shapeString(criteriaDoc, "$Type") + if since, ok := completionCriteriaSince[ct]; ok && l.pv.Compare(since) < 0 { + return nil, fmt.Errorf("multi user task %q is decided by %s, which Mendix introduced in %d.%d (this project is %s)", + name, completionRule[ct], since.Major, since.Minor, l.pv) + } + completion = append(completion, + bson.E{Key: "CompletionCriteria", Value: criteria}, + bson.E{Key: "TargetUserInput", Value: input}) + } else { + completion = append(completion, bson.E{Key: "$Type", Value: "Workflows$SingleInputCompletion"}) + } + task[shapeIndex(task, "$Type")].Value = "Workflows$UserTask" + l.changed = true + if !hasCompletion { + return task, nil + } + // In Studio Pro's place: after the user source, its neighbour in the + // alphabetical order Studio Pro writes. + at := len(task) + if i := shapeIndex(task, "UserSource"); i >= 0 { + at = i + 1 + } + out := append(append(bson.D{}, task[:at]...), bson.E{Key: "UserTaskCompletion", Value: completion}) + return append(out, task[at:]...), nil +} + +// takeKey removes key from d, returning d without it and the value it held. +func takeKey(d bson.D, key string) (bson.D, any) { + i := shapeIndex(d, key) + if i < 0 { + return d, nil + } + v := d[i].Value + return append(append(bson.D{}, d[:i]...), d[i+1:]...), v +} + +// derivedID is an $ID for an element the lowering adds to owner, derived from +// the owner's $ID and the key it is added under, as CompletePropertySets +// derives the ones it adds: the same owner always gets the same id, and one +// the unit already holds is stepped past. +func (l *shapeLowering) derivedID(owner bson.D, key string) bson.Binary { + if l.used == nil { + l.used = elementIDs(l.root) + } + return completionID(elementIDBytes(owner), "LowerToProjectShape/"+key, l.used) +} + // lowerUserTargeting replaces a user task's UserTargeting with the UserSource a // version before 11.2 stores, in its place. mxcli writes UserTargeting on every // version, and the floor never saw it: it is declared on the abstract diff --git a/modelsdk/mpr/writer_version_shape_1771_type_test.go b/modelsdk/mpr/writer_version_shape_1771_type_test.go new file mode 100644 index 000000000..29c747732 --- /dev/null +++ b/modelsdk/mpr/writer_version_shape_1771_type_test.go @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// studioPro924UserTask is the key set Studio Pro 9.24.3 stores on a user task +// (Multilevel Dynamic Workflow, Marketplace 221741). Employee Onboarding +// (9.1.1), converted by mx convert 9.24.30, stores the same set. +const studioPro924UserTask = "$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent," + + "Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion" + +// Below 10.12 a user task is a Workflows$UserTask. mendixmodelsdk deletes +// UserTask and introduces SingleUserTaskActivity and MultiUserTaskActivity in +// 10.12.0. Whether the task is single- or multi-user is its UserTaskCompletion +// (9.22.0): a SingleInputCompletion, or a MultiInputCompletion holding the +// completion criteria, the participants and, from 10.2.0, AwaitAllUsers. +// mxcli wrote the 10.12 types on every version, and 9.24.30 then did not load +// the project (TypeCacheUnknownTypeException … Workflows$SingleUserTaskActivity). +func TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + userTask bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 6, 0), true}, + {pv(10, 11, 0), true}, + {pv(10, 12, 0), false}, + {pv(10, 24, 24), false}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")))) + if err != nil { + t.Errorf("%s: refused: %v", vlabel(tt.v), err) + continue + } + task := findNamed(d, "Single") + typ, _ := get(task, "$Type") + if !tt.userTask { + if typ != "Workflows$SingleUserTaskActivity" { + t.Errorf("%s: $Type %v, want Workflows$SingleUserTaskActivity", vlabel(tt.v), typ) + } + if _, ok := get(task, "UserTaskCompletion"); ok { + t.Errorf("%s: a SingleUserTaskActivity holds a UserTaskCompletion", vlabel(tt.v)) + } + continue + } + if typ != "Workflows$UserTask" { + t.Errorf("%s: $Type %v, want Workflows$UserTask", vlabel(tt.v), typ) + } + if got := sortedKeys(task); got != studioPro924UserTask { + t.Errorf("%s: user task keys\n %s\nwant Studio Pro 9.24's\n %s", vlabel(tt.v), got, studioPro924UserTask) + } + completion, _ := get(task, "UserTaskCompletion") + cd, _ := completion.(bson.D) + if ct, _ := get(cd, "$Type"); ct != "Workflows$SingleInputCompletion" || len(cd) != 2 { + t.Errorf("%s: UserTaskCompletion %v, want a bare Workflows$SingleInputCompletion", vlabel(tt.v), cd) + } + if id, _ := get(cd, "$ID"); !nonZeroID(id) { + t.Errorf("%s: UserTaskCompletion has no $ID: %v", vlabel(tt.v), id) + } + } +} + +// The completion's $ID is derived from the task's, so writing the same task +// twice produces the same bytes. +func TestNormalize_UserTaskCompletionIDIsStable(t *testing.T) { + doc := workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']"))) + a, err := normalizeShape(t, pv(9, 24, 30), doc) + if err != nil { + t.Fatal(err) + } + b, err := normalizeShape(t, pv(9, 24, 30), doc) + if err != nil { + t.Fatal(err) + } + ida, _ := get(find(a, "Workflows$SingleInputCompletion"), "$ID") + idb, _ := get(find(b, "Workflows$SingleInputCompletion"), "$ID") + if !sameID(ida, idb) { + t.Errorf("the completion's $ID differs between two writes of the same task: %v, %v", ida, idb) + } +} + +func nonZeroID(v any) bool { + b, ok := v.(bson.Binary) + return ok && len(b.Data) == 16 && !bytes.Equal(b.Data, make([]byte, 16)) +} + +// A multi-user task below 10.12 is a Workflows$UserTask whose +// UserTaskCompletion is a Workflows$MultiInputCompletion. From 10.12 the +// MultiUserTaskActivity holds the same things itself: CompletionCriteria (its +// outcome pointers unchanged), TargetUserInput and, from 10.2, AwaitAllUsers. +func TestNormalize_MultiUserTaskBelowMendix10_12(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + completion string // keys of the MultiInputCompletion; "" = a MultiUserTaskActivity + }{ + {pv(9, 24, 30), "$ID,$Type,CompletionCriteria,TargetUserInput"}, + {pv(10, 1, 0), "$ID,$Type,CompletionCriteria,TargetUserInput"}, + {pv(10, 2, 0), "$ID,$Type,AwaitAllUsers,CompletionCriteria,TargetUserInput"}, + {pv(10, 11, 0), "$ID,$Type,AwaitAllUsers,CompletionCriteria,TargetUserInput"}, + {pv(10, 12, 0), ""}, + {pv(10, 24, 24), ""}, // control + } { + doc := userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")) + criteria, _ := get(doc, "CompletionCriteria") + input, _ := get(doc, "TargetUserInput") + d, err := normalizeShape(t, tt.v, workflowWithUserTasks(doc)) + if err != nil { + t.Errorf("%s: refused: %v", vlabel(tt.v), err) + continue + } + task := findNamed(d, "Multi") + typ, _ := get(task, "$Type") + if tt.completion == "" { + if typ != "Workflows$MultiUserTaskActivity" { + t.Errorf("%s: $Type %v, want Workflows$MultiUserTaskActivity", vlabel(tt.v), typ) + } + continue + } + if typ != "Workflows$UserTask" { + t.Errorf("%s: $Type %v, want Workflows$UserTask", vlabel(tt.v), typ) + } + if got := sortedKeys(task); got != studioPro924UserTask { + t.Errorf("%s: user task keys\n %s\nwant\n %s", vlabel(tt.v), got, studioPro924UserTask) + } + completion, _ := get(task, "UserTaskCompletion") + cd, _ := completion.(bson.D) + if ct, _ := get(cd, "$Type"); ct != "Workflows$MultiInputCompletion" { + t.Errorf("%s: UserTaskCompletion is %v, want Workflows$MultiInputCompletion", vlabel(tt.v), ct) + continue + } + if got := strings.Join(keysOf(cd), ","); got != tt.completion { + t.Errorf("%s: MultiInputCompletion keys %s, want %s", vlabel(tt.v), got, tt.completion) + } + gotCriteria, _ := get(cd, "CompletionCriteria") + gotInput, _ := get(cd, "TargetUserInput") + if !sameValue(gotCriteria, criteria) || !sameValue(gotInput, input) { + t.Errorf("%s: criteria %v and participants %v, want the task's own %v and %v", vlabel(tt.v), gotCriteria, gotInput, criteria, input) + } + } +} + +// What a Workflows$UserTask cannot hold is refused, naming the task: +// - a multi-user task below 9.22; +// - a rule Mendix had not introduced yet: majority and threshold are 10.0, +// a microflow rule is 10.3; +// - `await all users` below 10.2; +// - a boundary event on a user task below 10.14. +// +// An empty BoundaryEvents list is the codec's default, and it is dropped. +func TestNormalize_RefusesWhatAnOlderUserTaskCannotHold(t *testing.T) { + withCriteria := func(typ string, extra ...bson.E) bson.D { + doc := userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")) + cc := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typ}} + cc = append(cc, extra...) + doc[indexOf(doc, "CompletionCriteria")].Value = cc + return doc + } + awaiting := userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")) + awaiting[indexOf(awaiting, "AwaitAllUsers")].Value = true + withBoundary := userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")) + withBoundary[indexOf(withBoundary, "BoundaryEvents")].Value = bson.A{int32(2), + bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$InterruptingTimerBoundaryEvent"}, {Key: "Caption", Value: "Late"}}} + for _, tt := range []struct { + name string + v *version.ProjectVersion + doc bson.D + want string + }{ + {"multi-user task", pv(9, 21, 0), userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")), "9.22"}, + {"await all users", pv(9, 24, 30), awaiting, "10.2"}, + {"majority", pv(9, 24, 30), withCriteria("Workflows$MajorityCompletionCriteria", bson.E{Key: "CompletionType", Value: "Absolute"}), "10.0"}, + {"threshold", pv(9, 24, 30), withCriteria("Workflows$ThresholdCompletionCriteria", bson.E{Key: "Threshold", Value: int32(2)}), "10.0"}, + {"microflow rule", pv(10, 2, 0), withCriteria("Workflows$MicroflowCompletionCriteria", bson.E{Key: "Microflow", Value: "M.Decide"}), "10.3"}, + {"boundary event 9.24", pv(9, 24, 30), withBoundary, "10.14"}, + {"boundary event 10.13", pv(10, 13, 0), withBoundary, "10.14"}, + } { + name, _ := get(tt.doc, "Name") + _, err := normalizeShape(t, tt.v, workflowWithUserTasks(tt.doc)) + if err == nil || !strings.Contains(err.Error(), name.(string)) || !strings.Contains(err.Error(), tt.want) { + t.Errorf("%s on %s was not refused naming the task and %s: %v", tt.name, vlabel(tt.v), tt.want, err) + } + } + // Control: each is written where Mendix has it. + for _, tt := range []struct { + v *version.ProjectVersion + doc bson.D + }{ + {pv(9, 22, 0), userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']"))}, + {pv(10, 2, 0), awaiting}, + {pv(10, 0, 0), withCriteria("Workflows$MajorityCompletionCriteria", bson.E{Key: "CompletionType", Value: "Absolute"})}, + {pv(10, 3, 0), withCriteria("Workflows$MicroflowCompletionCriteria", bson.E{Key: "Microflow", Value: "M.Decide"})}, + {pv(10, 14, 0), withBoundary}, + } { + if _, err := normalizeShape(t, tt.v, workflowWithUserTasks(tt.doc)); err != nil { + t.Errorf("control %s: refused: %v", vlabel(tt.v), err) + } + } + // The empty list the codec writes on every task is dropped below 10.14. + d, err := normalizeShape(t, pv(10, 13, 0), workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")))) + if err != nil { + t.Fatal(err) + } + if _, ok := get(findNamed(d, "Single"), "BoundaryEvents"); ok { + t.Errorf("10.13: an empty BoundaryEvents is kept on a user task, which declares none before 10.14") + } +} + +func indexOf(d bson.D, key string) int { + for i, e := range d { + if e.Key == key { + return i + } + } + panic("no " + key) +} + +// sameValue compares two stored values by their BSON bytes. +func sameValue(a, b any) bool { + x, err1 := bson.Marshal(bson.D{{Key: "v", Value: a}}) + y, err2 := bson.Marshal(bson.D{{Key: "v", Value: b}}) + return err1 == nil && err2 == nil && bytes.Equal(x, y) +} From 3c69d95118c71fd3238b43e4cb0a6f918228cdff Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:38:05 +0000 Subject: [PATCH 37/61] fix(wfmutator): set a user task's targeting whichever key stores it Symptom (ako/mxcli#1771): on Mendix 9.x to 11.1, `alter workflow M.W { set (Targeting: xpath [...]) on Task; }`, and the deprecated `set activity 'Task' targeting ...` form, reported "Altered workflow" and changed nothing. describe still showed the old targeting, and mx check stayed clean. Cause: SetActivityProperty set UserTargeting with bsonnav.DSet, which only replaces a key that already exists. Below 11.2 a stored user task holds its targeting as UserSource, so no key matched and nothing changed. Fix: setUserTargeting replaces UserTargeting or UserSource in place, or adds the targeting when the task has neither. It writes the current shape; canon.LowerToProjectShape stores it as the project's version does, as it does for every other writer. Measured on fresh apps, the bug-test exec'd then described and checked: - 9.24.30.51827 and 10.24.24.119653: with the committed mutator (a scratch build), "Altered workflow" x2 and `[Name = 'admin']` kept. With the fix, `[Name = 'reviewer']` and ACT_GetReviewers, 0 errors, and the version's key set (Workflows$UserTask with UserSource on 9.24). - 11.13.0: the new targeting both ways, 0 errors. Control: TestWorkflowMutator_SetTargetingReplacesAStoredUserSource and ...AddsAMissingTargeting failed before the fix. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...9-alter-set-targeting-noop-below-11-2.json | 1 + CHANGELOG.md | 1 + .../1771-alter-user-task-targeting.mdl | 61 ++++++++++++++ mdl/backend/wfmutator/mutator.go | 26 ++++-- .../wfmutator/set_targeting_1771_test.go | 79 +++++++++++++++++++ .../writer_version_shape_1771_type_test.go | 28 +++++++ 6 files changed, 190 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json create mode 100644 mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl create mode 100644 mdl/backend/wfmutator/set_targeting_1771_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json new file mode 100644 index 000000000..bc94ff214 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json @@ -0,0 +1 @@ +{"area": "mdl/backend", "date": "2026-10-29", "symptom": "On Mendix 9.x to 11.1, `alter workflow M.W { set (Targeting: xpath […]) on Task; }`, and the deprecated `set activity 'Task' targeting …` form, printed 'Altered workflow' and changed nothing. describe showed the old targeting, and mx check stayed clean, because the old targeting was still valid", "cause": "Mutator.SetActivityProperty set the targeting with bsonnav.DSet(actDoc, \"UserTargeting\", …), and DSet only replaces a key that already exists. Below 11.2 a stored user task holds its targeting as UserSource, so no key matched, nothing changed, and the op returned nil", "file": "`mdl/backend/wfmutator/mutator.go` (setUserTargeting: replaces UserTargeting or UserSource in place, else appends; the write lowers it per version in canon.LowerToProjectShape)", "insight": "**A setter that only replaces an existing key is a silent no-op on a document in another version's shape.** bsonnav.DSet reports false, and every caller here ignored it. When a storage key is renamed at a version (ako/mxcli#1771: UserSource → UserTargeting at 11.2), grep the raw mutators for the new key: each one that DSets it is broken on the older versions, and the success message hides it. Have the mutator write the CURRENT shape in place of whichever key holds the fact, and let canon.LowerToProjectShape convert it. A second copy of the old shape in the mutator would be a second mapping to keep in step. **The control needs a pre-fix binary for the exact bug-test**: rebuilding with the committed file swapped in (go build -o scratch) showed 'Altered workflow' x2 with `[Name = 'admin']` kept on 9.24.30 and 10.24.24. Measured after the fix: 9.24.30 and 10.24.24 describe the new targeting, with 0 errors and the version's key set (Workflows$UserTask with UserSource on 9.24). 11.13.0 is unchanged. Tests: TestWorkflowMutator_SetTargetingReplacesAStoredUserSource and …AddsAMissingTargeting (both failed before the fix), TestNormalize_EditedMendix9UserTaskTargeting; mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl", "refs": ["#1771"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index f210553a8..5b5f0ce2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. - **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. diff --git a/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl new file mode 100644 index 000000000..dff335a7a --- /dev/null +++ b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl @@ -0,0 +1,61 @@ +mdl 1; +-- ako/mxcli#1771: on Mendix 9.x to 11.1, `alter workflow … { set (Targeting: …) +-- on … }` reported "Altered workflow" and changed nothing. describe +-- printed the old targeting afterwards, and mx check stayed clean, because +-- the old targeting was still valid. +-- +-- Below 11.2 a stored user task holds its targeting as UserSource. The ALTER +-- mutator set UserTargeting, the 11.2 key, and only where the task already had +-- one, so on those versions it found none and did nothing. It now replaces the +-- stored targeting, whichever key holds it. The write then stores it in the +-- version's shape (see 1771-user-task-targeting-below-11-2.mdl). +-- +-- Measured on fresh apps (`mxcli new … --skip-build`): exec, then describe, +-- then `mxcli docker check`. +-- 9.24.30.51827, 10.24.24.119653 before: the old targeting was kept. After: +-- `[Name = 'reviewer']` and ACT_GetReviewers. 0 errors, with +-- the version's key set (Workflows$UserTask on 9.24). +-- 11.13.0 the new targeting both before and after, 0 errors. + +create module Probe1771At; + +create persistent entity Probe1771At.Request ( Title: String(100) ); + +create microflow Probe1771At.ACT_GetUsers ($Workflow: System.Workflow, $Context: Probe1771At.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create microflow Probe1771At.ACT_GetReviewers ($Workflow: System.Workflow, $Context: Probe1771At.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User where Name = 'reviewer'; + return $Users; +end; + +create page Probe1771At.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771At.WF_Edit + parameter $WorkflowContext: Probe1771At.Request + display 'Edit' +begin + user task Review 'Review' + page Probe1771At.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Done' { }; + user task Handle 'Handle' + page Probe1771At.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Done' { }; +end workflow; + +alter workflow Probe1771At.WF_Edit { set (Targeting: xpath [Name = 'reviewer']) on Review; }; +alter workflow Probe1771At.WF_Edit { set (Targeting: microflow Probe1771At.ACT_GetReviewers) on Handle; }; diff --git a/mdl/backend/wfmutator/mutator.go b/mdl/backend/wfmutator/mutator.go index 00a04e75a..2dc409d3f 100644 --- a/mdl/backend/wfmutator/mutator.go +++ b/mdl/backend/wfmutator/mutator.go @@ -313,21 +313,19 @@ func (m *Mutator) SetActivityProperty(activityRef string, atPos int, prop string return nil case "targeting_microflow": - userTargeting := bson.D{ + m.setUserTargeting(actDoc, bson.D{ {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, {Key: "$Type", Value: "Workflows$MicroflowUserTargeting"}, {Key: "Microflow", Value: value}, - } - bsonnav.DSet(actDoc, "UserTargeting", userTargeting) + }) return nil case "targeting_xpath": - userTargeting := bson.D{ + m.setUserTargeting(actDoc, bson.D{ {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, {Key: "$Type", Value: "Workflows$XPathUserTargeting"}, {Key: "XPathConstraint", Value: value}, - } - bsonnav.DSet(actDoc, "UserTargeting", userTargeting) + }) return nil case "due_date": @@ -768,6 +766,22 @@ func (m *Mutator) Save() error { // matching on $ID. Needed when appending new keys to an activity document, // because the slice header returned by findActivityByCaption cannot propagate // appends back to the parent bson.A. +// setUserTargeting replaces a user task's targeting with targeting, in the +// current (11.2) shape and in the place of whichever key stores it: below 11.2 +// that is UserSource (ako/mxcli#1771), which the write lowers it back to +// (canon.LowerToProjectShape). Setting UserTargeting only where it already +// existed changed nothing on such a task, and the statement still reported +// success. +func (m *Mutator) setUserTargeting(actDoc bson.D, targeting bson.D) { + for i := range actDoc { + if actDoc[i].Key == "UserTargeting" || actDoc[i].Key == "UserSource" { + actDoc[i] = bson.E{Key: "UserTargeting", Value: targeting} + return + } + } + m.replaceActivity(append(actDoc, bson.E{Key: "UserTargeting", Value: targeting})) +} + func (m *Mutator) replaceActivity(updated bson.D) { actID := bsonnav.ExtractBinaryIDFromDoc(bsonnav.DGet(updated, "$ID")) if actID == "" { diff --git a/mdl/backend/wfmutator/set_targeting_1771_test.go b/mdl/backend/wfmutator/set_targeting_1771_test.go new file mode 100644 index 000000000..4b60857a9 --- /dev/null +++ b/mdl/backend/wfmutator/set_targeting_1771_test.go @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: Apache-2.0 + +package wfmutator + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "github.com/mendixlabs/mxcli/mdl/bsonutil" +) + +// ako/mxcli#1771: below Mendix 11.2 a stored user task holds its targeting as +// UserSource, not UserTargeting. `alter workflow … set activity '…' targeting +// …` set UserTargeting only where the key already existed, so on 9.24 and +// 10.24 it found none, changed nothing and still reported "Altered workflow" +// (measured: describe printed the old targeting afterwards). The edit replaces +// the stored targeting, whichever key holds it, in the current shape; the +// write lowers it to the project's (canon.LowerToProjectShape). +func TestWorkflowMutator_SetTargetingReplacesAStoredUserSource(t *testing.T) { + for _, tc := range []struct { + prop, value, wantType, valueKey string + }{ + {"targeting_xpath", "[Name = 'reviewer']", "Workflows$XPathUserTargeting", "XPathConstraint"}, + {"targeting_microflow", "M.ACT_Reviewers", "Workflows$MicroflowUserTargeting", "Microflow"}, + } { + for _, typ := range []string{"Workflows$UserTask", "Workflows$SingleUserTaskActivity"} { + act := makeWfActivity(typ, "Review", "task1") + act = append(act, + bson.E{Key: "UserSource", Value: bson.D{{Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, + {Key: "$Type", Value: "Workflows$XPathBasedUserSource"}, {Key: "XPathConstraint", Value: "[Name = 'admin']"}}}, + bson.E{Key: "UserTaskCompletion", Value: nil}) + sourceAt := keyIndex(act, "UserSource") + m := newMutator(makeWorkflowDoc(act)) + + if err := m.SetActivityProperty("Review", 0, tc.prop, tc.value); err != nil { + t.Fatalf("%s on a %s: %v", tc.prop, typ, err) + } + actDoc, _ := m.findActivityByCaption("Review", 0) + if bsonnav.DGet(actDoc, "UserSource") != nil { + t.Errorf("%s on a %s: the stored UserSource is still there beside the new targeting", tc.prop, typ) + } + targeting := bsonnav.DGetDoc(actDoc, "UserTargeting") + if got := bsonnav.DGetString(targeting, "$Type"); got != tc.wantType { + t.Errorf("%s on a %s: targeting %q, want %s", tc.prop, typ, got, tc.wantType) + continue + } + if got := bsonnav.DGetString(targeting, tc.valueKey); got != tc.value { + t.Errorf("%s on a %s: %s = %q, want %q", tc.prop, typ, tc.valueKey, got, tc.value) + } + // In the stored key's place, so the lowered document keeps its order. + if i := keyIndex(actDoc, "UserTargeting"); i != sourceAt { + t.Errorf("%s on a %s: targeting at %d, want the stored source's place %d", tc.prop, typ, i, sourceAt) + } + } + } +} + +// A task holding neither key gets the targeting added. +func TestWorkflowMutator_SetTargetingAddsAMissingTargeting(t *testing.T) { + m := newMutator(makeWorkflowDoc(makeWfActivity("Workflows$SingleUserTaskActivity", "Review", "task1"))) + if err := m.SetActivityProperty("Review", 0, "targeting_xpath", "[Name = 'reviewer']"); err != nil { + t.Fatal(err) + } + actDoc, _ := m.findActivityByCaption("Review", 0) + if got := bsonnav.DGetString(bsonnav.DGetDoc(actDoc, "UserTargeting"), "XPathConstraint"); got != "[Name = 'reviewer']" { + t.Errorf("targeting %q, want it added", got) + } +} + +func keyIndex(d bson.D, key string) int { + for i, e := range d { + if e.Key == key { + return i + } + } + return -1 +} diff --git a/modelsdk/mpr/writer_version_shape_1771_type_test.go b/modelsdk/mpr/writer_version_shape_1771_type_test.go index 29c747732..21a21ae5b 100644 --- a/modelsdk/mpr/writer_version_shape_1771_type_test.go +++ b/modelsdk/mpr/writer_version_shape_1771_type_test.go @@ -214,6 +214,34 @@ func TestNormalize_RefusesWhatAnOlderUserTaskCannotHold(t *testing.T) { } } +// `alter workflow … set activity '…' targeting …` on a stored Mendix 9 task +// puts a UserTargeting in the place of its UserSource (the mutator writes the +// current shape). The write lowers it back: the task keeps Studio Pro 9.24's +// key set, with the new targeting value. +func TestNormalize_EditedMendix9UserTaskTargeting(t *testing.T) { + task := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$UserTask"}, + {Key: "Annotation", Value: nil}, {Key: "AutoAssignSingleTargetUser", Value: false}, {Key: "Caption", Value: "Edited"}, + {Key: "DueDate", Value: ""}, {Key: "Name", Value: "Edited"}, + {Key: "OnCreatedEvent", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$NoEvent"}}}, + {Key: "Outcomes", Value: bson.A{int32(3), userTaskOutcome("Done")}}, + {Key: "TaskDescription", Value: stringTemplate("")}, {Key: "TaskName", Value: stringTemplate("Edited")}, + {Key: "TaskPage", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$PageReference"}, {Key: "Page", Value: "M.TaskPage"}}}, + {Key: "UserTargeting", Value: xpathUserTargeting("[Name = 'reviewer']")}, + {Key: "UserTaskCompletion", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$SingleInputCompletion"}}}} + d, err := normalizeShape(t, pv(9, 24, 30), workflowWithUserTasks(task)) + if err != nil { + t.Fatal(err) + } + got := findNamed(d, "Edited") + if keys := strings.Join(keysOf(got), ","); keys != studioPro924UserTask { + t.Errorf("edited task keys %s, want %s", keys, studioPro924UserTask) + } + src, _ := get(got, "UserSource") + if x, _ := get(src.(bson.D), "XPathConstraint"); x != "[Name = 'reviewer']" { + t.Errorf("edited targeting %v, want the new XPath", src) + } +} + func indexOf(d bson.D, key string) int { for i, e := range d { if e.Key == key { From 684a447f96a14b6dd010c65a0d522af852617f08 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:45:53 +0000 Subject: [PATCH 38/61] fix(executor): gate the user-task features older Mendix cannot store Symptom (ako/mxcli#1771): `mxcli check -p` passed scripts with user-task features the project's version cannot store, and exec wrote them. Group targeting on 10.24 failed mx check with CE1859, and mx convert emptied it. Once the user task is lowered to its pre-10.12 shape, a multi-user task's later rules were refused only at write time, part-way through a script. Cause: no version gate looked inside a workflow user task. The generated version data floors properties, never types, and these are type-level introductions (mendixmodelsdk 4.117.0 StructureVersionInfo): the multi-user task's UserTaskCompletion 9.22, Majority and ThresholdCompletionCriteria 10.0, MultiInputCompletion.awaitAllUsers 10.2, MicroflowCompletionCriteria 10.3, and XPath/MicroflowGroupTargeting 11.2. Fix: userTaskGates walks the activities a create workflow states, and the ones alter workflow inserts. It adds an MDL-VERSION01 gate for each clause the version lacks. A consensus or veto multi-user task with user targeting passes from 9.22. The floors are new sdk/versions entries: workflows.multi_user_task, multi_user_task_majority_threshold, multi_user_task_await_all, multi_user_task_microflow_rule and group_targeting. Measured with the bug-test (check -p, then exec, then mx check): - 9.24.30.51827: WF_Majority, WF_Await, WF_Microflow and WF_Groups reported; exec "Refusing to execute: 4 error(s) above. Nothing was written." - 10.24.24.119653: WF_Groups only. - 11.13.0: none; exec writes all five, mx check 0 errors. Control: TestCheckVersionGates_UserTaskFeatures failed for every case before the gates. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...9-user-task-features-gated-by-version.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 9 +- .../1771-user-task-features-by-version.mdl | 98 ++++++++++++++++ mdl/executor/version_gates.go | 67 ++++++++++- .../version_gates_usertask_1771_test.go | 107 ++++++++++++++++++ sdk/versions/mendix-10.yaml | 16 +++ sdk/versions/mendix-11.yaml | 20 ++++ sdk/versions/mendix-9.yaml | 4 + 9 files changed, 319 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json create mode 100644 mdl-examples/bug-tests/1771-user-task-features-by-version.mdl create mode 100644 mdl/executor/version_gates_usertask_1771_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json new file mode 100644 index 000000000..f65e83190 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-29", "symptom": "`mxcli check -p` passed scripts with user-task features the project's version cannot store, and exec wrote them: group targeting below 11.2 (CE1859 on 10.24, the targeting emptied by mx convert), and multi-user task rules and `await all users` below the version that introduced them. Once the user task was lowered to its pre-10.12 shape, these were refused only at write time, part-way through a script", "cause": "statementVersionGates had no gate for anything inside a workflow user task. The generated version data floors properties, never types, and the rule types (MajorityCompletionCriteria, ThresholdCompletionCriteria 10.0; MicroflowCompletionCriteria 10.3), the group targetings (11.2) and the multi-user task itself (UserTaskCompletion 9.22) are type-level introductions", "file": "`mdl/executor/version_gates.go` (userTaskGates, called from the CreateWorkflowStmt and AlterWorkflowStmt cases); `sdk/versions/mendix-9|10|11.yaml` (workflows.multi_user_task 9.22.0, multi_user_task_majority_threshold 10.0.0, multi_user_task_await_all 10.2.0, multi_user_task_microflow_rule 10.3.0, group_targeting 11.2.0)", "insight": "**A write-time refusal is the last line, not the check.** canon.LowerToProjectShape refuses what the older shape cannot hold, but by then the statements before it are written. Each such refusal wants a matching MDL-VERSION01 gate, so `check -p` reports it and exec refuses the script before writing anything ('Refusing to execute: 4 error(s) above. Nothing was written.' on 9.24.30). **Gate the clause, not the statement**: a workflow with a consensus or veto multi-user task is fine from 9.22, so the gates walk the activities (create, and every activity alter workflow inserts) and fire only for a rule, a targeting kind or `await all users` the version lacks. **Read the type floors from mendixmodelsdk's StructureVersionInfo** (gen/workflows.js; text-extract it, never execute it). The generated version.go has no row for a type. Measured with the bug-test: check -p on 9.24.30 reports WF_Majority (10.0), WF_Await (10.2), WF_Microflow (10.3) and WF_Groups (11.2). 10.24.24 reports WF_Groups only. 11.13.0 reports none, exec writes all five, and mx check gives 0 errors. Control: TestCheckVersionGates_UserTaskFeatures failed for every case before the gates. Tests: TestCheckVersionGates_UserTaskFeatures, …AtTheirFloors; mdl-examples/bug-tests/1771-user-task-features-by-version.mdl", "refs": ["#1771"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b5f0ce2d..265ecc5a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **User-task features an older Mendix cannot store are refused up front** (ako/mxcli#1771) — `mxcli check -p` and `exec` report MDL-VERSION01 before anything is written, naming the clause. The floors are group targeting (`targeting groups …`) 11.2, a multi-user task 9.22, `decide by majority` or `threshold` 10.0, `await all users` 10.2, and `decide by microflow` 10.3. Before, group targeting was written on 10.24 and failed `mx check` with CE1859. The others were refused only at write time, after the statements before them were written. Measured on 9.24.30 and 10.24.24 (each refused, nothing written) and on 11.13.0 (all written, 0 errors). - **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. - **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index f5c8c1325..83fc73373 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -119,6 +119,11 @@ body on Mendix 10 too, and Studio Pro could not open the project. |---------|-----------|-----|-------| | Basic workflows | `CREATE WORKFLOW` | Yes | Yes | | User tasks | `USER TASK ... OUTCOMES (...)` | Yes | Yes | +| Multi-user tasks | `MULTI USER TASK ...` | 9.22+ | Yes | +| Multi-user task rules majority and threshold | `DECIDE BY MAJORITY ...`, `DECIDE BY THRESHOLD ...` | No | Yes | +| Waiting for all users | `AWAIT ALL USERS` | No | 10.2+ | +| Deciding by a microflow | `DECIDE BY MICROFLOW ...` | No | 10.3+ | +| Group targeting | `TARGETING GROUPS XPATH ...`, `TARGETING GROUPS MICROFLOW ...` | No | 11.2+ | | Parallel splits | `PARALLEL SPLIT` | Yes | Yes | Below Mendix 11.3 a workflow's main flow has no stored start or end activity: @@ -199,8 +204,8 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a data view fed by another widget below 10.0; - a private constant value in a configuration below 10.9; - a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; -- a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); -- a multi-user task below 9.22, `decide by majority` or `decide by threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14 ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); +- a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only; +- a multi-user task below 9.22, `decide by majority` or `decide by threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. The user-task features are also refused up front, by `check -p` and by `exec` before anything is written, with MDL-VERSION01 ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl new file mode 100644 index 000000000..f84d8326f --- /dev/null +++ b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl @@ -0,0 +1,98 @@ +mdl 1; +-- ako/mxcli#1771: what an older user task cannot store is refused up front +-- with MDL-VERSION01, by `mxcli check -p` and by exec before anything is +-- written. Before, these were written as they are on 11.2+: +-- - group targeting below 11.2. mx check 10.24.24 then reported CE1859 +-- "XPath constraint is required", and `mx convert` emptied the +-- targeting. Before 11.2 a user task targets users only. +-- - a multi-user task below 10.12, with the 10.12 types. mx check 9.24.30 +-- could not load the project (TypeCacheUnknownTypeException). +-- Below 10.12 a multi-user task is now written as a Workflows$UserTask (see +-- 1771-user-task-below-10-12.mdl), which has no room for what came later: +-- multi user task 9.22 +-- decide by majority / threshold 10.0 +-- await all users 10.2 +-- decide by microflow 10.3 +-- targeting groups xpath / microflow 11.2 +-- (mendixmodelsdk 4.117.0 StructureVersionInfo.) +-- +-- EXPECTED: `mxcli check` passes this file. With -p on a project below a +-- floor, each workflow that states the feature is reported as MDL-VERSION01, +-- naming it. Measured on fresh apps: +-- 9.24.30.51827 WF_Majority, WF_Await, WF_Microflow, WF_Groups +-- 10.24.24.119653 WF_Groups +-- 11.13.0 none; exec writes all five, and mx check reports 0 errors. + +create module Probe1771Gt; + +create persistent entity Probe1771Gt.Request ( Title: String(100) ); + +create microflow Probe1771Gt.ACT_Groups ($Workflow: System.Workflow, $Context: Probe1771Gt.Request) +returns list of System.WorkflowGroup as $Groups +begin + retrieve $Groups from System.WorkflowGroup; + return $Groups; +end; + +create microflow Probe1771Gt.ACT_Decide ($WorkflowUserTask: System.WorkflowUserTask) +returns String as $Outcome +begin + return 'Yes'; +end; + +create page Probe1771Gt.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771Gt.WF_Veto + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + decide by veto 'No' + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Majority + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + decide by majority more than half fallback 'No' + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Await + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + await all users + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Microflow + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + decide by microflow Probe1771Gt.ACT_Decide + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Groups + parameter $WorkflowContext: Probe1771Gt.Request +begin + user task Review 'Review' + page Probe1771Gt.TaskPage + targeting groups microflow Probe1771Gt.ACT_Groups + outcomes 'Done' { }; +end workflow; diff --git a/mdl/executor/version_gates.go b/mdl/executor/version_gates.go index 33514acad..e2afaec8e 100644 --- a/mdl/executor/version_gates.go +++ b/mdl/executor/version_gates.go @@ -201,7 +201,7 @@ func statementVersionGates(stmt ast.Statement) []versionGate { if workflowUsesNotificationEvents(activities) { gates = append(gates, notificationEventsGate) } - return gates + return append(gates, userTaskGates(activities)...) case *ast.AlterWorkflowStmt: // Workflows require Mendix 9.12+. gates := []versionGate{{"workflows", "basic", "alter workflow", @@ -213,11 +213,74 @@ func statementVersionGates(stmt ast.Statement) []versionGate { if workflowUsesNotificationEvents(added) { gates = append(gates, notificationEventsGate) } - return gates + return append(gates, userTaskGates(added)...) } return nil } +// userTaskGates are the gates of what the user tasks among activities state +// that an older user task cannot store, each once (ako/mxcli#1771). Below 11.2 +// a task targets users only, and a multi-user task, its later rules and +// `await all users` arrived in 9.22, 10.0, 10.3 and 10.2 (mendixmodelsdk +// 4.117.0). The write refuses them too (canon.LowerToProjectShape); this names +// the statement before anything is written. +func userTaskGates(activities []ast.WorkflowActivityNode) []versionGate { + var groups, multi, majority, threshold, microflow, await bool + walkWorkflowActivities(activities, func(a ast.WorkflowActivityNode) { + ut, ok := a.(*ast.WorkflowUserTaskNode) + if !ok { + return + } + switch ut.Targeting.Kind { + case "group_xpath", "group_microflow": + groups = true + } + if !ut.IsMultiUser { + return + } + multi = true + await = await || ut.AwaitAllUsers + if ut.Completion != nil { + switch ut.Completion.Rule { + case "majority": + majority = true + case "threshold": + threshold = true + case "microflow": + microflow = true + } + } + }) + var gates []versionGate + if groups { + gates = append(gates, versionGate{"workflows", "group_targeting", "targeting groups on a user task", + "Before Mendix 11.2 a user task targets users only: `targeting users xpath […]`, or a microflow " + + "returning users. Workflow groups arrived in 11.2."}) + } + if multi { + gates = append(gates, versionGate{"workflows", "multi_user_task", "multi user task", + "Mendix added multi-user tasks in 9.22: use a user task, or upgrade the project"}) + } + for _, r := range []struct { + used bool + rule string + }{{majority, "majority"}, {threshold, "threshold"}} { + if r.used { + gates = append(gates, versionGate{"workflows", "multi_user_task_majority_threshold", "decide by " + r.rule, + "Mendix added the majority and threshold rules in 10.0: decide by consensus or veto, or upgrade the project"}) + } + } + if microflow { + gates = append(gates, versionGate{"workflows", "multi_user_task_microflow_rule", "decide by microflow", + "Mendix added deciding a multi-user task by a microflow in 10.3: decide by consensus or veto, or upgrade the project"}) + } + if await { + gates = append(gates, versionGate{"workflows", "multi_user_task_await_all", "await all users", + "Mendix added `await all users` in 10.2: leave it out, so the task completes once its rule is met, or upgrade the project"}) + } + return gates +} + // The floors of a flow call's `returns ( … )` mappings (ako/mxcli#977). The // builder applies the same two, for the widgets this walk cannot see (a // fragment's, expanded at build time). diff --git a/mdl/executor/version_gates_usertask_1771_test.go b/mdl/executor/version_gates_usertask_1771_test.go new file mode 100644 index 000000000..1d11cf45d --- /dev/null +++ b/mdl/executor/version_gates_usertask_1771_test.go @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#1771: what an older user task cannot store is refused by +// `check -p` and by exec's pre-flight with MDL-VERSION01, before anything is +// written. The write-time refusal in canon.LowerToProjectShape stays as the +// last line; these name the statement and say what to do instead. +// - Group targeting is 11.2. Before it, a user task targets users only. +// - A multi-user task is 9.22. +// - Its rules majority and threshold are 10.0, microflow is 10.3, and +// `await all users` is 10.2 (mendixmodelsdk 4.117.0). +func TestCheckVersionGates_UserTaskFeatures(t *testing.T) { + task := func(f func(*ast.WorkflowUserTaskNode)) *ast.WorkflowUserTaskNode { + n := &ast.WorkflowUserTaskNode{Name: "Review", Caption: "Review"} + f(n) + return n + } + groups := task(func(n *ast.WorkflowUserTaskNode) { + n.Targeting = ast.WorkflowTargetingNode{Kind: "group_xpath", XPath: "[Name = 'Reviewers']"} + }) + groupsMicroflow := task(func(n *ast.WorkflowUserTaskNode) { + n.Targeting = ast.WorkflowTargetingNode{Kind: "group_microflow", Microflow: ast.QualifiedName{Module: "M", Name: "Groups"}} + }) + multi := task(func(n *ast.WorkflowUserTaskNode) { n.IsMultiUser = true }) + rule := func(r string) *ast.WorkflowUserTaskNode { + return task(func(n *ast.WorkflowUserTaskNode) { + n.IsMultiUser = true + n.Completion = &ast.WorkflowCompletionRuleNode{Rule: r} + }) + } + await := task(func(n *ast.WorkflowUserTaskNode) { n.IsMultiUser, n.AwaitAllUsers = true, true }) + + for _, c := range []struct { + name string + act ast.WorkflowActivityNode + below [3]int + want string + }{ + {"groups xpath", groups, [3]int{11, 1, 0}, "targeting groups"}, + {"groups microflow", groupsMicroflow, [3]int{10, 24, 24}, "targeting groups"}, + {"multi user task", multi, [3]int{9, 21, 0}, "multi user task"}, + {"majority", rule("majority"), [3]int{9, 24, 30}, "decide by majority"}, + {"threshold", rule("threshold"), [3]int{9, 24, 30}, "decide by threshold"}, + {"microflow rule", rule("microflow"), [3]int{10, 2, 0}, "decide by microflow"}, + {"await all users", await, [3]int{10, 1, 0}, "await all users"}, + } { + for _, stmt := range []ast.Statement{ + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{c.act}}, + &ast.AlterWorkflowStmt{Operations: []ast.AlterWorkflowOp{ + &ast.InsertAfterOp{NewActivities: []ast.WorkflowActivityNode{c.act}}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, stmt}} + vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog) + if len(vs) != 1 || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, c.want) || vs[0].Suggestion == "" { + t.Errorf("%s (%T) below its floor: want one MDL-VERSION01 naming %q with a hint, got %+v", c.name, stmt, c.want, vs) + } + // Control: a version with every feature passes. + if vs := versionGateViolations(versionCtx(t, 11, 99, 0), prog); len(vs) != 0 { + t.Errorf("%s (%T): reported at 11.99: %+v", c.name, stmt, vs) + } + } + } +} + +// The floors are each feature's own: a user task targeting users, and a +// multi-user task deciding by consensus or veto, are written from 9.22. +func TestCheckVersionGates_UserTaskFeaturesAtTheirFloors(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.CreateModuleStmt{Name: "M"}, + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{ + &ast.WorkflowUserTaskNode{Name: "A", Targeting: ast.WorkflowTargetingNode{Kind: "xpath", XPath: "[Name = 'a']"}}, + &ast.WorkflowUserTaskNode{Name: "B", Targeting: ast.WorkflowTargetingNode{Kind: "microflow", Microflow: ast.QualifiedName{Module: "M", Name: "U"}}}, + &ast.WorkflowUserTaskNode{Name: "C", IsMultiUser: true, Completion: &ast.WorkflowCompletionRuleNode{Rule: "veto", Veto: "No"}}, + &ast.WorkflowUserTaskNode{Name: "D", IsMultiUser: true, Completion: &ast.WorkflowCompletionRuleNode{Rule: "consensus"}}, + }}, + }} + if vs := versionGateViolations(versionCtx(t, 9, 22, 0), prog); len(vs) != 0 { + t.Errorf("user targeting and consensus/veto multi-user tasks reported on 9.22: %+v", vs) + } + for _, v := range [][3]int{{10, 0, 0}, {10, 2, 0}, {10, 3, 0}, {11, 2, 0}} { + for _, act := range []*ast.WorkflowUserTaskNode{ + {Name: "M", IsMultiUser: true, Completion: &ast.WorkflowCompletionRuleNode{Rule: "majority"}}, + {Name: "G", Targeting: ast.WorkflowTargetingNode{Kind: "group_xpath", XPath: "[Name = 'g']"}}, + } { + floor := [3]int{10, 0, 0} + if act.Name == "G" { + floor = [3]int{11, 2, 0} + } + if v[0] < floor[0] || (v[0] == floor[0] && v[1] < floor[1]) { + continue + } + p := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{act}}}} + if vs := versionGateViolations(versionCtx(t, v[0], v[1], v[2]), p); len(vs) != 0 { + t.Errorf("%s reported on %v, at or past its floor: %+v", act.Name, v, vs) + } + } + } +} diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index 207443cab..96a25823d 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -211,6 +211,22 @@ features: min_version: "9.0.0" user_task: min_version: "9.0.0" + multi_user_task: + min_version: "9.22.0" + mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" + notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_majority_threshold: + min_version: "10.0.0" + mdl: "MULTI USER TASK … DECIDE BY MAJORITY MORE THAN HALF | DECIDE BY THRESHOLD 60 PERCENT" + notes: "MajorityCompletionCriteria and ThresholdCompletionCriteria introduced in 10.0.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_await_all: + min_version: "10.2.0" + mdl: "MULTI USER TASK … AWAIT ALL USERS" + notes: "MultiInputCompletion.awaitAllUsers introduced in 10.2.0, on MultiUserTaskActivity from 10.12 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_microflow_rule: + min_version: "10.3.0" + mdl: "MULTI USER TASK … DECIDE BY MICROFLOW M.Decide" + notes: "MicroflowCompletionCriteria introduced in 10.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." event_handlers: min_version: "10.7.0" mdl: "CREATE WORKFLOW … ON WORKFLOW EVENTS (…) MICROFLOW Module.Handler AS '…'" diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index 02a08812e..ef0935f85 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -244,6 +244,26 @@ features: min_version: "9.0.0" user_task: min_version: "9.0.0" + multi_user_task: + min_version: "9.22.0" + mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" + notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_majority_threshold: + min_version: "10.0.0" + mdl: "MULTI USER TASK … DECIDE BY MAJORITY MORE THAN HALF | DECIDE BY THRESHOLD 60 PERCENT" + notes: "MajorityCompletionCriteria and ThresholdCompletionCriteria introduced in 10.0.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_await_all: + min_version: "10.2.0" + mdl: "MULTI USER TASK … AWAIT ALL USERS" + notes: "MultiInputCompletion.awaitAllUsers introduced in 10.2.0, on MultiUserTaskActivity from 10.12 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_microflow_rule: + min_version: "10.3.0" + mdl: "MULTI USER TASK … DECIDE BY MICROFLOW M.Decide" + notes: "MicroflowCompletionCriteria introduced in 10.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + group_targeting: + min_version: "11.2.0" + mdl: "USER TASK … TARGETING GROUPS XPATH [Name = 'Reviewers'] | TARGETING GROUPS MICROFLOW M.Groups" + notes: "XPathGroupTargeting and MicroflowGroupTargeting introduced in 11.2.0 with UserTaskActivity.userTargeting (mendixmodelsdk 4.117.0). Before 11.2 a user task's UserSource targets users only (ako/mxcli#1771)." ai_agent_task: min_version: "11.9.0" mdl: "CALL AGENT MICROFLOW Module.InvokeAgent WITH (Param = '$WorkflowContext')" diff --git a/sdk/versions/mendix-9.yaml b/sdk/versions/mendix-9.yaml index 7995a0554..433c27531 100644 --- a/sdk/versions/mendix-9.yaml +++ b/sdk/versions/mendix-9.yaml @@ -61,6 +61,10 @@ features: min_version: "9.0.0" user_task: min_version: "9.0.0" + multi_user_task: + min_version: "9.22.0" + mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" + notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." navigation: profiles: From b47de7efcbd48e9c5d2fd4c34d93c194949096b7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:53:34 +0000 Subject: [PATCH 39/61] test(doctype): write user tasks in the Mendix 10 workflow script 24-workflow-examples.mdl, the only doctype script with user tasks, is gated 11.0+ as a whole. CI's 10.24 leg therefore never wrote a user task. That hid ako/mxcli#1771: every user task mxcli wrote on 10.12 to 11.1 failed mx check with CE1859. 24b-workflow-mendix10-examples.mdl (10.0+) now writes user tasks by XPath, by a microflow and with no targeting. It also writes multi-user tasks with the consensus, veto, majority and threshold rules and participants, which 10.0 has. Then it inserts a task with alter workflow and changes a task's targeting. Measured with TestMxCheck_DoctypeScripts/24b: - 10.24.24.119653: PASS, mx check 0 errors. Control: with the targeting lowering stubbed, it FAILS with CE1859 on all 8 user tasks. - 11.13.0: PASS (24, 24b and 25 all pass, on both versions). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../24b-workflow-mendix10-examples.mdl | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl b/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl index 8d504c304..665a85a5a 100644 --- a/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl +++ b/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl @@ -19,6 +19,13 @@ mdl 1; -- `annotation` is the workflow's own note; from 11.3 they are stored and the -- note is the start activity's. A branch's `end workflow` is stored on every -- version. +-- +-- User tasks (end of file) hid ako/mxcli#1771 the same way. Below 11.2 a task's +-- targeting is a UserSource, and every user task mxcli wrote on 10.24 failed mx +-- check with CE1859 "XPath constraint is required". Only the features 10.0 +-- has are used: user targeting, and the consensus, veto, majority and +-- threshold rules. Group targeting is 11.2, `await all users` 10.2 and +-- `decide by microflow` 10.3. -- ============================================================================ create module WfTen; @@ -75,3 +82,68 @@ begin false -> { end workflow; } ; end workflow; + +-- User tasks: users by XPath, users by a microflow, no targeting, and +-- multi-user tasks with each rule 10.0 has. +create microflow WfTen.ACT_Approvers ($Workflow: System.Workflow, $Context: WfTen.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create page WfTen.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txtTask (content: 'Task') +}; + +create workflow WfTen.WF_Review + parameter $WorkflowContext: WfTen.Request + display 'Review' +begin + user task Review 'Review' + page WfTen.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Approve' { } 'Reject' { end workflow; }; + user task Handle 'Handle' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + outcomes 'Done' { }; + user task Anyone 'Anyone' + page WfTen.TaskPage + outcomes 'Done' { }; + multi user task Agree 'Agree' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + outcomes 'Yes' { } 'No' { }; + multi user task Vote 'Vote' + page WfTen.TaskPage + targeting users xpath [Name = 'admin'] + participants 2 + decide by veto 'No' + outcomes 'Yes' { } 'No' { }; + multi user task Poll 'Poll' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + participants 50 percent + decide by majority more than half fallback 'No' + outcomes 'Yes' { } 'No' { }; + multi user task Quorum 'Quorum' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + decide by threshold 60 percent fallback 'No' + outcomes 'Yes' { } 'No' { }; +end workflow; + +-- ALTER inserts a user task through the mutator, and changes a task's +-- targeting in whichever key the version stores it. +alter workflow WfTen.WF_Review { + insert after Handle { user task Confirm 'Confirm' + page WfTen.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'OK' { }; } +}; +alter workflow WfTen.WF_Review { set (Targeting: xpath [Name = 'reviewer']) on Review; }; From ee1a059196bd19ef868f01b3db3ac0ebaa573844 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 08:29:47 +0000 Subject: [PATCH 40/61] test(bug-tests): measure the 1771 user-task shapes on Mendix 10.11 The 10.0-10.11 claims rested on mendixmodelsdk's version data and on the 9.24 measurements. They are now measured on 10.11.0.37074, the last version before the 10.12 type boundary (mxbuild downloaded for this, then deleted): - With the user-task lowering stubbed, a user task makes the project unloadable (TypeCacheUnknownTypeException ... Workflows$SingleUserTaskActivity). - With the fix, the 1771 bug-tests give 0 errors. The multi-user tasks' MultiInputCompletion holds AwaitAllUsers (10.2+). mx convert 10.11 changes nothing, and describe -> exec reports "Unchanged workflow". Majority, await all users and a microflow rule check clean, and alter workflow's targeting edit lands. 1771-user-task-features-by-version.mdl's group-targeted task used a microflow returning System.WorkflowGroup, an entity Mendix introduced in 11.2. Written on 10.11 it was CE1613, unrelated to the user task. The task now targets groups by XPath. Re-measured: check -p reports the expected gates on 9.24.30, 10.11.0 and 10.24.24. On 11.13.0 exec writes all five workflows with 0 errors. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../2026-10-29-user-task-type-below-10-12.json | 2 +- CHANGELOG.md | 2 +- .../1771-alter-user-task-targeting.mdl | 1 + .../bug-tests/1771-user-task-below-10-12.mdl | 4 ++++ .../1771-user-task-features-by-version.mdl | 17 +++++++---------- .../1771-user-task-targeting-below-11-2.mdl | 2 ++ 6 files changed, 16 insertions(+), 12 deletions(-) diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json index 6421029ff..137796213 100644 --- a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json @@ -1 +1 @@ -{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 9.x a workflow with a user task made the project unloadable. exec reported 'Created workflow', then mx check 9.24.30 and Studio Pro died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity`. The same happened on 10.0–10.11 by the SDK's version data, which was not measured there", "cause": "userTaskToGen (and the ALTER mutator's inserts, through it) writes Workflows$SingleUserTaskActivity / MultiUserTaskActivity, TYPES introduced in 10.12.0, which also deleted Workflows$UserTask (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared the written $Type with the project's version. Before 10.12 the kind of task is the UserTask's UserTaskCompletion (9.22): SingleInputCompletion, or MultiInputCompletion {CompletionCriteria, TargetUserInput, AwaitAllUsers from 10.2}", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTaskType, dropTaskBoundaryEvents, userTaskActivitySince, completionCriteriaSince), `mdl/backend/modelsdk/workflow_read.go` (UserTask case reads the MultiInputCompletion), `mdl/executor/validate_workflow_rewrite.go` (rawMultiUserTaskSettings sees a 9.x multi-user task)", "insight": "**A type rename can also be a restructure.** Converting MultiUserTaskActivity down is more than a $Type swap. Three of its keys move into a nested MultiInputCompletion, and the outcomes stay on the task, so the CompletionCriteria's outcome pointers still resolve. Studio Pro 9.24.3's key set is the target: Multilevel Dynamic Workflow (Marketplace 221741) stores `$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent,Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion`. **With no Studio Pro sample of a 9.x multi-user task, convert UP as the oracle.** The 9.24 output, after `mx convert -p` 10.24.24, has user tasks identical in keys and values to what mxcli writes on 10.24 directly, rule and participants included. **Every reader of the old shape needs the restructure too.** describe read a 9.x multi-user task as a single one, and the create-or-modify guard looked for the rule only on MultiUserTaskActivity, so a 9.24 rewrite would reset the rule silently. **Give an element the lowering ADDS a derived $ID** (completionID from the task's $ID), so the same task lowers to the same bytes. The boundary events on a user task (10.14, declared on the abstract UserTaskActivity) slipped past the floor the same way: an empty list is dropped below 10.14, and a boundary event is refused. Measured: bug-test on fresh apps. 9.24.30 before TypeCacheUnknownTypeException, after 0 errors. `mx convert -p` 9.24.30 changes no key and no value. describe → exec 'Unchanged workflow'. 10.24.24 and 11.13.0 give 0 errors. The CE2513 'Java 11' after converting up is in an untouched 9.24 app too. Control: stubbing lowerUserTaskType fails TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12, …MultiUserTask… and …RefusesWhatAnOlderUserTaskCannotHold with the 10.12 types on 9.24. The reader and guard tests failed before their fixes. Tests: those three, TestNormalize_UserTaskCompletionIDIsStable, TestWorkflowRead_UserTaskInTheMendix9Shape, TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings; mdl-examples/bug-tests/1771-user-task-below-10-12.mdl", "refs": ["#1771", "#1163"]} +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 9.x a workflow with a user task made the project unloadable. exec reported 'Created workflow', then mx check 9.24.30 and Studio Pro died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity`. The same happened on 10.11.0 (measured), and on all of 10.0–10.11 by the SDK's version data", "cause": "userTaskToGen (and the ALTER mutator's inserts, through it) writes Workflows$SingleUserTaskActivity / MultiUserTaskActivity, TYPES introduced in 10.12.0, which also deleted Workflows$UserTask (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared the written $Type with the project's version. Before 10.12 the kind of task is the UserTask's UserTaskCompletion (9.22): SingleInputCompletion, or MultiInputCompletion {CompletionCriteria, TargetUserInput, AwaitAllUsers from 10.2}", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTaskType, dropTaskBoundaryEvents, userTaskActivitySince, completionCriteriaSince), `mdl/backend/modelsdk/workflow_read.go` (UserTask case reads the MultiInputCompletion), `mdl/executor/validate_workflow_rewrite.go` (rawMultiUserTaskSettings sees a 9.x multi-user task)", "insight": "**A type rename can also be a restructure.** Converting MultiUserTaskActivity down is more than a $Type swap. Three of its keys move into a nested MultiInputCompletion, and the outcomes stay on the task, so the CompletionCriteria's outcome pointers still resolve. Studio Pro 9.24.3's key set is the target: Multilevel Dynamic Workflow (Marketplace 221741) stores `$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent,Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion`. **With no Studio Pro sample of a 9.x multi-user task, convert UP as the oracle.** The 9.24 output, after `mx convert -p` 10.24.24, has user tasks identical in keys and values to what mxcli writes on 10.24 directly, rule and participants included. **Every reader of the old shape needs the restructure too.** describe read a 9.x multi-user task as a single one, and the create-or-modify guard looked for the rule only on MultiUserTaskActivity, so a 9.24 rewrite would reset the rule silently. **Give an element the lowering ADDS a derived $ID** (completionID from the task's $ID), so the same task lowers to the same bytes. The boundary events on a user task (10.14, declared on the abstract UserTaskActivity) slipped past the floor the same way: an empty list is dropped below 10.14, and a boundary event is refused. Measured: bug-test on fresh apps. 9.24.30 before TypeCacheUnknownTypeException, after 0 errors. `mx convert -p` 9.24.30 changes no key and no value. describe → exec 'Unchanged workflow'. 10.24.24 and 11.13.0 give 0 errors. The CE2513 'Java 11' after converting up is in an untouched 9.24 app too. 10.11.0.37074 (downloaded for this, then deleted): before (lowering stubbed) TypeCacheUnknownTypeException, which confirms the SDK's 10.12 type boundary on the platform. After, 0 errors with AwaitAllUsers in the MultiInputCompletion, mx convert 10.11 changes nothing, and describe → exec 'Unchanged workflow'. Majority, await all users and a microflow rule written on 10.11 check clean. Control: stubbing lowerUserTaskType fails TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12, …MultiUserTask… and …RefusesWhatAnOlderUserTaskCannotHold with the 10.12 types on 9.24. The reader and guard tests failed before their fixes. Tests: those three, TestNormalize_UserTaskCompletionIDIsStable, TestWorkflowRead_UserTaskInTheMendix9Shape, TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings; mdl-examples/bug-tests/1771-user-task-below-10-12.mdl", "refs": ["#1771", "#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 265ecc5a4..1084df4b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **User-task features an older Mendix cannot store are refused up front** (ako/mxcli#1771) — `mxcli check -p` and `exec` report MDL-VERSION01 before anything is written, naming the clause. The floors are group targeting (`targeting groups …`) 11.2, a multi-user task 9.22, `decide by majority` or `threshold` 10.0, `await all users` 10.2, and `decide by microflow` 10.3. Before, group targeting was written on 10.24 and failed `mx check` with CE1859. The others were refused only at write time, after the statements before them were written. Measured on 9.24.30 and 10.24.24 (each refused, nothing written) and on 11.13.0 (all written, 0 errors). - **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. -- **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. +- **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30 and 10.11.0: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. diff --git a/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl index dff335a7a..31bb7b710 100644 --- a/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl +++ b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl @@ -15,6 +15,7 @@ mdl 1; -- 9.24.30.51827, 10.24.24.119653 before: the old targeting was kept. After: -- `[Name = 'reviewer']` and ACT_GetReviewers. 0 errors, with -- the version's key set (Workflows$UserTask on 9.24). +-- 10.11.0.37074 after: the new targeting, 0 errors. -- 11.13.0 the new targeting both before and after, 0 errors. create module Probe1771At; diff --git a/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl index bf4fe380e..d5c3934eb 100644 --- a/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl +++ b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl @@ -25,6 +25,10 @@ mdl 1; -- describe → exec prints "Unchanged workflow". Converted up -- to 10.24.24, the tasks are those mxcli writes on 10.24 -- directly. +-- 10.11.0.37074 before: TypeCacheUnknownTypeException. After: 0 errors, +-- each multi-user task's completion with AwaitAllUsers (10.2+). +-- `mx convert -p` 10.11 changes nothing. describe → exec +-- prints "Unchanged workflow". -- 10.24.24, 11.13.0 0 errors, with the same key sets as before the fix. create module Probe1771Ut; diff --git a/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl index f84d8326f..d7bf181e9 100644 --- a/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl +++ b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl @@ -19,21 +19,18 @@ mdl 1; -- EXPECTED: `mxcli check` passes this file. With -p on a project below a -- floor, each workflow that states the feature is reported as MDL-VERSION01, -- naming it. Measured on fresh apps: --- 9.24.30.51827 WF_Majority, WF_Await, WF_Microflow, WF_Groups --- 10.24.24.119653 WF_Groups +-- 9.24.30.51827 WF_Majority, WF_Await, WF_Microflow, WF_Groups; exec +-- writes nothing. +-- 10.11.0.37074 WF_Groups. Without it, exec writes the other four as +-- Workflows$UserTask with a MultiInputCompletion, and mx +-- check reports 0 errors. +-- 10.24.24.119653 WF_Groups. -- 11.13.0 none; exec writes all five, and mx check reports 0 errors. create module Probe1771Gt; create persistent entity Probe1771Gt.Request ( Title: String(100) ); -create microflow Probe1771Gt.ACT_Groups ($Workflow: System.Workflow, $Context: Probe1771Gt.Request) -returns list of System.WorkflowGroup as $Groups -begin - retrieve $Groups from System.WorkflowGroup; - return $Groups; -end; - create microflow Probe1771Gt.ACT_Decide ($WorkflowUserTask: System.WorkflowUserTask) returns String as $Outcome begin @@ -93,6 +90,6 @@ create workflow Probe1771Gt.WF_Groups begin user task Review 'Review' page Probe1771Gt.TaskPage - targeting groups microflow Probe1771Gt.ACT_Groups + targeting groups xpath [Name = 'Reviewers'] outcomes 'Done' { }; end workflow; diff --git a/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl index c23a49cc1..86737d195 100644 --- a/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl +++ b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl @@ -26,6 +26,8 @@ mdl 1; -- 10.24.24.119653 before: CE1859 on every task. After: 0 errors. -- `mx convert -p` 10.24.24 changes no key and no value. -- describe → exec prints "Unchanged workflow". +-- 10.11.0.37074 0 errors. The task is a Workflows$UserTask there (see +-- 1771-user-task-below-10-12.mdl), with the same UserSource. -- 11.13.0 0 errors before and after, with the same key sets. create module Probe1771Tg; From f62863bc653bb2ebccf6a4192d554f34c34ba135 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 09:59:53 +0000 Subject: [PATCH 41/61] ci: add Mendix 11.16.0 to the nightly test matrix Mendix 11.16.0 is published on the CDN (mxbuild-11.16.0.tar.gz, 27 Oct 2026) but was not exercised by CI. Added as a seventh entry rather than replacing an older one: 11.13.0 is the measured floor for timer-started workflow event sub-processes, and 11.15.0 is the first version with the MessageDefinition2 document shape (ako/mxcli#987), so each keeps a job running at its own gate. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82teqHJLFTo5pLuHMt1 --- .github/workflows/nightly.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index da3b5719e..6ea88b119 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -26,6 +26,7 @@ jobs: - '11.13.0' - '11.14.0' - '11.15.0' + - '11.16.0' fail-fast: false name: test (Mendix ${{ matrix.mendix-version }}) steps: From 2788811a1437b4e28874714a8f282cb41c6c7044 Mon Sep 17 00:00:00 2001 From: engalar Date: Fri, 30 Oct 2026 10:53:56 +0800 Subject: [PATCH 42/61] =?UTF-8?q?fix(git):=20the=20git-merge=20fixtures=20?= =?UTF-8?q?run=20git=20hermetically=20and=20show=20what=20the=20merge=20sa?= =?UTF-8?q?id=20=E2=80=94=20CI's=20modelConflictRepo=20saw=20no=20unresolv?= =?UTF-8?q?ed=20files=20on=20ubuntu,=20and=20the=20fixture=20threw=20the?= =?UTF-8?q?=20merge's=20output=20away?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- cmd/mxcli/gitdoctor_test.go | 10 ++++++++++ cmd/mxcli/gitmerge_test.go | 10 +++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index a621e2569..aeb61e2fd 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -177,10 +177,20 @@ func TestGitDoctor_SSHURLFormRewrittenByFix(t *testing.T) { // doctorNoteRepo makes a two-commit repository holding an MPR v2 project: // the base commit, then a page retitle. Returns dir and the .mpr path. +// +// The fixture's git is hermetic: no system or global config leaks in from the +// machine this runs on (runner images, developer setups and CI images differ), +// bytes are committed exactly as written, and no repository above the temp +// dir is discovered. Without this, git-merge fixtures see a different merge +// on each machine and fail with the merge's own evidence thrown away. func doctorNoteRepo(t *testing.T) (string, string) { t.Helper() dir := t.TempDir() + t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) + t.Setenv("GIT_CONFIG_NOSYSTEM", "1") + t.Setenv("GIT_CEILING_DIRECTORIES", filepath.Dir(dir)) gitIn(t, dir, "init", "-q", "-b", "feature") + gitIn(t, dir, "config", "core.autocrlf", "false") units := baseUnits(t) writeProject(t, dir, "11.13.0", units) gitIn(t, dir, "add", "-A") diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 668feed6d..2a905d783 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -40,12 +40,12 @@ func modelConflictRepo(t *testing.T) (string, string) { gitIn(t, dir, "add", "-A") gitIn(t, dir, "commit", "-q", "-m", "side change") gitIn(t, dir, "switch", "-q", "feature") - _, err := gitInE(t, dir, "merge", "side") + mergeOut, err := gitInE(t, dir, "merge", "side") if err != nil && !strings.Contains(err.Error(), "exit status") { - t.Fatalf("fixture: expected a conflicted merge, got %v", err) + t.Fatalf("fixture: expected a conflicted merge, got %v\n%s", err, mergeOut) } if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); !strings.Contains(uu, ".mxunit") { - t.Fatalf("fixture: expected unresolved model files, got %q", uu) + t.Fatalf("fixture: expected unresolved model files, got %q\nmerge said:\n%s", uu, mergeOut) } return dir, mprPath } @@ -70,9 +70,9 @@ func twoUnitConflictRepo(t *testing.T) (string, string) { gitIn(t, dir, "add", "-A") gitIn(t, dir, "commit", "-q", "-m", "side change") gitIn(t, dir, "switch", "-q", "feature") - _, err := gitInE(t, dir, "merge", "side") + mergeOut, err := gitInE(t, dir, "merge", "side") if err != nil && !strings.Contains(err.Error(), "exit status") { - t.Fatalf("fixture: expected a conflicted merge, got %v", err) + t.Fatalf("fixture: expected a conflicted merge, got %v\n%s", err, mergeOut) } return dir, mprPath } From 69a1fdf516b9c8d333c9cba20cfd144a7dd4acd2 Mon Sep 17 00:00:00 2001 From: engalar Date: Fri, 30 Oct 2026 11:47:31 +0800 Subject: [PATCH 43/61] fix(git): the model merge runs mx merge over three project folders, gates every resolution on hashes, and aborts cleanly on every failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ako's review of #1674: --ours/--theirs took a whole side including the .mpr, so the other side's added units became files no longer in the project and deleted ones stayed listed with no file; --plan refused exactly that case and its error paths left MERGE_HEAD behind (Studio Pro re-runs the half-done merge on every open); every non-zero engine exit was read as a merge decision, so a crash (exit 129) wrote a plan and pointed at side-taking; and mx git-merge's file form does not run on Linux at all (its git lookup is Windows-bound). - the engine is called as mx merge BASE MINE THEIRS over the three project folders (Linux devcontainers included) through docker.PrepareMxCommand, and it owns the .mpr: its result carries both sides' document sets, so a unit one side added while the other changed existing ones is kept - exit 2 alone is a decision (report + plan template + clean abort); any other exit, and an mx that will not start, is reported as the engine failing and aborted cleanly - --ours/--theirs stay but their commit is hash-gated: when the sides' document sets differ the commit is refused (orphan/missing units) and the merge aborts — a whole side never silently drops documents - every --plan refusal aborts the merge (stale already did; invalid take, unreadable snapshots and the hash refusal now do too) - a merge already in progress is resumed instead of re-run, so the mixed-conflict advice (resolve text files, git add, re-run) works - .merge-plan.json is junk for mxcli git commit --- .claude/skills/mendix/teamserver-git/SKILL.md | 8 +- CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_commit.go | 4 + cmd/mxcli/cmd_git_merge.go | 282 ++++++++++++------ cmd/mxcli/gitmerge_test.go | 204 ++++++++++++- docs-site/src/tools/outside-studio-pro.md | 45 +-- .../2026-10-30-git-merge-mx-folder-mode.md | 57 ++++ 7 files changed, 473 insertions(+), 129 deletions(-) create mode 100644 docs/plans/2026-10-30-git-merge-mx-folder-mode.md diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index ea51485aa..944bff868 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -117,11 +117,13 @@ git push origin refs/notes/mx_metadata is a binary conflict), and a textual merge of `mprcontents/` leaves an index that does not match the files. Model-aware merging exists in two forms: **`mxcli git merge `** from the command line (it hands the - conflict to `mx git-merge`, Studio Pro's own engine, and commits the - resolved model as a normal merge commit with the note), and **Version + conflict to `mx merge` over three project folders — Studio Pro's own + engine, Linux devcontainers included — and commits the resolved model as a + normal merge commit with the note), and **Version Control › Merge Changes Here** in Studio Pro (*Merge feature branch* or *Cherry Pick*) — which is also where the conflicts mx cannot auto-resolve - go. A conflict `mxcli git merge` cannot resolve aborts the merge cleanly. + go. A conflict `mxcli git merge` cannot resolve — and an engine failure — + aborts the merge cleanly; nothing is left half-done. Studio Pro's history on Team Server is linear for a reason: do not "fix" a conflicted model merge by hand-editing files. - **No `--amend` or force-push of a pushed commit.** Before pushing, an amend is diff --git a/CHANGELOG.md b/CHANGELOG.md index e57c8e024..c7ea8705c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the run also writes a plan template (`.merge-plan.json`) beside the project, and the human then re-runs with `--plan ` to resolve **per unit** — each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files (modified-on-both units only; added/deleted/moved goes to Studio Pro) — or with `--ours`/`--theirs` to take one side wholesale, or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx merge BASE MINE THEIRS` over three project folders, which runs on Linux devcontainers too (the older `mx git-merge` file form is Windows-bound). `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`; a merge stopped on text conflicts resumes once those are resolved and staged), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three project folders to the engine, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything the engine auto-resolves is staged, hash-checked and committed as the normal two-parent merge commit with the mx_metadata note — and the engine owns the `.mpr`, so a unit one side added while the other changed existing ones is kept, not dropped; a conflict the engine reports as `ManualResolutionRequired` (exit 2) is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit at property level (from `mx diff`), with a plan template (`.merge-plan.json`, never staged by `mxcli git commit`) beside the project — the human re-runs with `--plan ` to resolve **per unit** (each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files; modified-on-both units only, added/deleted/moved goes to Studio Pro), or with `--ours`/`--theirs` to take one side of every conflicted unit — hash-gated: when the sides' document sets differ (one side added or deleted a unit) the commit is refused and the merge aborted, because a whole side would orphan the other side's documents — or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; any other engine exit (its 129, an mx that will not start) is reported as the engine failing and the merge is aborted cleanly, never steered into a decision. Non-model conflicts (text sources) are left in place like plain git: resolve them, `git add` them, re-run the command. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index a51d88bf2..27e575613 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -261,6 +261,10 @@ func isJunkPath(rel, projDir string) bool { return true case strings.HasSuffix(strings.ToLower(base), ".mpr.lock"), strings.HasSuffix(strings.ToLower(base), ".mpr.bak"): return true + // `mxcli git merge` writes the plan template beside the .mpr for the + // human to edit; it is working material, never project content. + case strings.HasSuffix(strings.ToLower(base), ".merge-plan.json"): + return true } inProject := f if projDir != "." && projDir != "" { diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index 8d4679e54..3aeecbe6b 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -22,53 +22,71 @@ import ( // cmd_git_merge.go: `mxcli git merge ` — merge a branch the way Studio // Pro does. `git merge` runs first; when it stops on the model files (the // .mpr is SQLite and every .mxunit is BSON, so any two-sided model change is -// a binary conflict), the conflict is handed to `mx git-merge` — the same -// model-aware engine Studio Pro's own merge uses — with the three revisions -// checked out into temporary worktrees. What mx resolves is committed as a -// normal two-parent merge commit with the mx_metadata note; what it reports -// as ManualResolutionRequired aborts the merge with the conflict list and a -// pointer to Studio Pro, leaving nothing behind. +// a binary conflict), the three revisions are handed to `mx merge` — the same +// model-aware engine Studio Pro's own merge uses, over the base and theirs +// checkouts and the working tree — with the project's Mendix version's mx. +// What the engine resolves is staged, hash-checked and committed as a normal +// two-parent merge commit with the mx_metadata note; what it reports as +// ManualResolutionRequired (exit 2) aborts the merge with the conflict list +// and a pointer to Studio Pro, leaving nothing behind; any other non-zero +// exit is the engine failing, reported as such, also with a clean abort. var gitMergeCmd = &cobra.Command{ Use: "merge ", - Short: "Merge a branch with model-aware conflict resolution (mx git-merge)", + Short: "Merge a branch with model-aware conflict resolution (mx merge)", Long: `Merge into the current branch, resolving model conflicts the way Studio Pro does. 'git merge ' runs first. A merge git resolves itself (disjoint files, a fast-forward) is committed and annotated like 'mxcli git commit' does. When git stops on the model files — the .mpr and the .mxunit units are binary, so -two sides changing the model always conflict — the three revisions are -checked out into temporary worktrees and 'mx git-merge BASE MINE THEIRS' -(Studio Pro's own model-aware merge, resolved for the project's Mendix -version) decides: - - - everything mx auto-resolves is staged and committed as the merge commit, - with the mx_metadata note attached (this keeps both sides — the usual - best outcome); - - a conflict mx reports as ManualResolutionRequired is never merged - silently: the merge is aborted, and both sides' changes are printed for - review — at property level, from 'mx diff' (e.g. +two sides changing the model always conflict — the base and theirs revisions +are checked out into temporary worktrees and the three project folders go to +'mx merge BASE MINE THEIRS' (Studio Pro's own model-aware merge, resolved for +the project's Mendix version): + + - everything the engine auto-resolves is staged, hash-checked and committed + as the merge commit, with the mx_metadata note attached (this keeps both + sides — the usual best outcome; the engine owns the .mpr, so a unit one + side added while the other changed existing ones is kept, not dropped); + - a conflict the engine reports as ManualResolutionRequired (exit 2) is + never merged silently: the merge is aborted, and both sides' changes are + printed for review — at property level, from 'mx diff' (e.g. "New type > Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: --plan resolve per unit from the plan template written beside the .mpr (.merge-plan.json): set "take" to "ours"/"theirs" per unit; the .mpr index is repaired from the mixed files. Modified-on-both units only — added/deleted/moved conflicts go to Studio Pro. - --ours / --theirs take one side of every conflicted model file - (deterministic; mx is not consulted again) + --ours / --theirs take one side of every conflicted unit + (deterministic; mx is not consulted again). The .mpr is + taken from that side too and the result is hash-checked: + when the sides' document sets differ (one added or + deleted a unit), the commit is refused — a whole side + would leave the other side's units orphaned on disk or + listed with no file — and the merge is aborted. Resolve + those in Studio Pro instead. Studio Pro (Version Control › Merge Changes Here) for mixed within-unit resolutions — still the tool when a decision must mix both sides inside one unit; + - any other engine exit is mx failing, not a decision: it is reported and + the merge is aborted cleanly; - a conflict in non-model files (text sources) is left in place for manual - resolution, like plain git. + resolution, like plain git: resolve them, 'git add' them, and re-run + this command — it finishes the merge (model conflicts included) and + commits. + +--ours/--theirs take one side of every conflicted unit — never a side of the +.mpr without the hash check, because the .mpr lists every document: a whole +side would drop the other side's added units and resurrect its deleted ones. The merge is refused while Studio Pro has the project open (.mpr.lock). MPR v2 projects only. The mx binary is resolved for the project's Mendix version (PATH, the mxcli cache, or the Studio Pro installation).`, Example: ` mxcli git merge feature/orders -p app.mpr mxcli git merge origin/main -p app.mpr --push - mxcli git merge feature/orders -p app.mpr --theirs # after reviewing the report`, + mxcli git merge feature/orders -p app.mpr --theirs # after reviewing the report + mxcli git merge feature/orders -p app.mpr --plan app.mpr.merge-plan.json`, Args: cobra.ExactArgs(1), RunE: runGitMerge, SilenceUsage: true, @@ -105,11 +123,17 @@ func runGitMerge(cmd *cobra.Command, args []string) error { case theirs: strategy = "theirs" } - return mergeRun(project, ref, message, strategy, plan, push, mxRunners{merge: realMxGitMerge, diff: realMxDiff}, cmd.OutOrStdout()) + return mergeRun(project, ref, message, strategy, plan, push, mxRunners{merge: realMxMerge, diff: realMxDiff}, cmd.OutOrStdout()) } // mxRunners are the mx invocations, injected for tests. -type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, error) +// +// merge runs the engine over the three PROJECT FOLDERS — the base checkout, +// the working tree, the theirs checkout — and writes its result into the +// working tree, App.mpr included. Exit 0 is a resolved merge, 2 is a decision +// the engine could not make (its per-unit list is on stdout), anything else +// is the engine failing. +type mxMergeRunner func(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error) type mxDiffRunner func(base, mine, outJSON string) error type mxRunners struct { @@ -117,19 +141,20 @@ type mxRunners struct { diff mxDiffRunner } -// mergeRun is the command without its wiring. runMx is injected for tests. -// mergeRun is the command without its wiring. runMx is injected for tests. +// mergeRun is the command without its wiring. runners are injected for tests. // // Conflict handling is a three-step flow a human stays in charge of: // // 1. `git merge` runs; a merge it resolves itself is annotated and done. -// 2. A model conflict goes to `mx git-merge` first — what it auto-resolves -// keeps both sides, which is almost always the best outcome — and is -// committed with the note. -// 3. What mx reports as ManualResolutionRequired is never merged silently: -// with no strategy the merge is aborted and both sides' changes are -// printed for review; `--ours`/`--theirs` then take one side wholesale -// (mx is not consulted — the human has decided). +// 2. A model conflict goes to `mx merge` over the three project folders +// first — what the engine auto-resolves keeps both sides, which is almost +// always the best outcome — and is committed with the note. +// 3. What the engine reports as ManualResolutionRequired (exit 2) is never +// merged silently: with no strategy the merge is aborted and both sides' +// changes are printed for review; `--plan` then resolves per unit, and +// `--ours`/`--theirs` take one side of every conflicted unit, hash-gated +// so a side that would drop or orphan documents is refused. Any other +// engine exit is a failure, not a decision: reported, aborted cleanly. func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners mxRunners, out io.Writer) error { switch strategy { case "", "ours", "theirs": @@ -161,20 +186,24 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return applyPlan(root, repo, abs, mprPath, ref, plan, message, branch, push, out) } - mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) - if mergeErr == nil { - if strings.Contains(mergeOut, "Already up to date") { - fmt.Fprintln(out, "Already up to date.") - return nil - } - head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) - fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) - return finishMerge(out, repo, head, branch, push) - } - - // A ref that does not resolve fails before any conflict exists. + // A merge already in progress is resumed, not re-run: a previous run may + // have stopped on text conflicts, which the human resolved and staged + // since. Re-running git merge would only refuse ("not concluded"). if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + if strings.Contains(mergeOut, "Already up to date") { + fmt.Fprintln(out, "Already up to date.") + return nil + } + head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) + return finishMerge(out, repo, head, branch, push) + } + // A ref that does not resolve fails before any conflict exists. + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } } conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) @@ -187,7 +216,7 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m } } if len(other) > 0 { - fmt.Fprintln(out, "Non-model conflicts, resolve by hand, then 'git add' and 'git commit':") + fmt.Fprintln(out, "Non-model conflicts, resolve by hand, then 'git add' them and re-run this command to resolve the model and commit:") for _, p := range other { fmt.Fprintf(out, " %s\n", p) } @@ -198,27 +227,34 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) - // The human has decided: take one side of every conflicted model file. - // Deterministic, and mx is not consulted — `git checkout --ours/--theirs` - // reads the conflicted stages directly. + // The human has decided: take one side of every conflicted model file. mx + // is not consulted — `git checkout --ours/--theirs` reads the conflicted + // stages directly. The .mpr is taken from the side too, and the result is + // hash-checked before committing: a whole side includes the .mpr's list of + // documents, so when the sides' document sets differ (one side added or + // deleted a unit) committing would orphan the other side's units or list + // units with no file. That refusal aborts the merge — it never commits a + // half-project. if strategy != "" { side := "--" + strategy for _, p := range model { if _, err := gitOutIn(root, nil, "checkout", side, "--", p); err != nil { - return fmt.Errorf("take the %s version of %s: %w", strategy, p, err) + return abortMerge(root, fmt.Errorf("take the %s version of %s: %w", strategy, p, err)) } } - args := []string{"add", "--"} - args = append(args, model...) - if _, err := gitOutIn(root, nil, args...); err != nil { - return fmt.Errorf("stage the resolved model files: %w", err) + if err := repairOrRefuse(root, abs, out); err != nil { + return err } return commitTheMerge(root, repo, model, message, branch, push, out) } - // mx first: what it auto-resolves keeps both sides, which is almost - // always better than taking one side wholesale. - // Check out the base and theirs revisions; MINE is the working tree. + // The engine first: what it auto-resolves keeps both sides, which is + // almost always better than taking one side wholesale. The base and + // theirs revisions are checked out as folders; MINE is the working tree. + // The engine writes its result into the working tree and owns the .mpr — + // its unit list carries both sides' added and deleted documents, so a + // unit one side added while the other changed existing ones is kept, not + // dropped. tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") if err != nil { return err @@ -243,10 +279,13 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m version := projectMendixVersion(mprPath) fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", strings.Join(shortNames(model), ", "), version) + // Folder mode: the engine takes the three project folders and writes its + // result into MINE — the working tree. + projRel := filepath.FromSlash(filepath.Dir(repo.MPRPath)) // "." at the repo root code, mxOut, runErr := runners.merge( - filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), - abs, - filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), + filepath.Join(baseDir, projRel), + filepath.Dir(abs), + filepath.Join(theirsDir, projRel), [3]string{baseSha, oursSha, theirsSha}, ) if s := strings.TrimSpace(mxOut); s != "" { @@ -254,9 +293,16 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m } if runErr != nil { cleanWorktrees() - return fmt.Errorf("mx git-merge failed: %w", runErr) + return abortMerge(root, fmt.Errorf("mx merge failed to run: %w", runErr)) + } + if code != 0 && code != 2 { + // A crash, an unreadable base, anything but a merge decision: report + // the engine as failing and leave nothing behind. Only exit 2 asks + // for a decision. + cleanWorktrees() + return abortMerge(root, fmt.Errorf("mx merge failed (exit %d); the merge is aborted — if it persists, resolve in Studio Pro (Version Control › Merge Changes Here)", code)) } - if code != 0 { + if code == 2 { // The review step: what each side changed, at property level (mx // diff), plus a per-unit plan template. Then leave nothing behind — // the human decides. @@ -279,14 +325,20 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return fmt.Errorf("the model merge has conflicts and aborting it failed (%v); resolve in Studio Pro", abortErr) } fmt.Fprintln(out, "Merge aborted; the working tree is back to the pre-merge state.") - return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, --ours/--theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") + return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, or resolve in Studio Pro (Version Control › Merge Changes Here)") } + // Exit 0: the engine resolved the model into the working tree, .mpr + // included — both sides' document sets are in it. Stage, gate, commit. args := []string{"add", "--"} args = append(args, model...) if _, err := gitOutIn(root, nil, args...); err != nil { cleanWorktrees() - return fmt.Errorf("stage the resolved model files: %w", err) + return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) + } + if err := repairOrRefuse(root, abs, out); err != nil { + cleanWorktrees() + return err } if err := commitTheMerge(root, repo, nil, message, branch, push, out); err != nil { cleanWorktrees() @@ -296,6 +348,34 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return nil } +// abortMerge backs a failed resolution out entirely — working tree, index and +// MERGE_HEAD return to the pre-merge state — and returns the original error. +// Every refusal on this command ends here: a half-done merge makes Studio Pro +// re-run it (and choke) every time the project opens. +func abortMerge(root string, err error) error { + if _, abortErr := gitOutIn(root, nil, "merge", "--abort"); abortErr != nil { + return fmt.Errorf("%w (aborting the merge failed too: %v — resolve it in Studio Pro)", err, abortErr) + } + return err +} + +// repairOrRefuse is the hash gate every side- or plan-based resolution ends +// with: ContentsHash values are repaired from the files on disk (the files +// are the truth once units have been taken from both sides), and a MISSING or +// ORPHAN entry — a unit the .mpr lists with no file behind it, or a file no +// .mpr entry lists — refuses the commit: committing it would drop or orphan +// documents. The refusal aborts the merge. +func repairOrRefuse(root, mprPath string, out io.Writer) error { + err := fixHashes(mprPath, true, out) + if err == nil { + return nil + } + if !errors.Is(err, errHashIssuesRemain) { + return abortMerge(root, err) + } + return abortMerge(root, errors.New("refusing to commit: the ContentsHash index has MISSING or ORPHAN entries (see above) — the sides' document sets differ, and committing would drop or orphan units; resolve in Studio Pro (Version Control › Merge Changes Here)")) +} + // commitTheMerge commits the resolved merge (MERGE_HEAD is present), writes // the note and pushes. stagedPaths, when given, are staged first. func commitTheMerge(root string, repo *gitNoteRepo, stagedPaths []string, message, branch string, push bool, out io.Writer) error { @@ -370,15 +450,19 @@ func projectMendixVersion(mprPath string) string { return v } -// realMxGitMerge shells out to `mx git-merge -v …`, resolved for the -// project's Mendix version. -func realMxGitMerge(base, mine, theirs string, shas [3]string) (int, string, error) { - version := projectMendixVersion(mine) +// realMxMerge shells out to `mx merge BASE MINE THEIRS` over the three +// project folders, mx resolved for the project's Mendix version. +func realMxMerge(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error) { + version := "" + if matches, _ := filepath.Glob(filepath.Join(mineDir, "*.mpr")); len(matches) > 0 { + version = projectMendixVersion(matches[0]) + } mxPath, err := docker.ResolveMxForVersion("", version) if err != nil { return 1, "", fmt.Errorf("no mx for Mendix %s: %w (model conflicts need it)", version, err) } - cmd := exec.Command(mxPath, "git-merge", "-v", base, mine, theirs, shas[0], shas[1], shas[2]) + cmd := exec.Command(mxPath, "merge", baseDir, mineDir, theirsDir) + docker.PrepareMxCommand(cmd) b, err := cmd.CombinedOutput() code := 0 if err != nil { @@ -521,9 +605,11 @@ func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, model []stri return os.WriteFile(mprPath+".merge-plan.json", b, 0o644) } -// applyPlan re-runs the merge (it must conflict the same way again), checks -// the plan against those revisions, resolves per unit, repairs the .mpr index -// from the mixed files and commits — with the note. +// applyPlan re-runs the merge unless one is already in progress (it must +// conflict the same way), checks the plan against those revisions, resolves +// per unit, repairs the .mpr index from the mixed files and commits — with +// the note. Every refusal after the merge started aborts it: a half-done +// merge makes Studio Pro re-run it (and choke) on every open. func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, message, branch string, push bool, out io.Writer) error { raw, err := os.ReadFile(planPath) if err != nil { @@ -539,19 +625,22 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m return fmt.Errorf("plan %s is not valid JSON: %w", planPath, err) } - mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) - if mergeErr == nil { - return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") - } + // An unfinished merge (text conflicts resolved and staged, say) is + // resumed; otherwise the merge runs and must conflict the same way. if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") + } + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } } baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) if plan.Base != baseSha || plan.Ours != oursSha || plan.Theirs != theirsSha { - _, _ = gitOutIn(root, nil, "merge", "--abort") - return errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate") + return abortMerge(root, errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate")) } conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) @@ -566,25 +655,25 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m // (added on one side, deleted, moved) only Studio Pro can decide. baseSnap, err := repo.snapshot(baseSha) if err != nil { - return err + return abortMerge(root, err) } oursSnap, err := repo.snapshot(oursSha) if err != nil { - return err + return abortMerge(root, err) } theirsSnap, err := repo.snapshot(theirsSha) if err != nil { - return err + return abortMerge(root, err) } oursStatus := changeStatuses(computeModelChanges(baseSnap, oursSnap, nil)) theirsStatus := changeStatuses(computeModelChanges(baseSnap, theirsSnap, nil)) take := map[string]string{} for _, u := range plan.Units { if u.Take != "ours" && u.Take != "theirs" { - return fmt.Errorf("plan: unit %s has take %q (ours|theirs)", shortCommit(u.ID), u.Take) + return abortMerge(root, fmt.Errorf("plan: unit %s has take %q (ours|theirs)", shortCommit(u.ID), u.Take)) } if oursStatus[u.ID] != "Modified" || theirsStatus[u.ID] != "Modified" { - return fmt.Errorf("plan: unit %s is %s on ours and %s on theirs — only modified-on-both conflicts can be planned; resolve it in Studio Pro", shortCommit(u.ID), statusOrNone(oursStatus[u.ID]), statusOrNone(theirsStatus[u.ID])) + return abortMerge(root, fmt.Errorf("plan: unit %s is %s on ours and %s on theirs — only modified-on-both conflicts can be planned; resolve it in Studio Pro", shortCommit(u.ID), statusOrNone(oursStatus[u.ID]), statusOrNone(theirsStatus[u.ID]))) } take[u.ID] = u.Take } @@ -592,7 +681,7 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m // Ours everywhere, then the planned units from theirs. for _, p := range model { if _, err := gitOutIn(root, nil, "checkout", "--ours", "--", p); err != nil { - return fmt.Errorf("take the ours version of %s: %w", p, err) + return abortMerge(root, fmt.Errorf("take the ours version of %s: %w", p, err)) } } for _, id := range conflictedUnitIDs(model) { @@ -601,23 +690,22 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m } p := path.Join(repo.Contents, id[:2], id[2:4], id+".mxunit") if _, err := gitOutIn(root, nil, "checkout", "--theirs", "--", p); err != nil { - return fmt.Errorf("take the theirs version of %s: %w", p, err) + return abortMerge(root, fmt.Errorf("take the theirs version of %s: %w", p, err)) } fmt.Fprintf(out, " %s: theirs\n", shortCommit(id)) } // The .mpr index is ours; the mixed unit files have different bytes, so - // the hashes are rewritten from the files — files are the truth. - if err := fixHashes(mprPath, true, out); err != nil { - if !errors.Is(err, errHashIssuesRemain) { - return err - } - return errors.New("refusing to commit: the ContentsHash index has MISSING or ORPHAN entries (see above)") + // the hashes are rewritten from the files — files are the truth. The + // gate refuses (and aborts) when the sides' document sets differ: a unit + // theirs added is not in ours' .mpr, and committing would orphan it. + if err := repairOrRefuse(root, mprPath, out); err != nil { + return err } args := []string{"add", "--"} args = append(args, model...) if _, err := gitOutIn(root, nil, args...); err != nil { - return fmt.Errorf("stage the resolved model files: %w", err) + return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) } return commitTheMerge(root, repo, nil, message, branch, push, out) } @@ -637,14 +725,16 @@ func statusOrNone(s string) string { return s } -// realMxDiff shells out to `mx diff BASE MINE OUTPUT` (resolved like git-merge). +// realMxDiff shells out to `mx diff BASE MINE OUTPUT` (resolved like the merge). func realMxDiff(base, mine, outJSON string) error { version := projectMendixVersion(mine) mxPath, err := docker.ResolveMxForVersion("", version) if err != nil { return fmt.Errorf("no mx for Mendix %s: %w", version, err) } - if b, err := exec.Command(mxPath, "diff", base, mine, outJSON).CombinedOutput(); err != nil { + cmd := exec.Command(mxPath, "diff", base, mine, outJSON) + docker.PrepareMxCommand(cmd) + if b, err := cmd.CombinedOutput(); err != nil { return fmt.Errorf("mx diff: %v: %s", err, strings.TrimSpace(string(b))) } return nil diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 2a905d783..22ec07ba9 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -5,6 +5,7 @@ package main import ( "bytes" "encoding/json" + "errors" "fmt" "os" "path/filepath" @@ -19,7 +20,7 @@ func bsonKeyValue(key, value string) bson.E { } // `mxcli git merge ` — merge a branch with model-aware resolution: git -// merge first, and on a model-file conflict `mx git-merge` (the same engine +// merge first, and on a model-file conflict `mx merge` (the same engine // Studio Pro uses). The mx run is injected as runMx so the orchestration is // testable without a Mendix installation; the real runner shells out to mx. @@ -77,6 +78,36 @@ func twoUnitConflictRepo(t *testing.T) (string, string) { return dir, mprPath } +// addConflictRepo is a repository where each side ADDED a different unit: only +// the .mpr conflicts, and the sides' document sets differ — the class a whole +// side must refuse, never commit. +func addConflictRepo(t *testing.T) (string, string) { + t.Helper() + dir, mprPath := doctorNoteRepo(t) + ours := baseUnits(t) + oursAdd := "11111111-0000-0000-0000-000000000009" + ours[oursAdd] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Ours")} + writeProject(t, dir, "11.13.0", ours) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "ours adds a microflow") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + theirs := baseUnits(t) + theirsAdd := "11111111-0000-0000-0000-00000000000a" + theirs[theirsAdd] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Theirs")} + writeProject(t, dir, "11.13.0", theirs) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "theirs adds a microflow") + gitIn(t, dir, "switch", "-q", "feature") + mergeOut, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflicted merge, got %v\n%s", err, mergeOut) + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); !strings.Contains(uu, ".mpr") { + t.Fatalf("fixture: expected the .mpr conflict, got %q\nmerge said:\n%s", uu, mergeOut) + } + return dir, mprPath +} + // unitRel is the mprcontents path of a unit id inside a work tree. func unitRel(id string) string { return "mprcontents/" + id[:2] + "/" + id[2:4] + "/" + id + ".mxunit" @@ -132,25 +163,29 @@ func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { var gotBase, gotMine, gotTheirs string var gotShas [3]string + var baseHasMpr, theirsHasMpr bool runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas + // The worktrees are removed when the merge finishes; check inside. + _, baseHasMprErr := os.Stat(filepath.Join(base, "App.mpr")) + _, theirsHasMprErr := os.Stat(filepath.Join(theirs, "App.mpr")) + baseHasMpr, theirsHasMpr = baseHasMprErr == nil, theirsHasMprErr == nil return 0, "Merge successful.", nil }} var out bytes.Buffer if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } - if !strings.Contains(gotBase, "mx-git-lab") && !strings.Contains(filepath.Base(gotBase), ".mpr") { - // the base path must be an .mpr inside a temporary checkout - if filepath.Ext(gotBase) != ".mpr" { - t.Errorf("runner got base %q, want the base checkout's .mpr", gotBase) - } + // Folder mode: the three args are project folders, and MINE is the + // working tree — the engine writes its result there. + if gotMine != dir { + t.Errorf("runner's mine = %q, want the working project folder %q", gotMine, dir) } - if filepath.Base(gotMine) != "App.mpr" { - t.Errorf("runner's mine = %q, want the working .mpr", gotMine) + if !baseHasMpr { + t.Errorf("runner's base folder %q must hold the project's .mpr", gotBase) } - if filepath.Base(gotTheirs) != "App.mpr" { - t.Errorf("runner's theirs = %q, want the theirs checkout's .mpr", gotTheirs) + if !theirsHasMpr { + t.Errorf("runner's theirs folder %q must hold the project's .mpr", gotTheirs) } if gotShas[0] == "" || gotShas[1] == gotShas[0] || gotShas[2] == gotShas[1] { t.Errorf("shas must be base/ours/theirs distinct: %v", gotShas) @@ -267,7 +302,7 @@ func TestGitMerge_ConflictReportShowsBothSidesAndAborts(t *testing.T) { }} var out bytes.Buffer err := mergeRun(mprPath, "side", "", "", "", false, noMx, &out) - if err == nil || !strings.Contains(err.Error(), "--ours") { + if err == nil || !strings.Contains(err.Error(), "--plan") { t.Fatalf("the report must point at the resolution flags, got %v", err) } report := out.String() @@ -445,3 +480,150 @@ func TestGitMerge_ManualWritesPlanAndPropertyReport(t *testing.T) { } gitInE(t, dir, "merge", "--abort") // clean up the fixture } + +// A whole side of a project includes the .mpr's list of documents. When the +// sides' document sets differ (one side added a unit), taking --theirs would +// leave ours' unit on disk with no .mpr entry: the commit is refused and the +// merge aborted, never silently half-committed. +func TestGitMerge_SideTakingRefusesWhenDocumentSetsDiffer(t *testing.T) { + dir, mprPath := addConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--theirs must not consult mx") + return 0, "", nil + }} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "theirs", "", false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "refusing to commit") { + t.Fatalf("--theirs must refuse a side that drops ours' unit, got %v\n%s", err, out.String()) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the refusal must abort the merge") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// Exit 129 is the engine failing, not a merge decision: report it, abort +// cleanly, and never write a plan template. +func TestGitMerge_EngineCrashAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + crash := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 129, "Exception in MergeLib", nil + }} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", "", false, crash, &out) + if err == nil || !strings.Contains(err.Error(), "exit 129") { + t.Fatalf("want the engine-failure report, got %v\n%s", err, out.String()) + } + if strings.Contains(out.String(), "Plan template") { + t.Errorf("a crash must not produce a plan:\n%s", out.String()) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } + if _, err := os.Stat(mprPath + ".merge-plan.json"); !os.IsNotExist(err) { + t.Errorf("a crash must not write a plan template: %v", err) + } +} + +// mx not being found or not starting is the engine failing to run: the merge +// aborts with the reason, and nothing is left behind. +func TestGitMerge_EngineStartFailureAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + missing := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 0, "", errors.New("no mx for Mendix 11.13.0: not on PATH or in the cache (model conflicts need it)") + }} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", "", false, missing, &out) + if err == nil || !strings.Contains(err.Error(), "failed to run") { + t.Fatalf("want the mx failure, got %v\n%s", err, out.String()) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } +} + +// A plan whose take value is not ours/theirs is refused, and the refusal +// aborts the merge: nothing may be left half-done. +func TestGitMerge_PlanRefusalAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + base := strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")) + ours := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) + theirs := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")) + plan := fmt.Sprintf(`{"base":%q,"ours":%q,"theirs":%q,"units":[{"id":%q,"take":"sideways"}]}`, + base, ours, theirs, tDM) + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + noMx := mxRunners{} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "ours|theirs") { + t.Fatalf("want the invalid-take refusal, got %v\n%s", err, out.String()) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the refusal must abort the merge") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// Re-entry: with the merge already in progress (a previous run stopped and +// the human resolved what they handle by hand), re-running resolves the model +// and finishes — it does not re-run git merge and die on MERGE_HEAD. +func TestGitMerge_ResumesAMergeAlreadyInProgress(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + + calls := 0 + runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + calls++ + return 0, "Merge successful.", nil + }} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + if calls != 1 { + t.Errorf("engine runs = %d, want 1", calls) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } +} + +// The plan template is working material: `mxcli git commit` never stages it. +func TestJunkPath_CoversMergePlanTemplate(t *testing.T) { + if !isJunkPath("App.mpr.merge-plan.json", ".") { + t.Error("App.mpr.merge-plan.json must be junk for staging") + } + if !isJunkPath("app/Project.mpr.merge-plan.json", "app") { + t.Error("a project-folder plan template must be junk too") + } +} diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 0d2f0145b..da43493ea 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -219,7 +219,9 @@ the index, so whatever you had staged is left exactly as it was. `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` — there is nothing for git's textual merge to work on. Studio Pro's *Merge Changes Here* resolves it with a model-aware merge, and -that engine ships as a command: `mx git-merge BASE MINE THEIRS …`. +that engine ships as a command: `mx merge BASE MINE THEIRS` over three +project folders (it runs on Linux devcontainers too; the older `mx git-merge` +file form is Windows-bound). `mxcli git merge ` puts the two together: @@ -231,25 +233,32 @@ mxcli git merge origin/main -p app.mpr --push `git merge ` runs first; a merge git resolves itself (disjoint files, a fast-forward) is annotated with the mx_metadata note like `mxcli git commit` does. On a model-file conflict the base and theirs revisions are -checked out into temporary worktrees and `mx git-merge` runs (resolved for -the project's Mendix version): - -- what mx auto-resolves is staged and committed as the normal two-parent - merge commit, with the note attached; -- a conflict mx reports as `ManualResolutionRequired` is never merged - silently: the merge is aborted, and **both sides' changes are printed for - review** — at property level, from `mx diff` ("New type > Length: - 100 → 150" on ours, "→ 200" on theirs). Then decide: +checked out into temporary worktrees and the three project folders go to +`mx merge` (resolved for the project's Mendix version); the engine writes +its result into the working tree and owns the `.mpr`: + +- what the engine auto-resolves is staged, hash-checked and committed as the + normal two-parent merge commit, with the note attached. The engine's + `.mpr` carries both sides' document sets, so a unit one side added while + the other changed existing ones is kept, not dropped; +- a conflict the engine reports as `ManualResolutionRequired` (exit 2) is + never merged silently: the merge is aborted, and **both sides' changes are + printed for review** — at property level, from `mx diff` ("New type > + Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: `--plan ` resolves **per unit** from the plan template written beside - the .mpr (`.merge-plan.json`): set `"take"` to `ours`/`theirs` per - unit, and the `.mpr` index is repaired from the mixed files (modified-on- - both units only; added/deleted/moved conflicts go to Studio Pro). - `--ours` or `--theirs` takes one side of every conflicted model file - (deterministic; mx is not consulted again). Resolutions that must mix both - sides *inside* one unit stay in Studio Pro (*Version Control › Merge - Changes Here*); + the .mpr (`.merge-plan.json`, which `mxcli git commit` never stages): + set `"take"` to `ours`/`theirs` per unit, and the `.mpr` index is repaired + from the mixed files (modified-on-both units only; added/deleted/moved + conflicts go to Studio Pro). `--ours` or `--theirs` takes one side of every + conflicted unit, hash-gated: when the sides' document sets differ (one side + added or deleted a unit), committing a whole side would orphan the other + side's documents, so the commit is refused and the merge aborted. Any other + engine exit is mx failing, not a decision: it is reported and the merge is + aborted cleanly. Resolutions that must mix both sides *inside* one unit + stay in Studio Pro (*Version Control › Merge Changes Here*); - a conflict in non-model files (text sources) is left in place for manual - resolution, like plain git. + resolution, like plain git: resolve them, `git add` them, and re-run the + command — it finishes the merge (model conflicts included) and commits. Merging is still never done by plain `git merge` on a Mendix repository: use `mxcli git merge` or Studio Pro, never a textual merge of `mprcontents/`. diff --git a/docs/plans/2026-10-30-git-merge-mx-folder-mode.md b/docs/plans/2026-10-30-git-merge-mx-folder-mode.md new file mode 100644 index 000000000..85de356f7 --- /dev/null +++ b/docs/plans/2026-10-30-git-merge-mx-folder-mode.md @@ -0,0 +1,57 @@ +# Rework `mxcli git merge` per ako's review (PR #1674) + +Decisions (user, 2026-10-30): switch the engine call to `mx merge` (folder mode); +keep `--ours`/`--theirs` but limited to conflicted units with a hash gate; one +push after the rework passes `make ci-local`. + +## Review claims → verified + +1. --ours/--theirs take App.mpr wholesale → orphan/missing documents. TRUE (strategy path, cmd_git_merge.go). +2. --plan takes ours' App.mpr → refuses on theirs' adds; error paths leave MERGE_HEAD. TRUE (applyPlan; abort only on stale). +3. Every non-zero mx exit treated as decision-needed. TRUE (code != 0 → plan + decision error; 129 = crash). +4. mx git-merge broken on Linux ("Git not found", 129). Reviewer-measured; repo finding corroborates mx merge works (0/2/129). +5. Should-fix: upstream merge conflicts (DONE, 2ef7e1068); merge-plan.json not junk-pathed; mixed-conflict advice; PrepareMxCommand nit. + +## Changes + +### A. Engine interface (cmd_git_merge.go) +- `mxMergeRunner func(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error)` — folders. +- realMxMerge: `mx merge ` + docker.PrepareMxCommand (11.10 LD_PRELOAD nit). +- baseDir/theirsDir: temp worktrees (unchanged); project dir inside = filepath.Join(dir, projDirRel). +- mineDir = the working project dir (filepath.Dir(abs mpr)). + +### B. Exit-code honesty +- 0 → stage model paths → fixHashes gate (MISSING/ORPHAN → refuse + git merge --abort) → commitTheMerge. +- 2 → property report (mx diff) + plan template (units = engine JSON's manual units; fallback git's conflicted ids) + git merge --abort + decision error. +- other (129…) → "mx merge failed (exit N)" + git merge --abort. Never a plan, never a decision suggestion. + +### C. --ours/--theirs limited +- No engine call (unchanged). Conflicted UNITS only from git's list (skip the .mpr). +- App.mpr takes the flag's side; then fixHashes gate: MISSING/ORPHAN → refuse + git merge --abort (loud, Studio Pro pointer). No silent drops. + +### D. --plan aborts everywhere +- invalid JSON / invalid take / not-modified-on-both / fixHashes refusal → git merge --abort before returning. +- stale-plan path already aborts (keep). + +### E. Re-entry +- At entry (non-plan): if MERGE_HEAD already exists, skip `git merge` and go straight to conflict classification. Supports: mixed conflict → resolve text files → `git add` → re-run command finishes the merge. + +### F. Mixed-conflict advice +- Early-return kept, reworded: resolve the text files, `git add` them, then re-run `mxcli git merge ` (it resumes, resolves the model, commits). + +### G. merge-plan.json is junk +- isJunkPath: `.merge-plan.json` never staged by `mxcli git commit`. + +### H. Words +- Long help, CHANGELOG git-merge entry, docs-site outside-studio-pro.md, teamserver-git SKILL.md "aborts the merge cleanly" — match the new reality. + +### I. Tests (gitmerge_test.go) +- Runner fixtures → folder form (assert dirs end in the project folder, contain App.mpr). +- exit 129 → no plan text, MERGE_HEAD gone, tree restored, error names mx. +- --theirs with ours-added unit → refuse + abort (no ORPHAN commit, MERGE_HEAD gone). +- --plan error paths → MERGE_HEAD gone. +- re-entry: staged text resolution + MERGE_HEAD → finishes. +- isJunkPath: x.merge-plan.json → junk. + +## Gate +- go vet ./cmd/mxcli/; focused go test; then make ci-local; then one push. From 4fa453f152e8c1d3bb44244de276e345da64bec7 Mon Sep 17 00:00:00 2001 From: engalar Date: Fri, 30 Oct 2026 12:24:07 +0800 Subject: [PATCH 44/61] docs(skills): regenerate the teamserver-git section index after the mx merge wording --- .claude/skills/mendix/teamserver-git/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index 944bff868..9c4be9fe2 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -14,9 +14,9 @@ description: "Commit, branch and push a Mendix project in git the way Studio Pro | 42-55 | Branches: create them on the server | | 56-97 | Commit from the agent | | 98-112 | Push the branch and the notes together, and soon | -| 113-132 | Never rewrite or merge model history with git | -| 133-143 | Reading what Studio Pro recorded | -| 144-152 | Checklist | +| 113-134 | Never rewrite or merge model history with git | +| 135-145 | Reading what Studio Pro recorded | +| 146-154 | Checklist | # Working with Git on a Mendix Team Server Project From 82188cfb6042aa8b6b724ec68b395d84f7afbbd9 Mon Sep 17 00:00:00 2001 From: engalar Date: Sun, 1 Nov 2026 14:13:31 +0800 Subject: [PATCH 45/61] feat(git-merge): parse the engine's -v conflict list --- cmd/mxcli/cmd_git_merge.go | 44 ++++++++++++++++++++++++++++++++++++++ cmd/mxcli/gitmerge_test.go | 27 +++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index 3aeecbe6b..034d38af3 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -495,6 +495,50 @@ func conflictedUnitIDs(model []string) []string { return ids } +// engineUnitConflict is one entry of the engine's per-unit conflict list, +// printed only with `-v` as one JSON array on one stdout line. Only +// ManualResolutionDetails "ManualResolutionRequired" needs a decision; +// "AutoResolved" entries are already merged. +type engineUnitConflict struct { + UnitID string `json:"UnitId"` + UnitName *string `json:"UnitName"` + ModuleName string `json:"ModuleName"` + ContainmentName string `json:"ContainmentName"` + Type string `json:"Type"` + TreeConflict string `json:"TreeConflict"` + ConflictResolutionDetails string `json:"ConflictResolutionDetails"` +} + +// parseEngineConflicts finds the engine's conflict list in its output: the +// last line that parses as a JSON array of objects. nil when the engine +// printed none — the caller falls back to git's conflicted-file list. +func parseEngineConflicts(mxOut string) []engineUnitConflict { + var found []engineUnitConflict + for _, line := range strings.Split(mxOut, "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, "[") { + continue + } + var cs []engineUnitConflict + if err := json.Unmarshal([]byte(line), &cs); err == nil && len(cs) > 0 { + found = cs + } + } + return found +} + +// engineManualIDs keeps the entries that need a human decision; entries +// without a unit id say nothing decidable and are skipped. +func engineManualIDs(cs []engineUnitConflict) []string { + var ids []string + for _, c := range cs { + if c.ConflictResolutionDetails == "ManualResolutionRequired" && c.UnitID != "" { + ids = append(ids, c.UnitID) + } + } + return ids +} + // --- property-level conflict report and the per-unit plan ------------------- type mxDiffChange struct { diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 22ec07ba9..677cc63ef 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -627,3 +627,30 @@ func TestJunkPath_CoversMergePlanTemplate(t *testing.T) { t.Error("a project-folder plan template must be junk too") } } + +// The engine prints its per-unit conflict list only with -v, as one JSON +// array on one stdout line; the parser must find it in the engine's noise +// and keep only what needs a human decision. +func TestParseEngineConflicts_FindsArrayInEngineNoise(t *testing.T) { + mxOut := "Merging...\n" + `[{"UnitId":"0131b342-0000-0000-0000-000000000001","UnitName":null,"ModuleName":"FeedbackModule","ContainmentName":"DomainModel","Type":"MergeContents","TreeConflict":"None","ConflictResolutionDetails":"ManualResolutionRequired"},{"UnitId":"6a912436-0000-0000-0000-000000000002","UnitName":"ACT_Theirs","ModuleName":"MyFirstModule","ContainmentName":"Documents","Type":"AddFromHead","TreeConflict":"None","ConflictResolutionDetails":"AutoResolved"}]` + "\nDone.\n" + cs := parseEngineConflicts(mxOut) + if len(cs) != 2 || cs[0].UnitID != "0131b342-0000-0000-0000-000000000001" || cs[1].Type != "AddFromHead" { + t.Fatalf("parsed: %+v", cs) + } +} + +func TestParseEngineConflicts_NoneYieldsNil(t *testing.T) { + if cs := parseEngineConflicts("Merge successful.\n"); cs != nil { + t.Errorf("want nil, got %+v", cs) + } +} + +// Only ManualResolutionRequired entries need a decision; entries without a +// unit id say nothing decidable and are skipped. +func TestEngineManualIDs_FiltersAutoResolvedAndEmpty(t *testing.T) { + cs := parseEngineConflicts(`[{"UnitId":"aaa","ConflictResolutionDetails":"ManualResolutionRequired"},{"UnitId":"","ConflictResolutionDetails":"ManualResolutionRequired"},{"UnitId":"bbb","ConflictResolutionDetails":"AutoResolved"}]`) + ids := engineManualIDs(cs) + if len(ids) != 1 || ids[0] != "aaa" { + t.Errorf("manual ids: %v", ids) + } +} From 6608d2040b48a2973b72c7ca8ea6dfcc29e3f368 Mon Sep 17 00:00:00 2001 From: engalar Date: Sun, 1 Nov 2026 14:21:37 +0800 Subject: [PATCH 46/61] feat(git-merge): run mx merge over .mpr paths with MINE as a clean worktree --- cmd/mxcli/cmd_git_merge.go | 175 +++++++++++++++++++++++++++---------- cmd/mxcli/gitmerge_test.go | 146 ++++++++++++++++++++++++++++--- 2 files changed, 263 insertions(+), 58 deletions(-) diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index 034d38af3..3966d460c 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -8,6 +8,7 @@ import ( "errors" "fmt" "io" + "io/fs" "os" "os/exec" "path" @@ -23,9 +24,11 @@ import ( // Pro does. `git merge` runs first; when it stops on the model files (the // .mpr is SQLite and every .mxunit is BSON, so any two-sided model change is // a binary conflict), the three revisions are handed to `mx merge` — the same -// model-aware engine Studio Pro's own merge uses, over the base and theirs -// checkouts and the working tree — with the project's Mendix version's mx. -// What the engine resolves is staged, hash-checked and committed as a normal +// model-aware engine Studio Pro's own merge uses — as three .mpr paths over +// clean checkouts (base, ours at HEAD, theirs), never the working tree +// mid-merge, with the project's Mendix version's mx. The engine writes its +// result into MINE, which is copied back into the working tree. What the +// engine resolves is hash-checked, staged and committed as a normal // two-parent merge commit with the mx_metadata note; what it reports as // ManualResolutionRequired (exit 2) aborts the merge with the conflict list // and a pointer to Studio Pro, leaving nothing behind; any other non-zero @@ -40,15 +43,18 @@ Studio Pro does. 'git merge ' runs first. A merge git resolves itself (disjoint files, a fast-forward) is committed and annotated like 'mxcli git commit' does. When git stops on the model files — the .mpr and the .mxunit units are binary, so -two sides changing the model always conflict — the base and theirs revisions -are checked out into temporary worktrees and the three project folders go to -'mx merge BASE MINE THEIRS' (Studio Pro's own model-aware merge, resolved for -the project's Mendix version): - - - everything the engine auto-resolves is staged, hash-checked and committed - as the merge commit, with the mx_metadata note attached (this keeps both - sides — the usual best outcome; the engine owns the .mpr, so a unit one - side added while the other changed existing ones is kept, not dropped); +two sides changing the model always conflict — the base, ours (a clean +checkout of HEAD) and theirs revisions are checked out into temporary +worktrees and the three projects' .mpr files go to +'mx merge BASE MINE THEIRS' (Studio Pro's own model-aware merge, resolved +for the project's Mendix version). MINE is never the working tree mid-merge: +the engine chokes on the conflicted state. Its result is copied back: + + - everything the engine auto-resolves is copied back, hash-checked, + staged and committed as the merge commit, with the mx_metadata note + attached (this keeps both sides — the usual best outcome; the engine + owns the .mpr, so a unit one side added while the other changed + existing ones is kept, not dropped); - a conflict the engine reports as ManualResolutionRequired (exit 2) is never merged silently: the merge is aborted, and both sides' changes are printed for review — at property level, from 'mx diff' (e.g. @@ -128,12 +134,13 @@ func runGitMerge(cmd *cobra.Command, args []string) error { // mxRunners are the mx invocations, injected for tests. // -// merge runs the engine over the three PROJECT FOLDERS — the base checkout, -// the working tree, the theirs checkout — and writes its result into the -// working tree, App.mpr included. Exit 0 is a resolved merge, 2 is a decision -// the engine could not make (its per-unit list is on stdout), anything else +// merge runs the engine as `mx merge -v BASE MINE THEIRS` over the three +// revisions' project files — clean checkouts of base, ours (HEAD) and +// theirs — and writes its result into MINE. The -v conflict list (one JSON +// array on one stdout line) is what the review step reports from. Exit 0 is +// a resolved merge, 2 is a decision the engine could not make, anything else // is the engine failing. -type mxMergeRunner func(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error) +type mxMergeRunner func(baseMpr, mineMpr, theirsMpr string, shas [3]string) (int, string, error) type mxDiffRunner func(base, mine, outJSON string) error type mxRunners struct { @@ -146,8 +153,9 @@ type mxRunners struct { // Conflict handling is a three-step flow a human stays in charge of: // // 1. `git merge` runs; a merge it resolves itself is annotated and done. -// 2. A model conflict goes to `mx merge` over the three project folders -// first — what the engine auto-resolves keeps both sides, which is almost +// 2. A model conflict goes to `mx merge` over the three revisions' .mpr +// files (clean worktrees; the result is copied back from MINE) first — +// what the engine auto-resolves keeps both sides, which is almost // always the best outcome — and is committed with the note. // 3. What the engine reports as ManualResolutionRequired (exit 2) is never // merged silently: with no strategy the merge is aborted and both sides' @@ -249,29 +257,36 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m } // The engine first: what it auto-resolves keeps both sides, which is - // almost always better than taking one side wholesale. The base and - // theirs revisions are checked out as folders; MINE is the working tree. - // The engine writes its result into the working tree and owns the .mpr — - // its unit list carries both sides' added and deleted documents, so a - // unit one side added while the other changed existing ones is kept, not - // dropped. + // almost always better than taking one side wholesale. The three + // revisions are checked out as clean worktrees; MINE is the one at HEAD, + // never the working tree mid-merge — the engine chokes on the conflicted + // state (measured: it exits 129 when theirs deleted a document). Its + // result lands in MINE and is copied back. tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") if err != nil { return err } baseDir := filepath.Join(tmp, "base") + oursDir := filepath.Join(tmp, "ours") theirsDir := filepath.Join(tmp, "theirs") if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", baseDir, baseSha); err != nil { os.RemoveAll(tmp) return fmt.Errorf("check out the merge base: %w", err) } + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", oursDir, oursSha); err != nil { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + os.RemoveAll(tmp) + return fmt.Errorf("check out HEAD: %w", err) + } if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", theirsDir, theirsSha); err != nil { _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", oursDir) os.RemoveAll(tmp) return fmt.Errorf("check out %s: %w", ref, err) } cleanWorktrees := func() { _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", oursDir) _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", theirsDir) _ = os.RemoveAll(tmp) } @@ -279,13 +294,12 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m version := projectMendixVersion(mprPath) fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", strings.Join(shortNames(model), ", "), version) - // Folder mode: the engine takes the three project folders and writes its - // result into MINE — the working tree. - projRel := filepath.FromSlash(filepath.Dir(repo.MPRPath)) // "." at the repo root + // File form: the engine takes the three revisions' project files; MINE is + // the clean worktree at HEAD. code, mxOut, runErr := runners.merge( - filepath.Join(baseDir, projRel), - filepath.Dir(abs), - filepath.Join(theirsDir, projRel), + filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), + filepath.Join(oursDir, filepath.FromSlash(repo.MPRPath)), + filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), [3]string{baseSha, oursSha, theirsSha}, ) if s := strings.TrimSpace(mxOut); s != "" { @@ -306,10 +320,11 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m // The review step: what each side changed, at property level (mx // diff), plus a per-unit plan template. Then leave nothing behind — // the human decides. + oursMpr := filepath.Join(oursDir, filepath.FromSlash(repo.MPRPath)) if runners.diff != nil { oursJSON := filepath.Join(tmp, "ours-diff.json") theirsJSON := filepath.Join(tmp, "theirs-diff.json") - d1 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), abs, oursJSON) + d1 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), oursMpr, oursJSON) d2 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), theirsJSON) if d1 == nil && d2 == nil { fmt.Fprintln(out, propertyReport(oursJSON, theirsJSON, conflictedUnitIDs(model))) @@ -328,18 +343,27 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, or resolve in Studio Pro (Version Control › Merge Changes Here)") } - // Exit 0: the engine resolved the model into the working tree, .mpr - // included — both sides' document sets are in it. Stage, gate, commit. - args := []string{"add", "--"} - args = append(args, model...) - if _, err := gitOutIn(root, nil, args...); err != nil { + // Exit 0: the engine resolved the model into MINE, the clean worktree at + // HEAD. Copy its result into the working tree — units the result dropped + // go with it — gate the .mpr, and only then stage and commit: the gate + // repairs hashes from the files, so staging must come after it, or the + // commit would carry the pre-repair .mpr. + if err := copyEngineResult(root, repo, oursDir); err != nil { cleanWorktrees() - return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) + return abortMerge(root, fmt.Errorf("copy the engine's merge result: %w", err)) } if err := repairOrRefuse(root, abs, out); err != nil { cleanWorktrees() return err } + // Stage the whole model surface, not just the conflicted paths: a unit + // the engine's result dropped was never conflicted, and its deletion + // must be staged too. + if _, err := gitOutIn(root, nil, "add", "-A", "--", + filepath.FromSlash(repo.MPRPath), filepath.FromSlash(repo.Contents)); err != nil { + cleanWorktrees() + return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) + } if err := commitTheMerge(root, repo, nil, message, branch, push, out); err != nil { cleanWorktrees() return err @@ -348,6 +372,67 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return nil } +// copyEngineResult copies the engine's merged model files — the .mpr and +// every .mxunit under mprcontents — from the MINE worktree into the working +// tree, removing working-tree units the result no longer has (a deletion the +// engine resolved). Non-model files are untouched, so a human's staged text +// resolutions survive as they are. +func copyEngineResult(root string, repo *gitNoteRepo, mineDir string) error { + srcContents := filepath.Join(mineDir, filepath.FromSlash(repo.Contents)) + dstContents := filepath.Join(root, filepath.FromSlash(repo.Contents)) + if err := copyFile( + filepath.Join(mineDir, filepath.FromSlash(repo.MPRPath)), + filepath.Join(root, filepath.FromSlash(repo.MPRPath)), + ); err != nil { + return err + } + present := map[string]bool{} + err := filepath.WalkDir(srcContents, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(d.Name(), ".mxunit") { + return nil + } + rel, err := filepath.Rel(srcContents, p) + if err != nil { + return err + } + present[filepath.ToSlash(rel)] = true + return copyFile(p, filepath.Join(dstContents, rel)) + }) + if err != nil { + return err + } + return filepath.WalkDir(dstContents, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(d.Name(), ".mxunit") { + return nil + } + rel, err := filepath.Rel(dstContents, p) + if err != nil { + return err + } + if !present[filepath.ToSlash(rel)] { + return os.Remove(p) + } + return nil + }) +} + +func copyFile(src, dst string) error { + raw, err := os.ReadFile(src) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return err + } + return os.WriteFile(dst, raw, 0o644) +} + // abortMerge backs a failed resolution out entirely — working tree, index and // MERGE_HEAD return to the pre-merge state — and returns the original error. // Every refusal on this command ends here: a half-done merge makes Studio Pro @@ -450,18 +535,16 @@ func projectMendixVersion(mprPath string) string { return v } -// realMxMerge shells out to `mx merge BASE MINE THEIRS` over the three -// project folders, mx resolved for the project's Mendix version. -func realMxMerge(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error) { - version := "" - if matches, _ := filepath.Glob(filepath.Join(mineDir, "*.mpr")); len(matches) > 0 { - version = projectMendixVersion(matches[0]) - } +// realMxMerge shells out to `mx merge -v BASE MINE THEIRS` over the three +// revisions' project files, mx resolved for the project's Mendix version; +// -v makes the engine print its per-unit conflict list. +func realMxMerge(baseMpr, mineMpr, theirsMpr string, shas [3]string) (int, string, error) { + version := projectMendixVersion(mineMpr) mxPath, err := docker.ResolveMxForVersion("", version) if err != nil { return 1, "", fmt.Errorf("no mx for Mendix %s: %w (model conflicts need it)", version, err) } - cmd := exec.Command(mxPath, "merge", baseDir, mineDir, theirsDir) + cmd := exec.Command(mxPath, "merge", "-v", baseMpr, mineMpr, theirsMpr) docker.PrepareMxCommand(cmd) b, err := cmd.CombinedOutput() code := 0 diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 677cc63ef..4fa10ec95 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -7,6 +7,7 @@ import ( "encoding/json" "errors" "fmt" + "io/fs" "os" "path/filepath" "strings" @@ -85,15 +86,13 @@ func addConflictRepo(t *testing.T) (string, string) { t.Helper() dir, mprPath := doctorNoteRepo(t) ours := baseUnits(t) - oursAdd := "11111111-0000-0000-0000-000000000009" - ours[oursAdd] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Ours")} + ours[addOursID] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Ours")} writeProject(t, dir, "11.13.0", ours) gitIn(t, dir, "add", "-A") gitIn(t, dir, "commit", "-q", "-m", "ours adds a microflow") gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") theirs := baseUnits(t) - theirsAdd := "11111111-0000-0000-0000-00000000000a" - theirs[theirsAdd] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Theirs")} + theirs[addTheirsID] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Theirs")} writeProject(t, dir, "11.13.0", theirs) gitIn(t, dir, "add", "-A") gitIn(t, dir, "commit", "-q", "-m", "theirs adds a microflow") @@ -118,6 +117,12 @@ var ( sideKey = bsonKeyValue("DocB", "side") ) +// The units each side adds in addConflictRepo. +const ( + addOursID = "11111111-0000-0000-0000-000000000009" + addTheirsID = "11111111-0000-0000-0000-00000000000a" +) + func okRunner(out string) mxMergeRunner { return func(base, mine, theirs string, shas [3]string) (int, string, error) { return 0, out, nil @@ -157,9 +162,47 @@ func TestGitMerge_CleanMergeTextFileWritesNote(t *testing.T) { } } +// fakeEngineResult simulates the engine having written its merge result into +// the MINE project folder: it overwrites dstDir's project with srcDir's +// (the .mpr and all of mprcontents, extras in dst removed). +func fakeEngineResult(t *testing.T, srcDir, dstDir string) { + t.Helper() + raw, err := os.ReadFile(filepath.Join(srcDir, "App.mpr")) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dstDir, "App.mpr"), raw, 0o644); err != nil { + t.Fatal(err) + } + if err := os.RemoveAll(filepath.Join(dstDir, "mprcontents")); err != nil { + t.Fatal(err) + } + srcContents := filepath.Join(srcDir, "mprcontents") + _ = filepath.WalkDir(srcContents, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + rel, err := filepath.Rel(srcContents, p) + if err != nil { + return err + } + dst := filepath.Join(dstDir, "mprcontents", rel) + if d.IsDir() { + return os.MkdirAll(dst, 0o755) + } + raw, err := os.ReadFile(p) + if err != nil { + return err + } + return os.WriteFile(dst, raw, 0o644) + }) +} + func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { dir, mprPath := modelConflictRepo(t) doctorEnv(t, dir) + oursUnits := baseUnits(t) + oursUnits[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} var gotBase, gotMine, gotTheirs string var gotShas [3]string @@ -167,25 +210,30 @@ func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas // The worktrees are removed when the merge finishes; check inside. - _, baseHasMprErr := os.Stat(filepath.Join(base, "App.mpr")) - _, theirsHasMprErr := os.Stat(filepath.Join(theirs, "App.mpr")) + _, baseHasMprErr := os.Stat(base) + _, theirsHasMprErr := os.Stat(theirs) baseHasMpr, theirsHasMpr = baseHasMprErr == nil, theirsHasMprErr == nil + // The engine writes its result into MINE. + writeProject(t, filepath.Dir(mine), "11.13.0", oursUnits) return 0, "Merge successful.", nil }} var out bytes.Buffer if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } - // Folder mode: the three args are project folders, and MINE is the - // working tree — the engine writes its result there. - if gotMine != dir { - t.Errorf("runner's mine = %q, want the working project folder %q", gotMine, dir) + // File form: the three args are the revisions' .mpr paths, and MINE is a + // clean worktree at HEAD — never the working tree mid-merge. + if filepath.Base(gotMine) != "App.mpr" { + t.Errorf("runner's mine = %q, want the project's .mpr path", gotMine) + } + if filepath.Dir(gotMine) == dir { + t.Errorf("runner's mine = the working tree %q; MINE must be a clean worktree", dir) } if !baseHasMpr { - t.Errorf("runner's base folder %q must hold the project's .mpr", gotBase) + t.Errorf("runner's base %q must hold the project's .mpr", gotBase) } if !theirsHasMpr { - t.Errorf("runner's theirs folder %q must hold the project's .mpr", gotTheirs) + t.Errorf("runner's theirs %q must hold the project's .mpr", gotTheirs) } if gotShas[0] == "" || gotShas[1] == gotShas[0] || gotShas[2] == gotShas[1] { t.Errorf("shas must be base/ours/theirs distinct: %v", gotShas) @@ -201,6 +249,80 @@ func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { if !strings.Contains(noteOn(t, dir, "HEAD"), `"HasModelerVersion":true`) { t.Errorf("merge commit has no note:\n%s", noteOn(t, dir, "HEAD")) } + // The hash gate repairs the .mpr; staging must come after it, or the + // commit would carry the pre-repair bytes and the tree would stay dirty. + if st := gitOutT(t, dir, "status", "--porcelain"); strings.TrimSpace(st) != "" { + t.Errorf("working tree not clean after the merge:\n%s", st) + } +} + +// The engine owns the .mpr, so its result carries both sides' document sets: +// a unit each side added must all be on disk and in the index after the merge. +func TestGitMerge_KeepsUnitsBothSidesAdded(t *testing.T) { + dir, mprPath := addConflictRepo(t) + doctorEnv(t, dir) + merged := baseUnits(t) + merged[addOursID] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Ours")} + merged[addTheirsID] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Theirs")} + + runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + writeProject(t, filepath.Dir(mine), "11.13.0", merged) + return 0, "Merge successful.", nil + }} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + for _, id := range []string{addOursID, addTheirsID} { + if _, err := os.Stat(filepath.Join(dir, unitRel(id))); err != nil { + t.Errorf("unit %s must exist after the merge: %v", id, err) + } + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + if err := fixHashes(mprPath, false, &out); err != nil { + t.Errorf("hashes drift after the merge: %v\n%s", err, out.String()) + } + if st := gitOutT(t, dir, "status", "--porcelain"); strings.TrimSpace(st) != "" { + t.Errorf("working tree not clean after the merge:\n%s", st) + } +} + +// A deletion the engine resolved travels with its result: a unit file the +// result no longer has is removed from the working tree, not left behind as +// an orphan the hash gate would refuse. +func TestGitMerge_CopyBackRemovesUnitsTheResultDrops(t *testing.T) { + dir, mprPath := addConflictRepo(t) + doctorEnv(t, dir) + + runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + // result = theirs: theirs' project is the merged outcome, without + // ours' added unit. + fakeEngineResult(t, filepath.Dir(theirs), filepath.Dir(mine)) + return 0, "Merge successful.", nil + }} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + if _, err := os.Stat(filepath.Join(dir, unitRel(addOursID))); !os.IsNotExist(err) { + t.Errorf("ours' unit must be gone with the engine's result: %v", err) + } + if _, err := os.Stat(filepath.Join(dir, unitRel(addTheirsID))); err != nil { + t.Errorf("theirs' unit must be on disk: %v", err) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + if err := fixHashes(mprPath, false, &out); err != nil { + t.Errorf("hashes drift after the merge: %v\n%s", err, out.String()) + } + if st := gitOutT(t, dir, "status", "--porcelain"); strings.TrimSpace(st) != "" { + t.Errorf("working tree not clean after the merge:\n%s", st) + } } func TestGitMerge_ModelConflictUnresolvedAborts(t *testing.T) { From 71084098f9012ac6443d29844f937812356ed582 Mon Sep 17 00:00:00 2001 From: engalar Date: Sun, 1 Nov 2026 14:27:07 +0800 Subject: [PATCH 47/61] feat(git-merge): drive the review material from the engine's conflict list --- cmd/mxcli/cmd_git_merge.go | 18 +++++++++++++----- cmd/mxcli/gitmerge_test.go | 25 +++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 5 deletions(-) diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index 3966d460c..b0fe8c84f 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -319,7 +319,15 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m if code == 2 { // The review step: what each side changed, at property level (mx // diff), plus a per-unit plan template. Then leave nothing behind — - // the human decides. + // the human decides. The unit list follows the engine's own conflict + // list — git's can differ (a stale one-side index makes the engine + // content-merge a unit git sees as changed). Without a parsable + // list, fall back to git's conflicted units: err toward showing work + // rather than hiding it. + manual := engineManualIDs(parseEngineConflicts(mxOut)) + if len(manual) == 0 { + manual = conflictedUnitIDs(model) + } oursMpr := filepath.Join(oursDir, filepath.FromSlash(repo.MPRPath)) if runners.diff != nil { oursJSON := filepath.Join(tmp, "ours-diff.json") @@ -327,10 +335,10 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m d1 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), oursMpr, oursJSON) d2 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), theirsJSON) if d1 == nil && d2 == nil { - fmt.Fprintln(out, propertyReport(oursJSON, theirsJSON, conflictedUnitIDs(model))) + fmt.Fprintln(out, propertyReport(oursJSON, theirsJSON, manual)) } } - if err := writePlanTemplate(mprPath, baseSha, oursSha, theirsSha, model); err != nil { + if err := writePlanTemplate(mprPath, baseSha, oursSha, theirsSha, manual); err != nil { fmt.Fprintf(out, " (could not write the plan template: %v)\n", err) } else { fmt.Fprintf(out, "Plan template: %s — set \"take\" per unit, re-run with --plan.\n", filepath.Base(mprPath)+".merge-plan.json") @@ -720,9 +728,9 @@ type planUnit struct { Take string `json:"take"` } -func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, model []string) error { +func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, unitIDs []string) error { tmpl := mergePlanTemplate{Base: baseSha, Ours: oursSha, Theirs: theirsSha} - for _, id := range conflictedUnitIDs(model) { + for _, id := range unitIDs { tmpl.Units = append(tmpl.Units, planUnit{ID: id, Take: "ours"}) } b, err := json.MarshalIndent(tmpl, "", " ") diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 4fa10ec95..f023b497b 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -776,3 +776,28 @@ func TestEngineManualIDs_FiltersAutoResolvedAndEmpty(t *testing.T) { t.Errorf("manual ids: %v", ids) } } + +// The review material follows the engine's conflict list: a unit the engine +// auto-resolved (git still lists it — one side's stale ContentsHash index +// content-merges cleanly) never lands in the plan template, and one only the +// engine flags as manual does. The engine's list here names tMf manual and +// tDM auto — the opposite of git's conflicted-unit list — so the test can +// tell which one the plan was built from. +func TestGitMerge_PlanTemplateFollowsEngineConflictList(t *testing.T) { + dir, mprPath := modelConflictRepo(t) // git's conflicted unit: tDM + doctorEnv(t, dir) + auto := `[{"UnitId":"` + tDM + `","Type":"MergeContents","ConflictResolutionDetails":"AutoResolved"},{"UnitId":"` + tMf + `","Type":"MergeContents","ConflictResolutionDetails":"ManualResolutionRequired"}]` + mx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, auto, nil + }} + var out bytes.Buffer + _ = mergeRun(mprPath, "side", "", "", "", false, mx, &out) + raw, err := os.ReadFile(mprPath + ".merge-plan.json") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(raw), tMf) || strings.Contains(string(raw), tDM) { + t.Errorf("plan must hold only the engine's manual unit:\n%s", raw) + } + gitInE(t, dir, "merge", "--abort") +} From e62ce46665b097fab090755a9681b0ed507df997 Mon Sep 17 00:00:00 2001 From: engalar Date: Sun, 1 Nov 2026 14:46:05 +0800 Subject: [PATCH 48/61] test(git-merge): real-mx integration for the engine path (exit 0 and 2) --- cmd/mxcli/gitmerge_integration_test.go | 227 +++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 cmd/mxcli/gitmerge_integration_test.go diff --git a/cmd/mxcli/gitmerge_integration_test.go b/cmd/mxcli/gitmerge_integration_test.go new file mode 100644 index 000000000..82e5244c9 --- /dev/null +++ b/cmd/mxcli/gitmerge_integration_test.go @@ -0,0 +1,227 @@ +//go:build integration + +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// The engine path against a REAL mx: `mx merge` over three .mpr paths with +// MINE as a clean worktree, the result copied back — the part the fake-runner +// unit tests cannot see. One scaffold, both engine verdicts: +// +// TestGitMergeIntegration_EngineAutoResolvesAddAdd exit 0: two sides each +// add a page; both kept. +// TestGitMergeIntegration_PropertyConflictNeedsDecisionAborts +// exit 2: both sides +// change the same page's +// title; clean abort. +// +// Requires a resolvable mx for the scaffolded project's version (provided by +// the CI integration job's `mxcli setup mxbuild`); skips otherwise. +// +// MXCLI_BIN=/path/to/mxcli go test -tags integration ./cmd/mxcli/ -run TestGitMergeIntegration -v +func TestGitMergeIntegration_EngineAutoResolvesAddAdd(t *testing.T) { + dir, mpr, bin := mergeIntegrationRepo(t) + + applyPage(t, bin, dir, mpr, "PageMain", "Main page") + commitAll(t, dir, "main adds PageMain") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + applyPage(t, bin, dir, mpr, "PageSide", "Side page") + commitAll(t, dir, "side adds PageSide") + gitIn(t, dir, "switch", "-q", "main") + + var out bytes.Buffer + runners := mxRunners{merge: realMxMerge, diff: realMxDiff} + if err := mergeRun(mpr, "side", "", "", "", false, runners, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + if st := gitOutT(t, dir, "status", "--porcelain"); strings.TrimSpace(st) != "" { + t.Errorf("working tree not clean after the merge:\n%s", st) + } + var out2 bytes.Buffer + if err := fixHashes(mpr, false, &out2); err != nil { + t.Errorf("the merged .mpr index is inconsistent: %v\n%s", err, out2.String()) + } + for _, name := range []string{"PageMain", "PageSide"} { + if unitFilesContaining(t, dir, name) == 0 { + t.Errorf("no unit file carries %s: the engine's result lost a side", name) + } + } +} + +func TestGitMergeIntegration_PropertyConflictNeedsDecisionAborts(t *testing.T) { + dir, mpr, bin := mergeIntegrationRepo(t) + + // The shared page exists in the base, so both sides MODIFY the same unit: + // side branches from the commit that already has the page. + applyPage(t, bin, dir, mpr, "Shared", "Base title") + commitAll(t, dir, "base adds Shared") + gitIn(t, dir, "switch", "-q", "-c", "side") + applyPage(t, bin, dir, mpr, "Shared", "Side title") + commitAll(t, dir, "side retitles Shared") + gitIn(t, dir, "switch", "-q", "main") + applyPage(t, bin, dir, mpr, "Shared", "Main title") + commitAll(t, dir, "main retitles Shared") + + before := gitOutT(t, dir, "rev-parse", "HEAD") + var out bytes.Buffer + runners := mxRunners{merge: realMxMerge, diff: realMxDiff} + err := mergeRun(mpr, "side", "", "", "", false, runners, &out) + if err == nil || !strings.Contains(err.Error(), "Studio Pro") { + t.Fatalf("a property conflict must end in the Studio Pro referral, got %v\n%s", err, out.String()) + } + if got := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")); got != strings.TrimSpace(before) { + t.Errorf("HEAD moved despite the abort: %s", got) + } + if _, mhErr := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// mergeIntegrationRepo scaffolds a real Mendix project (`mx create-project`) +// in a hermetic git repository and returns its dir, .mpr path and a built +// mxcli binary for applying MDL. The project and the binary live in SHORT +// temp dirs: the test names make t.TempDir() paths long enough for +// create-project's template extraction to hit PathTooLongException. +func mergeIntegrationRepo(t *testing.T) (string, string, string) { + t.Helper() + dir, err := os.MkdirTemp("", "mxmrg") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.RemoveAll(dir) }) + t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) + t.Setenv("GIT_CONFIG_NOSYSTEM", "1") + t.Setenv("GIT_CEILING_DIRECTORIES", filepath.Dir(dir)) + + bin := os.Getenv("MXCLI_BIN") + if bin == "" { + binDir, err := os.MkdirTemp("", "mxbin") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.RemoveAll(binDir) }) + bin = filepath.Join(binDir, "mxcli.exe") + build := exec.Command("go", "build", "-o", bin, ".") + build.Dir = "." // the package dir: cmd/mxcli + if out, err := build.CombinedOutput(); err != nil { + t.Fatalf("go build: %v\n%s", err, out) + } + } + + scaffoldProject(t, dir) + mpr := filepath.Join(dir, "App.mpr") + if _, err := docker.ResolveMxForVersion("", projectMendixVersion(mpr)); err != nil { + t.Skipf("no mx for the scaffolded project: %v", err) + } + // mxcli's own unit index is working material, not project content: it is + // rewritten on every exec and would conflict on both sides. + if err := os.WriteFile(filepath.Join(dir, ".gitignore"), []byte(".mxcli/\n"), 0o644); err != nil { + t.Fatal(err) + } + + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "core.autocrlf", "false") + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "scaffold") + return dir, mpr, bin +} + +// scaffoldProject runs `mx create-project` into dir like the docker suite +// does; its failure skips the test (no mx, no fixture). +func scaffoldProject(t *testing.T, dir string) { + t.Helper() + mxPath, err := docker.ResolveMxForVersion("", "") + if err != nil { + t.Skipf("no mx at all: %v", err) + } + create := exec.Command(mxPath, "create-project") + create.Dir = dir + docker.PrepareMxCommand(create) + if out, err := create.CombinedOutput(); err != nil { + t.Skipf("mx create-project: %v\n%s", err, out) + } + if _, err := os.Stat(filepath.Join(dir, "App.mpr")); err != nil { + t.Skipf("mx create-project did not produce App.mpr: %v", err) + } +} + +// applyPage writes and applies one page change through the real mxcli exec +// path (in-process model write, --no-check skips the semantic preflight). +// The script lives outside the repository: it is test scaffolding, not +// project content, and committing it would conflict on both sides. +func applyPage(t *testing.T, bin, dir, mpr, name, title string) { + t.Helper() + f, err := os.CreateTemp("", "mxpage") + if err != nil { + t.Fatal(err) + } + script := f.Name() + t.Cleanup(func() { os.Remove(script) }) + mdl := fmt.Sprintf("mdl 1;\ncreate or modify page MyFirstModule.%s (title: '%s', layout: Atlas_Core.Atlas_Default) {\n dynamictext t1 (content: '%s')\n};\n", name, title, title) + if _, err := f.WriteString(mdl); err != nil { + t.Fatal(err) + } + if err := f.Close(); err != nil { + t.Fatal(err) + } + cmd := exec.Command(bin, "exec", "-p", mpr, "--no-check", script) + cmd.Dir = dir + var stderr bytes.Buffer + cmd.Stderr = &stderr + if err := cmd.Run(); err != nil { + t.Fatalf("apply %s: %v\n%s", name, err, stderr.String()) + } +} + +func commitAll(t *testing.T, dir, msg string) { + t.Helper() + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", msg) +} + +// unitFilesContaining counts the .mxunit files whose bytes carry needle +// (page names are ASCII in a unit's BSON). +func unitFilesContaining(t *testing.T, dir, needle string) int { + t.Helper() + n := 0 + err := filepath.WalkDir(filepath.Join(dir, "mprcontents"), func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + return nil + } + raw, err := os.ReadFile(p) + if err != nil { + return err + } + if bytes.Contains(raw, []byte(needle)) { + n++ + } + return nil + }) + if err != nil { + t.Fatal(err) + } + return n +} From b499c076a4d4c16cc536063bb010478076adeb8d Mon Sep 17 00:00:00 2001 From: engalar Date: Sun, 1 Nov 2026 14:55:20 +0800 Subject: [PATCH 49/61] docs(git-merge): the engine is called with .mpr paths over clean worktrees --- .claude/skills/mendix/teamserver-git/SKILL.md | 6 ++--- CHANGELOG.md | 2 +- docs-site/src/tools/outside-studio-pro.md | 25 ++++++++++--------- 3 files changed, 17 insertions(+), 16 deletions(-) diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index a547ad008..60b9e47c1 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -121,9 +121,9 @@ git push origin refs/notes/mx_metadata is a binary conflict), and a textual merge of `mprcontents/` leaves an index that does not match the files. Model-aware merging exists in two forms: **`mxcli git merge `** from the command line (it hands the - conflict to `mx merge` over three project folders — Studio Pro's own - engine, Linux devcontainers included — and commits the resolved model as a - normal merge commit with the note), and **Version + conflict to `mx merge` over the three revisions' project files — Studio + Pro's own engine, Linux devcontainers included — and commits the resolved + model as a normal merge commit with the note), and **Version Control › Merge Changes Here** in Studio Pro (*Merge feature branch* or *Cherry Pick*) — which is also where the conflicts mx cannot auto-resolve go. A conflict `mxcli git merge` cannot resolve — and an engine failure — diff --git a/CHANGELOG.md b/CHANGELOG.md index ccf12d5a8..9813dffa6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx merge BASE MINE THEIRS` over three project folders, which runs on Linux devcontainers too (the older `mx git-merge` file form is Windows-bound). `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`; a merge stopped on text conflicts resumes once those are resolved and staged), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three project folders to the engine, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything the engine auto-resolves is staged, hash-checked and committed as the normal two-parent merge commit with the mx_metadata note — and the engine owns the `.mpr`, so a unit one side added while the other changed existing ones is kept, not dropped; a conflict the engine reports as `ManualResolutionRequired` (exit 2) is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit at property level (from `mx diff`), with a plan template (`.merge-plan.json`, never staged by `mxcli git commit`) beside the project — the human re-runs with `--plan ` to resolve **per unit** (each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files; modified-on-both units only, added/deleted/moved goes to Studio Pro), or with `--ours`/`--theirs` to take one side of every conflicted unit — hash-gated: when the sides' document sets differ (one side added or deleted a unit) the commit is refused and the merge aborted, because a whole side would orphan the other side's documents — or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; any other engine exit (its 129, an mx that will not start) is reported as the engine failing and the merge is aborted cleanly, never steered into a decision. Non-model conflicts (text sources) are left in place like plain git: resolve them, `git add` them, re-run the command. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx merge BASE MINE THEIRS` over the three revisions' project files, which runs on Linux devcontainers too. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`; a merge stopped on text conflicts resumes once those are resolved and staged), and on a model-file conflict checks the base, ours (a clean checkout of HEAD — never the working tree mid-merge, where the engine chokes) and theirs revisions out into temporary worktrees and hands the three projects' `.mpr` files to the engine, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything the engine auto-resolves is copied back from MINE, hash-checked, staged and committed as the normal two-parent merge commit with the mx_metadata note — and the engine owns the `.mpr`, so a unit one side added while the other changed existing ones is kept, not dropped, and a deletion it resolved removes the unit file with it; the engine's own `-v` conflict list (one JSON array per run, `ManualResolutionRequired` vs `AutoResolved`) decides which units the review material and plan template cover, with git's conflicted-file list as the fallback; a conflict the engine reports as `ManualResolutionRequired` (exit 2) is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit at property level (from `mx diff`), with a plan template (`.merge-plan.json`, never staged by `mxcli git commit`) beside the project — the human re-runs with `--plan ` to resolve **per unit** (each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files; modified-on-both units only, added/deleted/moved goes to Studio Pro), or with `--ours`/`--theirs` to take one side of every conflicted unit — hash-gated: when the sides' document sets differ (one side added or deleted a unit) the commit is refused and the merge aborted, because a whole side would orphan the other side's documents — or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; any other engine exit (its 129, an mx that will not start) is reported as the engine failing and the merge is aborted cleanly, never steered into a decision. Non-model conflicts (text sources) are left in place like plain git: resolve them, `git add` them, re-run the command. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Changed diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 193301bed..77317d526 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -239,9 +239,8 @@ the index, so whatever you had staged is left exactly as it was. `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` — there is nothing for git's textual merge to work on. Studio Pro's *Merge Changes Here* resolves it with a model-aware merge, and -that engine ships as a command: `mx merge BASE MINE THEIRS` over three -project folders (it runs on Linux devcontainers too; the older `mx git-merge` -file form is Windows-bound). +that engine ships as a command: `mx merge BASE MINE THEIRS` over the three +revisions' project files (it runs on Linux devcontainers too). `mxcli git merge ` puts the two together: @@ -252,15 +251,17 @@ mxcli git merge origin/main -p app.mpr --push `git merge ` runs first; a merge git resolves itself (disjoint files, a fast-forward) is annotated with the mx_metadata note like `mxcli git -commit` does. On a model-file conflict the base and theirs revisions are -checked out into temporary worktrees and the three project folders go to -`mx merge` (resolved for the project's Mendix version); the engine writes -its result into the working tree and owns the `.mpr`: - -- what the engine auto-resolves is staged, hash-checked and committed as the - normal two-parent merge commit, with the note attached. The engine's - `.mpr` carries both sides' document sets, so a unit one side added while - the other changed existing ones is kept, not dropped; +commit` does. On a model-file conflict the base, ours (a clean checkout of +HEAD) and theirs revisions are checked out into temporary worktrees and the +three projects' .mpr files go to `mx merge` (resolved for the project's +Mendix version; MINE is never the working tree mid-merge — the engine +chokes on the conflicted state); the engine writes its result into MINE and +it is copied back, so the engine owns the `.mpr`: + +- what the engine auto-resolves is copied back, hash-checked, staged and + committed as the normal two-parent merge commit, with the note attached. + The engine's `.mpr` carries both sides' document sets, so a unit one side + added while the other changed existing ones is kept, not dropped; - a conflict the engine reports as `ManualResolutionRequired` (exit 2) is never merged silently: the merge is aborted, and **both sides' changes are printed for review** — at property level, from `mx diff` ("New type > From 2c118daf9ce67785fba0fe55ed201de9940da496 Mon Sep 17 00:00:00 2001 From: engalar Date: Sun, 1 Nov 2026 16:13:54 +0800 Subject: [PATCH 50/61] test(git-merge): pin the modified-on-both plan gate --- cmd/mxcli/gitmerge_test.go | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index f023b497b..e67ba1a13 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -777,6 +777,37 @@ func TestEngineManualIDs_FiltersAutoResolvedAndEmpty(t *testing.T) { } } +// A plan unit that is not modified on both sides is refused outright — the +// gate that also makes the engine's list and git's index unable to disagree +// dangerously: a unit git resolved on its own (one-side or no change, even +// behind a stale ContentsHash index) never passes "modified on both", so it +// can never be silently skipped to ours by the theirs loop. +func TestGitMerge_PlanRefusesUnitsNotModifiedOnBoth(t *testing.T) { + dir, mprPath := modelConflictRepo(t) // git conflicts tDM; tMf untouched by both + doctorEnv(t, dir) + base := strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")) + ours := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) + theirs := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")) + plan := fmt.Sprintf(`{"base":%q,"ours":%q,"theirs":%q,"units":[{"id":%q,"take":"theirs"}]}`, + base, ours, theirs, tMf) + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + noMx := mxRunners{} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "only modified-on-both conflicts can be planned") { + t.Fatalf("want the modified-on-both refusal, got %v\n%s", err, out.String()) + } + if _, mhErr := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr == nil { + t.Error("MERGE_HEAD still present: the refusal must abort the merge") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + // The review material follows the engine's conflict list: a unit the engine // auto-resolved (git still lists it — one side's stale ContentsHash index // content-merges cleanly) never lands in the plan template, and one only the From 41698502e89bf60688fd95f2afa63f990424fa69 Mon Sep 17 00:00:00 2001 From: Ako Date: Sun, 1 Nov 2026 15:52:21 +0000 Subject: [PATCH 51/61] feat(mcp): drop Concord; delete, check and run through Studio Pro's MCP Concord, the optional second MCP server, is no longer needed: Studio Pro's own server covers what it supplied. - DROP calls ped_delete_document (11.16+) and no longer falls back to Concord; a server without it gets an error naming the version. - --mcp-check runs a project-wide ped_check_errors (11.15+, all pages). - --mcp-run starts the app through execute_handler mendix_app (11.16+), gated on tool presence and a query_handlers getDetails lookup. - --mcp-save goes: PED persists every write to disk as it lands. - --mcp-concord / --mcp-concord-dial, ConcordConnected and concord.go are removed; the capability report names the three paths instead. - Includes the native-delete work: Client.HasTool, and CallTool classifying "Error executing tool handler: ..." (isError:false) as an error. Docs, the live-edit skill, the feature matrix and PED_MCP_CAPABILITIES.md are updated; MCP_SAVE_VERIFICATION.md (a --mcp-save procedure) is removed. Co-Authored-By: Claude Opus 5.5 --- ...-mcp-drop-ignores-ped-delete-document.json | 1 + .../mendix/live-edit-with-studio-pro/SKILL.md | 36 ++-- CHANGELOG.md | 2 + cmd/mxcli/engine.go | 12 +- cmd/mxcli/main.go | 25 +-- cmd/mxcli/mcp.go | 3 - docs-site/src/tools/mcp-connect.md | 35 ++- docs-wiki/architecture/mcp-backend.md | 8 +- docs/01-project/MDL_FEATURE_MATRIX.md | 68 +++--- docs/03-development/MCP_SAVE_VERIFICATION.md | 89 -------- docs/03-development/PED_MCP_CAPABILITIES.md | 183 +++------------- mdl/backend/mcp/app.go | 159 ++++++++++++++ mdl/backend/mcp/app_test.go | 202 ++++++++++++++++++ mdl/backend/mcp/backend.go | 83 +++---- mdl/backend/mcp/businessevent.go | 10 +- mdl/backend/mcp/capabilities.go | 33 +-- mdl/backend/mcp/capabilities.yaml | 2 +- mdl/backend/mcp/check_errors.go | 63 +++++- mdl/backend/mcp/client.go | 26 ++- mdl/backend/mcp/concord.go | 178 --------------- mdl/backend/mcp/concord_test.go | 62 ------ mdl/backend/mcp/constant.go | 6 +- mdl/backend/mcp/delete.go | 84 ++++++++ mdl/backend/mcp/delete_test.go | 95 ++++++++ mdl/backend/mcp/enumeration.go | 6 +- mdl/backend/mcp/enumeration_test.go | 10 +- mdl/backend/mcp/handler_error_test.go | 20 ++ mdl/backend/mcp/microflow.go | 6 +- mdl/backend/mcp/nanoflow.go | 8 +- mdl/backend/mcp/page.go | 6 +- mdl/backend/mcp/workflow.go | 6 +- 31 files changed, 828 insertions(+), 699 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-backend/2026-11-01-mcp-drop-ignores-ped-delete-document.json delete mode 100644 docs/03-development/MCP_SAVE_VERIFICATION.md create mode 100644 mdl/backend/mcp/app.go create mode 100644 mdl/backend/mcp/app_test.go delete mode 100644 mdl/backend/mcp/concord.go delete mode 100644 mdl/backend/mcp/concord_test.go create mode 100644 mdl/backend/mcp/delete.go create mode 100644 mdl/backend/mcp/delete_test.go create mode 100644 mdl/backend/mcp/handler_error_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend/2026-11-01-mcp-drop-ignores-ped-delete-document.json b/.claude/skills/fix-issue/findings/mdl-backend/2026-11-01-mcp-drop-ignores-ped-delete-document.json new file mode 100644 index 000000000..8bfa6aab9 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-backend/2026-11-01-mcp-drop-ignores-ped-delete-document.json @@ -0,0 +1 @@ +{"area": "mdl/backend/mcp", "date": "2026-11-01", "symptom": "Over `--mcp` on Studio Pro 11.16, every DROP of a standalone document (constant, enumeration, microflow, nanoflow, page, workflow, business-event service) demanded `--mcp-concord`, though 11.16 ships `ped_delete_document`. Separately, a PED tool whose handler throws answers `Error executing tool handler: …` with `isError:false`, which any caller trusting the flag reads as success", "cause": "`concordDeleteDocument` was the only DROP path, written when PED had no delete tool. `Client.CallTool` passed `isError` through verbatim, and only the create/update callers (`pedOpError`'s `SUCCESS` prefix) noticed failures reported in the text", "file": "`mdl/backend/mcp/delete.go` (new `deleteDocument`: native when `HasTool(\"ped_delete_document\")`, else an error naming 11.16+; Concord support removed in the same change), `client.go` (`HasTool`, `handlerFailed` sets `IsError`), the seven `Delete*` call sites, `capabilities.yaml`/`.go` report text", "insight": "**Probe tool presence, never the version**: `serverInfo.version` was 1.0.0 on 11.12.0 and 11.16.0 reports itself, but the delete tool is the fact that matters. **On 11.16.0 the native delete fails for every caller outside Studio Pro's own agent session** (`Session 'mcp-server' not found`, measured from a devcontainer through four clientInfo names), so the success reply has never been seen. The code therefore settles an unrecognised reply by `ped_find_document` instead of guessing the text. **Revert check**: stubbing `handlerFailed` fails `TestCallTool_HandlerErrorIsAnError`, and removing the `HasTool` gate fails `TestDeleteDocument_NoNativeToolNeeds1116`. Live runs are in mx-test-projects/runs/native-delete/"} diff --git a/.claude/skills/mendix/live-edit-with-studio-pro/SKILL.md b/.claude/skills/mendix/live-edit-with-studio-pro/SKILL.md index 47fcff31d..d96bdd6ff 100644 --- a/.claude/skills/mendix/live-edit-with-studio-pro/SKILL.md +++ b/.claude/skills/mendix/live-edit-with-studio-pro/SKILL.md @@ -14,9 +14,9 @@ description: "Change the model through mxcli while Studio Pro has the project op | 36-42 | How it works (hybrid: local reads, live writes) | | 43-61 | Connect | | 62-91 | What you can change via MCP — check first | -| 92-104 | Two MCP servers — use the built-in one by default | -| 105-114 | Caveats | -| 115-122 | Verify your change | +| 92-99 | The MCP server | +| 100-112 | Caveats | +| 113-120 | Verify your change | # Live-Editing an Open Studio Pro Project with mxcli (MCP) @@ -89,26 +89,24 @@ New modules and their dependents resolve within the same run, so document in a folder at create time (`create microflow X.MF folder 'A/B'`), since MOVE can't re-parent over MCP. -## Two MCP servers — use the built-in one by default +## The MCP server -The machine may run two MCP servers: - -- **Studio Pro built-in (port 7782 by default)** — model authoring. **Use this by - default.** From **11.13** Studio Pro **auto-selects a free port** when 7782 is - taken, so multiple instances can run side by side. The active port is shown in - Studio Pro's **status bar** (and set under Preferences > AI > MCP Server) — read - it there rather than assuming 7782, and pass it to `--mcp-dial`. -- **Concord (port 7783)** — a temporary gap-filler with operational/refactor tools - (`delete_document`, `save_all`, `run_app`, `check_model`). **Only** reach for - Concord when the built-in server lacks the capability you need. +Studio Pro's built-in MCP server (port 7782 by default) is the only one mxcli +uses. From **11.13** Studio Pro **auto-selects a free port** when 7782 is taken, so +multiple instances can run side by side. The active port is shown in Studio Pro's +**status bar** (and set under Preferences > AI > MCP Server) — read it there rather +than assuming 7782, and pass it to `--mcp-dial`. ## Caveats -- **Writes are unsaved.** They land in Studio Pro's in-memory model (shown as - unsaved). Save in Studio Pro to persist to disk. -- **No DROP of standalone documents** (enumeration / microflow / page) through the - built-in server — it has no delete tool. Remove them in Studio Pro (or via - Concord's `delete_document`). +- **Writes are saved as they land.** PED writes each change to disk immediately + (measured on 11.12.0 and 11.16.0), so there is no save step. +- **DROP of standalone documents needs Studio Pro 11.16+** (`ped_delete_document`). + On 11.16.0 that tool fails with `Session 'mcp-server' not found` for callers + outside Studio Pro's own agent session; delete the document in Studio Pro then. +- **`--mcp-check`** runs Studio Pro's project-wide error check (11.15+), and + **`--mcp-run`** starts the app through the `mendix_app` handler and prints its URL + (11.16+). - **`-p` must match the open project.** A mismatched `-p` silently reads the wrong model. diff --git a/CHANGELOG.md b/CHANGELOG.md index 619faabcc..25a4c196b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli lint --baseline ` reports only new findings** (ako/mxcli#1328, mendixlabs/mxcli#1455) — record a baseline with `--update-baseline`, commit it, and later runs report only the findings it does not have, with an exit code that follows them alone and a `N new, M already known, K fixed` summary on stderr. Findings are matched on rule and document, not message, so rewording a rule does not make its findings new; a scoped run (`-m`, `-d`, `-r`) reports new findings but cannot update the baseline. +- **DROP over `--mcp` uses Studio Pro 11.16's `ped_delete_document`; Concord support is removed** — DROP of a constant, enumeration, microflow, nanoflow, page, workflow or business-event service calls the native delete tool. A server without it (Studio Pro before 11.16) gets a clear error naming the version it needs. The optional second MCP server (Concord) and its `--mcp-concord`, `--mcp-concord-dial`, `--mcp-save`, `--mcp-check` and `--mcp-run` flags are gone: PED persists every write to disk, `--mcp-check` is `ped_check_errors`, and running the app goes through Studio Pro 11.16's `mendix_app` handler. - **Lint permission rows name their access rule** (ako/mxcli#1437, mendixlabs/mxcli#1539) — `permissions()` and `permissions_for()` rows carry `access_rule_id` and `access_rule_index`, so a rule can count an entity's access rules per setting or list one rule's module roles exactly, instead of guessing by grouping on XPath, default and rights. The catalog schema moves to version 31, so a cached catalog is rebuilt once. - **Lint rules can read the data guideline rules asked for** (ako/mxcli#1453, ako/mxcli#1301) — a microflow's `excluded` and `apply_entity_access`, an enumeration's `values` with their captions (so a rule can resolve `getCaption(Module.Enum.Value)`), `constant_values()` for the run-configuration values, and `project_settings()` (default language, OQL version, Java version, hash algorithm, time zone, rounding, decimal scale, optimistic locking, foreign keys, web client). The catalog schema moves to version 30, so a cached catalog is rebuilt once. - **Configurable microflow and nanoflow layout** (ako/mxcli#1832) — a committed `layout-config.yaml` (in `.claude/` or next to the `.mpr`) chooses how `create microflow`/`create nanoflow` and `mxcli layout flows` place what a script leaves unpositioned: preset `happy-line` (the default, unchanged) or `true-right-false-down`, and the options `if-else: true-right-else-column` (ELSE stacked as a column under its decision), `case-split: first-right-rest-columns` (a CASE's or type split's first outcome on the line, the rest as columns), `guard: true-right`, `wrap: off` and `max-row-width`. A flow laid out in a non-default style records it as an `@mxcli-layout:` line in its documentation, which DESCRIBE and `create or modify` keep using for that flow, so its description stays free of `@position` when the project style changes. `mxcli layout config` shows the style in force; `mxcli layout flows --style ` tries one. Without a config file nothing changes: measured on 137 flows, the geometry and DESCRIBE output are identical to before. @@ -20,6 +21,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A PED tool whose handler throws is reported as a failure** — Studio Pro answers `Error executing tool handler: …` with `isError:false` (measured on 11.16.0), which a caller trusting the flag read as success; the MCP client now classifies that text as an error. - **MPR012 follows the app's Web UI client** (ako/mxcli#1287, mendixlabs/mxcli#1226) — a Mendix 11 app can still run the classic client, where the legacy static and dynamic image widgets build clean, but MPR012 warned on each of them that "mxbuild reports CE0582". On the classic client or in migration mode it is now an info-level note that the widget needs converting before the app switches; under the React client it stays a warning. - **Every `mxcli init --tool ` installs the bundled lint rules, and `lint` says when it runs without them** (ako/mxcli#1292, mendixlabs/mxcli#1242) — only `claude` and `opencode` wrote `.claude/lint-rules/`, so `init --tool cursor` (or aider, continue, copilot, vibe, windsurf) left `mxcli lint` running the built-in Go rules alone, silently. `lint` and `report` now print a warning when no `.claude/lint-rules/` is found, naming how many rules run instead. - **`show references to` an entity, attribute or association lists the message definitions that expose it** (#1781) — a message definition (the pre-11.15 collection, `MESSAGE_DEFINITION_COLLECTION`, and the 11.15+ document, `MESSAGE_DEFINITION`) was not walked for references, so impact analysis called an exposed entity or attribute unused. The entity edges are `mapping`, the attribute and association edges `member`, as for an import or export mapping. diff --git a/cmd/mxcli/engine.go b/cmd/mxcli/engine.go index 992d4ce56..1f261869a 100644 --- a/cmd/mxcli/engine.go +++ b/cmd/mxcli/engine.go @@ -54,17 +54,9 @@ func newBackendFactory() func() backend.FullBackend { // writes route to Studio Pro, reads come from -p. if globalMCPURL != "" { url, dial := globalMCPURL, globalMCPDial - concordURL, concordDial := globalMCPConcord, globalMCPConcordDial - saveOnExit, checkOnExit, runOnExit := globalMCPSave, globalMCPCheck, globalMCPRun + checkOnExit, runOnExit := globalMCPCheck, globalMCPRun return func() backend.FullBackend { - b := mcpbackend.New(url, dial) - if concordURL != "" || saveOnExit || checkOnExit || runOnExit { - b = b.WithConcord(mcpbackend.ConcordConfig{ - URL: concordURL, Dial: concordDial, - SaveOnExit: saveOnExit, CheckOnExit: checkOnExit, RunOnExit: runOnExit, - }) - } - return b + return mcpbackend.New(url, dial).WithExitActions(mcpbackend.ExitActions{Check: checkOnExit, Run: runOnExit}) } } warnIfEngineRequested() diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 926bd612a..2af995617 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -178,9 +178,6 @@ beta language; --mdl 0 starts them in the alpha language, and in the REPL an globalJSONFlag, _ = cmd.Flags().GetBool("json") globalMCPURL, _ = cmd.Flags().GetString("mcp") globalMCPDial, _ = cmd.Flags().GetString("mcp-dial") - globalMCPConcord, _ = cmd.Flags().GetString("mcp-concord") - globalMCPConcordDial, _ = cmd.Flags().GetString("mcp-concord-dial") - globalMCPSave, _ = cmd.Flags().GetBool("mcp-save") globalMCPCheck, _ = cmd.Flags().GetBool("mcp-check") globalMCPRun, _ = cmd.Flags().GetBool("mcp-run") globalMCPVerbose, _ = cmd.Flags().GetBool("mcp-verbose") @@ -248,15 +245,12 @@ var globalJSONFlag bool // routes model writes to a live Studio Pro while reads come from the local // .mpr given by -p. See docs/11-proposals/PROPOSAL_mcp_backend.md. var ( - globalMCPURL string - globalMCPDial string - globalMCPConcord string - globalMCPConcordDial string - globalMCPSave bool - globalMCPCheck bool - globalMCPRun bool - globalMCPVerbose bool - globalMCPTrace bool + globalMCPURL string + globalMCPDial string + globalMCPCheck bool + globalMCPRun bool + globalMCPVerbose bool + globalMCPTrace bool ) // resolveFormat returns the effective output format for a command. @@ -356,11 +350,8 @@ func init() { rootCmd.PersistentFlags().Bool("json", false, "Output in JSON format") rootCmd.PersistentFlags().String("mcp", "", "Route model writes to a live Studio Pro MCP server (e.g. http://localhost:7782/mcp); reads come from -p") rootCmd.PersistentFlags().String("mcp-dial", "", "Override the TCP address dialed for --mcp (e.g. host.docker.internal:7782 from a devcontainer)") - rootCmd.PersistentFlags().String("mcp-concord", "", "Optional second MCP server (Concord) for capabilities the built-in server lacks: delete, save, validate, run (e.g. http://localhost:7783/mcp)") - rootCmd.PersistentFlags().String("mcp-concord-dial", "", "Override the TCP address dialed for --mcp-concord") - rootCmd.PersistentFlags().Bool("mcp-save", false, "After the command, save all changes in Studio Pro via Concord (requires --mcp-concord; the built-in server has no save tool)") - rootCmd.PersistentFlags().Bool("mcp-check", false, "After the command, run Studio Pro's model consistency check via Concord and print the report (requires --mcp-concord)") - rootCmd.PersistentFlags().Bool("mcp-run", false, "After the command, start the app in Studio Pro via Concord (run_app) and print its URL (requires --mcp-concord)") + rootCmd.PersistentFlags().Bool("mcp-check", false, "After the command, run Studio Pro's project-wide error check (ped_check_errors, Studio Pro 11.15+) and print the report") + rootCmd.PersistentFlags().Bool("mcp-run", false, "After the command, start the app in Studio Pro (mendix_app handler, Studio Pro 11.16+) and print its URL") rootCmd.PersistentFlags().Bool("mcp-verbose", false, "Print each PED tool call the MCP backend makes (requires --mcp)") rootCmd.PersistentFlags().Bool("mcp-trace", false, "Print each MDL command with the PED tool calls it makes (implies --mcp-verbose; requires --mcp)") rootCmd.PersistentFlags().String("engine", "", "Deprecated and ignored: there is one model engine. Kept so scripts pinning the old one keep running.") diff --git a/cmd/mxcli/mcp.go b/cmd/mxcli/mcp.go index 016eb42f4..a9ec849be 100644 --- a/cmd/mxcli/mcp.go +++ b/cmd/mxcli/mcp.go @@ -38,9 +38,6 @@ Example: os.Exit(1) } b := mcpbackend.New(globalMCPURL, globalMCPDial) - if globalMCPConcord != "" { - b = b.WithConcord(mcpbackend.ConcordConfig{URL: globalMCPConcord, Dial: globalMCPConcordDial}) - } if err := b.Connect(projectPath); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) diff --git a/docs-site/src/tools/mcp-connect.md b/docs-site/src/tools/mcp-connect.md index 8386daeef..c0cfb0600 100644 --- a/docs-site/src/tools/mcp-connect.md +++ b/docs-site/src/tools/mcp-connect.md @@ -33,11 +33,8 @@ of routing through MCP, see |---|---| | `--mcp ` | Enable the MCP backend. The URL is what the **server** must see; its host drives the `Host` header and must stay `localhost` — e.g. `http://localhost/mcp`. | | `--mcp-dial ` | The TCP address mxcli actually connects to, independent of the URL host. From a devcontainer this is typically `host.docker.internal:`. | -| `--mcp-concord ` | Optional second MCP server ("Concord") for operations PED lacks: **delete**, **save**, **validate**, **run** (default port `7783`). | -| `--mcp-concord-dial ` | Dial override for Concord. | -| `--mcp-save` | After the command, save all changes in Studio Pro (requires `--mcp-concord`). | -| `--mcp-check` | After the command, run Studio Pro's consistency check and print the report (requires `--mcp-concord`). | -| `--mcp-run` | After the command, start the app in Studio Pro and print its URL (requires `--mcp-concord`). | +| `--mcp-check` | After the command, run Studio Pro's project-wide error check (`ped_check_errors`) and print the report. Needs Studio Pro 11.15+. | +| `--mcp-run` | After the command, start the app in Studio Pro (the `mendix_app` handler) and print its URL. Needs Studio Pro 11.16+. | | `--mcp-verbose` | Print each PED tool call the backend makes (to stderr). | | `--mcp-trace` | Print each MDL command with the PED tool calls it makes (implies `--mcp-verbose`). | @@ -121,24 +118,26 @@ see what actually goes over the wire — for debugging, or just to understand th This makes the statement-to-tool-call mapping (and surprises like a read-back reconstructing a dirty module) visible at runtime. -## Adding Concord (optional) +## Delete, check and run -PED cannot delete documents, save, validate, or run the app. A companion server, -**Concord**, fills those gaps (default port `7783`, usually reachable directly from -the container — no bridge needed): +Studio Pro's own MCP server covers what used to need a second server: + +- **Saving** needs nothing: PED writes each change to disk as it lands. +- **`DROP`** of a standalone document (enumeration, constant, microflow, nanoflow, + page, workflow, business-event service) calls `ped_delete_document`, which + Studio Pro 11.16 added. An older Studio Pro gets an error naming the version. +- **`--mcp-check`** runs `ped_check_errors` over the whole project (11.15+). +- **`--mcp-run`** starts the app through the `mendix_app` handler (11.16+), waits + up to two minutes and prints its URL. A project with model errors is not run; + the refusal is printed instead. The runtime log is `read_file /app-logs/app.log` + on the same server. ```bash mxcli exec changes.mdl -p app.mpr \ --mcp http://localhost/mcp --mcp-dial host.docker.internal:7784 \ - --mcp-concord http://localhost:7783/mcp \ - --mcp-save --mcp-check + --mcp-check --mcp-run ``` -Here mxcli applies the script through PED, then asks Concord to **save** the -project and run a **consistency check**, printing the report. `DROP` of standalone -documents (enumerations, microflows, pages, …) also requires `--mcp-concord`, -because the delete tool lives on Concord. - ## Same machine as Studio Pro If mxcli runs on the host itself, point `--mcp-dial` at the loopback bridge @@ -168,8 +167,8 @@ connected version supports. See also |---|---| | `connection refused` on dial | The bridge isn't running, or the port is wrong. Re-run `socat` on the host and confirm PED's port with `lsof` (it changes per session). | | Request rejected / 403-style error | The `Host` header guard. Keep the **URL host `localhost`** (`http://localhost/mcp`) and redirect the connection with `--mcp-dial` — never put the container/host address in the URL itself. | -| Reads don't reflect a change you just made | Expected for some edits until the project is saved — reads of an *unchanged* doc come from the local `.mpr`. A module you just wrote is reconstructed live; other doc types may need a save (`--mcp-save`) to appear locally. | -| `DROP` / save / check / run fails or is rejected | Those need Concord — add `--mcp-concord` (and `--mcp-save` / `--mcp-check` / `--mcp-run`). | +| Reads don't reflect a change you just made | Expected for some edits until the project is saved — reads of an *unchanged* doc come from the local `.mpr`. A module you just wrote is reconstructed live. | +| `DROP` / `--mcp-check` / `--mcp-run` reports it needs a newer Studio Pro | `DROP` and `--mcp-run` need Studio Pro 11.16+, `--mcp-check` 11.15+. On 11.16.0, `ped_delete_document` fails with `Session 'mcp-server' not found` for callers outside Studio Pro's own agent session; delete the document in Studio Pro. | | Intermittent timeouts | The transport can be flaky across a bridge; a timed-out write may still have applied. Re-run `mxcli mcp capabilities` to re-establish, and prefer idempotent scripts. | ## See also diff --git a/docs-wiki/architecture/mcp-backend.md b/docs-wiki/architecture/mcp-backend.md index a817e5174..234458a21 100644 --- a/docs-wiki/architecture/mcp-backend.md +++ b/docs-wiki/architecture/mcp-backend.md @@ -6,7 +6,8 @@ sources: - docs/03-development/PED_MCP_CAPABILITIES.md - mdl/backend/mcp/backend.go - mdl/backend/mcp/read_router.go - - mdl/backend/mcp/concord.go + - mdl/backend/mcp/delete.go + - mdl/backend/mcp/app.go --- > **Do not duplicate**: the tool matrix, transport setup, and capability gaps @@ -20,8 +21,9 @@ A second implementation of the executor's backend interface (alongside the MPR backend) that does **not** write BSON to disk. Instead it routes model mutations to a **live, running Studio Pro** through that IDE's embedded MCP server — codenamed **PED** — while still reading structure from the local `.mpr` on disk. It is a -*hybrid* backend: disk for reads, the live IDE for writes. A second optional MCP -server, **Concord**, fills a few gaps PED lacks (notably document deletion). +*hybrid* backend: disk for reads, the live IDE for writes. Deleting a document and +running the app go through the same server: `ped_delete_document` and the +`mendix_app` handler, both new in Studio Pro 11.16. ## How it fits diff --git a/docs/01-project/MDL_FEATURE_MATRIX.md b/docs/01-project/MDL_FEATURE_MATRIX.md index 03029c7bb..f8c7c8bb6 100644 --- a/docs/01-project/MDL_FEATURE_MATRIX.md +++ b/docs/01-project/MDL_FEATURE_MATRIX.md @@ -17,42 +17,41 @@ sits above both axes — how the whole Maia/mxcli spectrum compares.) ## Agent × tooling at a glance (Maia vs mxcli) -One-slide comparison of how a Mendix app can be edited by an AI agent across five +One-slide comparison of how a Mendix app can be edited by an AI agent across four setups: **Maia** (Studio Pro's built-in agent — reads/writes the model through Studio Pro's *internal* model APIs) → **Maia + MCP** (those same APIs now exposed -via the PED MCP server) → **Maia + MCP + Concord** (plus the Concord extension's -tools) → **mxcli** (external CLI, file-based on the on-disk `.mpr`, project closed) → -**mxcli + MCP + Concord** (external CLI driving the **live** model while Studio Pro -is open). +via the PED MCP server) → **mxcli** (external CLI, file-based on the on-disk `.mpr`, +project closed) → **mxcli + MCP** (external CLI driving the **live** model while +Studio Pro is open). -The first three columns share Maia's **native reach** — MCP/Concord don't change +The first two columns share Maia's **native reach** — MCP doesn't change *what Maia can do*; they expose that reach to **external** agents (the bottom *drivable externally* row). PED is exactly that exposure of Studio Pro's model APIs, -and mxcli (cols 4–5) is the external agent consuming it. The capability rows show +and mxcli (cols 3–4) is the external agent consuming it. The capability rows show each setup's *own* reach; a Maia–vs–mxcli-live gap on a row is either a PED-subset limit or just where each agent's implementation has gotten to. Legend: ✓ full · ◑ partial / subset · ✗ none · — n/a. -| Capability | Maia | Maia + MCP | Maia + MCP + Concord | mxcli (file) | mxcli + MCP + Concord | -|------------|:----:|:----------:|:--------------------:|:------------:|:---------------------:| -| **Authoring** | | | | | | -| Domain model — entities, view entities, enums, associations, constraints | ✓ | ✓ | ✓ | ✓ | ◑¹ | -| Pages (incl. pluggable widgets) | ◑² | ◑² | ◑² | ✓ | ◑² | -| Microflows | ◑³ | ◑³ | ◑³ | ✓ | ◑³ | -| Workflows | ✓ | ✓ | ✓ | ✓ | ✓ | -| Theming / styling | ✓ | ✓ | ✓ | ◑⁴ | ◑⁴ | -| Integrations — REST/OData/SOAP, business events, mappings, mobile | ✗ | ✗ | ✗ | ◑⁵ | ✗ | -| Security — roles & access | ✗ | ✗ | ✗ | ✓ | ✗ | -| Java / JS / CSS source | ✓⁶ | ✓⁶ | ✓⁶ | ✓ | ✓ | -| **Search & analysis** | | | | | | -| Impact / lint / catalog SQL / full-text | ✗ | ✗ | ✗ | ✓ | ✓ | -| Mendix KB search / NL→OQL | ✓ | ✓ | ✓ | ✗ | ✗ | -| **Reach & workflow** | | | | | | -| Validation before writing | ◑⁷ | ◑⁷ | ◑⁷ | ✓ | ✓ | -| Batch / scriptable / CI | ✗ | ✗ | ✗ | ✓ | ✓ | -| Live edits in the open IDE | ✓ | ✓ | ✓ | ✗⁸ | ✓ | -| Drivable by an external agent (headless) | ✗ | ◑⁹ | ✓ | ✓ | ✓ | +| Capability | Maia | Maia + MCP | mxcli (file) | mxcli + MCP | +|------------|:----:|:----------:|:------------:|:-----------:| +| **Authoring** | | | | | +| Domain model — entities, view entities, enums, associations, constraints | ✓ | ✓ | ✓ | ◑¹ | +| Pages (incl. pluggable widgets) | ◑² | ◑² | ✓ | ◑² | +| Microflows | ◑³ | ◑³ | ✓ | ◑³ | +| Workflows | ✓ | ✓ | ✓ | ✓ | +| Theming / styling | ✓ | ✓ | ◑⁴ | ◑⁴ | +| Integrations — REST/OData/SOAP, business events, mappings, mobile | ✗ | ✗ | ◑⁵ | ✗ | +| Security — roles & access | ✗ | ✗ | ✓ | ✗ | +| Java / JS / CSS source | ✓⁶ | ✓⁶ | ✓ | ✓ | +| **Search & analysis** | | | | | +| Impact / lint / catalog SQL / full-text | ✗ | ✗ | ✓ | ✓ | +| Mendix KB search / NL→OQL | ✓ | ✓ | ✗ | ✗ | +| **Reach & workflow** | | | | | +| Validation before writing | ◑⁷ | ◑⁷ | ✓ | ✓ | +| Batch / scriptable / CI | ✗ | ✗ | ✓ | ✓ | +| Live edits in the open IDE | ✓ | ✓ | ✗⁸ | ✓ | +| Drivable by an external agent (headless) | ✗ | ◑⁹ | ✓ | ✓ | ¹ mxcli over MCP can't author indexes or entity validation rules (file mode can); Maia does constraints, not indexes.   ² Pluggable widgets: Maia not yet; mxcli @@ -65,8 +64,9 @@ reads.   ⁶ Maia via the `write_file` virtual FS; mxcli edits source on di directly (any mode).   ⁷ Maia: live, reactive validation in the IDE; mxcli: static pre-flight (`mxcli check`) *before* writing.   ⁸ File mode edits the on-disk `.mpr`; reopen Studio Pro to see changes.   ⁹ MCP exposes Maia's -read + authoring surface (the PED subset) to external agents; Concord additionally -exposes delete + save / build-run. Maia performs all of this natively in-IDE — its +read + authoring surface (the PED subset) to external agents; Studio Pro 11.16 +adds delete (`ped_delete_document`) and run (the `mendix_app` handler), though on +11.16.0 delete fails for callers outside Studio Pro's own agent session. Maia performs all of this natively in-IDE — its column is about *external* reach. **Takeaway:** Maia has broad **native** authoring (domain model, pages, microflows, @@ -74,7 +74,7 @@ workflows, theming, source), and **MCP is what exposes that surface to external agents** — the architectural point. mxcli is that external agent: **mxcli (file)** is the only setup with *full* authoring (indexes, security, integrations) plus analysis + pre-flight validation + batch/CI, at the cost of the project being closed; -**mxcli + MCP + Concord** keeps the analysis / validation / scripting while editing +**mxcli + MCP** keeps the analysis / validation / scripting while editing the **live** model (authoring capped to the PED subset, which differs slightly from Maia's — pluggable widgets yes, theming/source via disk). Maia owns the conversational + Mendix-KB experience and in-IDE liveness. (Maia capabilities per the @@ -110,19 +110,19 @@ live distinction is **MPR vs MCP**. | **Modules** | Y | Y | Y | P | CREATE only (flushes to disk); no module DROP | | **Entities** | Y | Y | Y | Y | CREATE/DROP; ALTER add/drop/rename/docs — MODIFY *type* rejected (PED can't migrate) | | **Associations** | Y | Y | Y | P | Within-module CREATE/DROP; no custom delete behavior; cross-module / external N | -| **Enumerations** | Y | Y | Y | P | CREATE; DROP via Concord; ALTER (modify values) N | +| **Enumerations** | Y | Y | Y | P | CREATE; DROP via `ped_delete_document` (Studio Pro 11.16+); ALTER (modify values) N | | **Microflows** | Y | Y | Y | P | Broad CREATE (most actions + control flow); rejects show-page, cast, retrieve sort/range, inheritance/rule splits | -| **Nanoflows** | Y | Y | Y | N | **Not creatable** — PED's `ped_create_document` rejects `Microflows$Nanoflow` (create whitelist covers microflows, not nanoflows); CREATE/ALTER rejected with a clear error. DROP of an existing nanoflow works via Concord. | +| **Nanoflows** | Y | Y | Y | N | **Not creatable** — PED's `ped_create_document` rejects `Microflows$Nanoflow` (create whitelist covers microflows, not nanoflows); CREATE/ALTER rejected with a clear error. DROP of an existing nanoflow uses `ped_delete_document` (Studio Pro 11.16+). | | **Pages** | Y | Y | Y | P | CREATE + ALTER via `pg_*`; widget/data-source coverage grows one type at a time | | **Snippets** | Y | Y | Y | P | ALTER via the page mutator; CREATE not wired | | **Layouts** | Y | Y | Y | N | Not wired (reads delegate to the local `.mpr`) | | **View Entities** | Y | Y | Y | P | CREATE; CREATE OR REPLACE fails (no delete-document); DROP orphans the source doc | | **Workflows** | Y | Y | Y | Y | **Complete** — CREATE / CREATE OR REPLACE / DROP / ALTER, all activity types & ops, any nesting depth | | **Java Actions** | Y | Y | Y | N | **Not authorable** — PED refuses to create `JavaActions$JavaAction` (backed by a `.java` source the IDE manages); CREATE/ALTER rejected with a clear error. *Calling* a Java action from a microflow IS supported. | -| **Constants** | Y | Y | Y | P | CREATE / CREATE OR MODIFY (value + exposed-to-client) / DROP (via Concord); type limited to String/Integer/Decimal/Boolean/DateTime; in-place type change rejected; documentation not carried | +| **Constants** | Y | Y | Y | P | CREATE / CREATE OR MODIFY (value + exposed-to-client) / DROP (`ped_delete_document`, Studio Pro 11.16+); type limited to String/Integer/Decimal/Boolean/DateTime; in-place type change rejected; documentation not carried | | **OData Clients / Services** | Y | Y | Y | N | Not wired | | **External Entities** | Y | Y | Y | N | Not wired | -| **Business Events** | Y | Y | Y | P | SHOW/DESCRIBE read the local `.mpr`; DROP via Concord. **CREATE/ALTER blocked** — PED won't create `BusinessEvents$BusinessEventService` (off the create whitelist). The supporting domain model (published-event entities `extends BusinessEvents.PublishedBusinessEvent`, constants) IS creatable. | +| **Business Events** | Y | Y | Y | P | SHOW/DESCRIBE read the local `.mpr`; DROP via `ped_delete_document` (Studio Pro 11.16+). **CREATE/ALTER blocked** — PED won't create `BusinessEvents$BusinessEventService` (off the create whitelist). The supporting domain model (published-event entities `extends BusinessEvents.PublishedBusinessEvent`, constants) IS creatable. | | **Navigation** | Y | Y | Y | N | Not wired | | **Project Settings** | Y | Y | Y | N | Not wired | @@ -130,7 +130,7 @@ live distinction is **MPR vs MCP**. | Feature | Mendix | MDL | MPR | MCP | MCP notes | |---------|:------:|:---:|:---:|:---:|-----------| -| **Module / user roles, demo users, project security, entity / microflow / nanoflow / page access** | Y | Y | Y | N | **Not authorable via MCP** — PED's document tools reject every `Security$*` type and Concord exposes security *reads* only; neither server has a write path. | +| **Module / user roles, demo users, project security, entity / microflow / nanoflow / page access** | Y | Y | Y | N | **Not authorable via MCP** — PED's document tools reject every `Security$*` type; there is no write path. | ### Project organization diff --git a/docs/03-development/MCP_SAVE_VERIFICATION.md b/docs/03-development/MCP_SAVE_VERIFICATION.md deleted file mode 100644 index 27c9929e3..000000000 --- a/docs/03-development/MCP_SAVE_VERIFICATION.md +++ /dev/null @@ -1,89 +0,0 @@ -# Verifying `--mcp-save` from Concord (in-Studio-Pro test) - -`--mcp-save` flushes PED-authored, in-memory writes to disk via Concord's -`save_all` (PED has no save tool). `save_all` synthesizes a macOS Cmd-S -(osascript → System Events), so its reliability depends on the environment — see -the `save_all` caveats in [`PED_MCP_CAPABILITIES.md`](PED_MCP_CAPABILITIES.md). - -Testing it from a **devcontainer** is inconclusive: with two Studio Pro instances -sharing one bundle id, the synthetic Cmd-S targets the wrong window and silently -no-ops (`{"status":"save_command_sent"}` with no disk change). The authoritative -test is from **Claude Code running in the Concord terminal inside Studio Pro on -macOS**, with a single active instance — there the keystroke has an unambiguous -target. - -## The test prompt - -Paste this into Claude Code in the Concord terminal (Studio Pro, macOS): - -> **Test whether mxcli's `--mcp-save` actually persists a change to disk.** -> -> You're in the Concord terminal inside Studio Pro on macOS. Two local MCP -> servers: **PED** (Studio Pro's built-in authoring) on `localhost:7782`, and -> **Concord** on `localhost:7783`. mxcli is built from `main`. The MCP backend -> reads from the local `.mpr` and writes via PED; `--mcp-save` flushes via -> Concord's `save_all`. Everything is local — **no socat/port-forwarding**. -> -> Goal: confirm a PED-authored entity, flushed with `--mcp-save`, lands on disk. -> -> 1. **Find the open project.** It's the project Studio Pro has open. Set: -> ```bash -> MPR="" # e.g. .../mx-test-projects/test7-app/test7.mpr -> DIR=$(dirname "$MPR") -> ``` -> Pick an **existing user module** in it (e.g. `MyFirstModule` in test7) — call it `MOD`. -> -> 2. **Record disk state before:** -> ```bash -> stat -f '%m %N' "$DIR/mprcontents" # note this mtime -> ``` -> -> 3. **Create an entity via PED and save via Concord:** -> ```bash -> mxcli -p "$MPR" \ -> --mcp http://localhost/mcp --mcp-dial localhost:7782 \ -> --mcp-concord http://localhost/mcp --mcp-concord-dial localhost:7783 \ -> --mcp-save --mcp-check \ -> -c "create entity $MOD.SaveProbeMac (Code: string)" -> ``` -> If it can't connect to PED, retry with bare `--mcp http://localhost:7782/mcp` -> (and same for `--mcp-concord` on 7783). Note any `warning: --mcp-save failed …` line. -> -> 4. **Verify it persisted** (these are the same files Studio Pro saves to): -> ```bash -> stat -f '%m %N' "$DIR/mprcontents" # mtime should have advanced -> grep -rl SaveProbeMac "$DIR/mprcontents/" # should find the entity -> ``` -> -> 5. **Report:** -> - Did mxcli print `Created entity`? Any `--mcp-save failed` warning? What did `--mcp-check` print? -> - Did the `mprcontents` mtime change, and is `SaveProbeMac` on disk? -> - **Conclusion: did `--mcp-save` persist the change?** -> -> Important conditions (the whole point of this test): -> - Run with **only ONE Studio Pro instance** open — `save_all` sends a synthetic -> Cmd-S, which is ambiguous if two instances share the bundle id. -> - Make sure **Studio Pro is the frontmost/active app** when the command runs. -> - If it didn't persist, do a **manual Cmd-S** in Studio Pro and re-check — that -> isolates "the automation didn't land" from "the project can't save." - -## Result (2026-06-08, Studio Pro 11.11 Beta) - -**Confirmed broken.** Run from Claude Code in the Concord terminal with a single -active Studio Pro instance (frontmost), `save_all` did **not** persist the change -**and hung Studio Pro** while Concord tried to drive the save. This rules out the -two-instance ambiguity — it is a genuine **Concord/Studio Pro `save_all` bug** -(the synthetic Cmd-S hangs the IDE), to be reported upstream. Use manual Cmd-S in -Studio Pro meanwhile. `--mcp-save` is wired correctly and will work unchanged once -`save_all` is fixed; re-run this procedure to re-confirm after a Concord update. - -## Interpreting the result - -- **mtime advances + `SaveProbeMac` on disk** → `--mcp-save` works; the - devcontainer failure was purely the two-instance ambiguity. Update the - `save_all` caveat in `PED_MCP_CAPABILITIES.md` accordingly. -- **No disk change, but manual Cmd-S does save** → Concord's `save_all` keystroke - isn't reaching Studio Pro even single-instance; report upstream to Concord (its - own error text invites this). -- **Even manual Cmd-S doesn't save** → a Studio Pro / project issue unrelated to - mxcli. diff --git a/docs/03-development/PED_MCP_CAPABILITIES.md b/docs/03-development/PED_MCP_CAPABILITIES.md index b2e24eae8..0af8c554d 100644 --- a/docs/03-development/PED_MCP_CAPABILITIES.md +++ b/docs/03-development/PED_MCP_CAPABILITIES.md @@ -564,7 +564,7 @@ behaviour change, not an error. | **Changed** | Workflows constructor (`ped_get_schema`) | `Workflows$Workflow`'s constructor renamed `caption` to **`title`**; `eventSubProcesses` gained a default `[]`; the `EventSubProcess` constructor no longer lists `persistentId` (reads still return it, so the boundary-event lookup by `persistentId` is unaffected). **Neither release's constructor applies the title or `workflowName` it is given** — both store the document name — so DISPLAY was lost over MCP before 11.15 too. **Fixed:** the probe picks the key, and `workflowCreateLeafOps` sets `/title` and `/workflowName/text` after the create, which both releases accept. | | **Removed** | `read_skill` | Skills are files under `/skills` (`glob`/`read_file`). Unused by mxcli. | | **Removed** | `install_marketplace_module`, `search_mendix_knowledge_base` | Now **handlers**, reached through `query_handlers` / `execute_handler`. Unused by mxcli. | -| **New** | `query_handlers`, `execute_handler` | A handler registry: `install_marketplace_module`, `search_mendix_knowledge_base`, `mendix_app` (run / stop / status of the runtime), `vc_commit`, `vc_push`, the translation tools (`batch_translate`, `check_translation_status`, `find_text_usages`, `list_text_containers`, `reset_translations`) and marketplace search. `mendix_app` and `vc_commit` overlap what Concord supplies today (`--mcp-run`; and a commit is a save point) — not wired yet. | +| **New** | `query_handlers`, `execute_handler` | A handler registry: `install_marketplace_module`, `search_mendix_knowledge_base`, `mendix_app` (run / stop / status of the runtime), `vc_commit`, `vc_push`, the translation tools (`batch_translate`, `check_translation_status`, `find_text_usages`, `list_text_containers`, `reset_translations`) and marketplace search. `mendix_app` backs `--mcp-run` (see Delete, check and run); `vc_commit` is not wired. | | **Changed** | `read_file`, `write_file` | `startLine/endLine` → `offset/limit`; `newContent`+`span` → `oldString`/`newString`. Unused by mxcli (see the tool matrix). | | **Changed** | `glob` | New `/app-logs` file domain. | @@ -673,15 +673,15 @@ The right-hand column carries the latest measured status, not the 11.11 one. |-----|-----------------------------|-------------------------------| | **No delete-document tool** | `DROP` of any *standalone document* (enum, microflow, page) is impossible. Only entities/associations delete, via a `ped_update_document` remove op on the domain-model array. Test docs created via MCP cannot be cleaned up — they persist until the user closes Studio Pro without saving. | Watch for a `ped_delete_document` / equivalent. | | **No list-modules tool** ⟶ **CLOSED in 11.12** | PED cannot enumerate modules. The backend must read modules/structure from the local mounted `.mpr` (hybrid model); `-p` must be the same project Studio Pro has open. | **11.12 adds `list_modules`** (`[{moduleName, writable, fromMarketplace}]`) → pure-MCP module enumeration is now possible; the local-`.mpr` dependency for *module listing* can be lifted on 11.12+. Other reads (structure, IDs) still use the hybrid model. | -| **No save/flush tool** | `ped_update_document` edits stay in Studio Pro's in-memory model; the on-disk `.mpr` is stale until the user saves. Drives the dirty-set read router. **Refined on 11.12 (observed live, test8):** *created* documents (`ped_create_document`, `pg_patch_page` page-create) flush to `mprcontents/` immediately — no Concord `save_all` needed for them — but *update* ops still stay in memory (an `alter entity add attribute` was visible to `ped_check_errors` yet absent on disk → CE1613 on an `mx check` of a mid-session disk copy). (`ped_create_module` already flushed immediately on 11.11.) | Watch for a save tool / autosave; recheck the create-flush behavior each version. | +| **No save/flush tool** | `ped_update_document` edits stay in Studio Pro's in-memory model; the on-disk `.mpr` is stale until the user saves. Drives the dirty-set read router. **Refined on 11.12 (observed live, test8):** *created* documents (`ped_create_document`, `pg_patch_page` page-create) flush to `mprcontents/` immediately — but *update* ops still stay in memory (an `alter entity add attribute` was visible to `ped_check_errors` yet absent on disk → CE1613 on an `mx check` of a mid-session disk copy). (`ped_create_module` already flushed immediately on 11.11.) | Watch for a save tool / autosave; recheck the create-flush behavior each version. | | **Reads omit `$ID`** | Reads expose `$QualifiedName` only. Association refs need entity GUIDs, recovered from the local reader (= live `$ID` for saved entities). Reconstructed reads use synthetic IDs mapped to names. | Recheck whether reads expose `$ID`. | | **Array reads omit primitive types** | A `/entities/N/attributes` read gives attribute names (`$QualifiedName`) but not their primitive type or documentation — those need a per-*leaf* read (`/entities/N/attributes/M/type` → a `DomainModels$*AttributeType` constructor; `/…/documentation` → string). Reconstruction recovers them with a single **batched** leaf read (`enrichReconstructedEntities`) so a dirty/session module reports real types (correct DESCRIBE + a reliable ALTER diff); it falls back to placeholder `String` only if the read fails. | **11.14 — partially CLOSED.** An attributes-array read now carries each attribute's `type.$Type`, `value.$Type` and `documentation` inline (`{"$Type":"DomainModels$Attribute","$QualifiedName":"…Title","type":{"$Type":"DomainModels$StringAttributeType"},"value":{"$Type":"DomainModels$StoredValue"},"name":"Title","documentation":""}`). But the type's **parameters are still dropped**: the same attribute's leaf read returns `{"$Type":"DomainModels$StringAttributeType","length":200}` — `length` appears **only** in the leaf read. So `enrichReconstructedEntities` is still required for `length` / enumeration refs, but the type *kind* is now free. Not yet exploited; measured on one String attribute, so widen the sample before relying on it. | | **No in-place attribute-type change** | `set /entities/N/attributes/M/type` is rejected (`"only allowed to set primitive or reference properties directly"` — the type is a nested element), and a whole-attribute replace is rejected too (`"only allowed to update elements …"`). The only route to a new type is remove+add, which drops the attribute's `$ID` → **drops the column data**. So `ALTER ENTITY … MODIFY ATTRIBUTE ` is rejected over MCP; do it against a local `.mpr` (Studio Pro does an in-place migration PED can't). | **11.14 — the structural refusal is GONE.** 11.14's `set` explicitly *"can set or replace element-valued properties"*. Probed with a deliberately invalid value so it could not land: `set /entities/0/attributes/0/type` = `{"$Type":"DomainModels$NoSuchAttributeType"}` is now rejected as a **value validation** error (`"Type … is not a valid concrete subtype of DomainModels$AttributeType."`), not with 11.13's structural *"only allowed to set primitive or reference properties directly"*. That means the operation now reaches validation — i.e. a **valid** subtype would plausibly be accepted. **Not confirmed:** proving it needs a successful mutation against a throwaway project, and the data-safety question behind the gap (does an in-place type change migrate the column, or drop it?) is unanswered and is the *real* gate on `ALTER ENTITY … MODIFY ATTRIBUTE`. `capabilities.yaml` therefore still reports it unauthorable. | | **Two write protocols** | Pages **must** use `pg_*`; everything else uses `ped_*`. The system prompt forbids PED for pages. | Stable, but reconfirm. | -| **`ped_create_document` doc-type whitelist** | The create tool accepts only certain document types; some model documents are rejected even though they have a `$constructor` schema. Confirmed off the whitelist: **`Microflows$Nanoflow`** (`"… Did you mean: Microflows$Microflow?"`), **`Projects$Folder`** (empty folders), and **`BusinessEvents$BusinessEventService`** (a `$element`, not a `$constructor` — its CREATE/ALTER are rejected in `businessevent.go`, but SHOW/DESCRIBE read the `.mpr`, DROP goes via Concord, and the published-event entities/constants it relies on are creatable). So nanoflows aren't creatable over MCP (CREATE/ALTER rejected — `nanoflow.go`; DROP works via Concord), and an *empty* folder can't be created. **But `ped_create_document` accepts a `folderPath` per document, which auto-creates the whole folder path** — so a document is placed in a (possibly nested) folder at create time (`folder.go`/`resolveDocContainer`). What you can't do: re-parent an *existing* document — `/folderPath` is not a settable property (so `MOVE`/`DROP FOLDER`/`MOVE FOLDER` are rejected), and pages can't be foldered (`pg_write_page` takes no folderPath). | Re-probe the create whitelist each version. | +| **`ped_create_document` doc-type whitelist** | The create tool accepts only certain document types; some model documents are rejected even though they have a `$constructor` schema. Confirmed off the whitelist: **`Microflows$Nanoflow`** (`"… Did you mean: Microflows$Microflow?"`), **`Projects$Folder`** (empty folders), and **`BusinessEvents$BusinessEventService`** (a `$element`, not a `$constructor` — its CREATE/ALTER are rejected in `businessevent.go`, but SHOW/DESCRIBE read the `.mpr`, DROP uses `ped_delete_document` (11.16+), and the published-event entities/constants it relies on are creatable). So nanoflows aren't creatable over MCP (CREATE/ALTER rejected — `nanoflow.go`; DROP uses `ped_delete_document`, 11.16+), and an *empty* folder can't be created. **But `ped_create_document` accepts a `folderPath` per document, which auto-creates the whole folder path** — so a document is placed in a (possibly nested) folder at create time (`folder.go`/`resolveDocContainer`). What you can't do: re-parent an *existing* document — `/folderPath` is not a settable property (so `MOVE`/`DROP FOLDER`/`MOVE FOLDER` are rejected), and pages can't be foldered (`pg_write_page` takes no folderPath). | Re-probe the create whitelist each version. | | **No Java action document creation** | `ped_create_document` rejects `JavaActions$JavaAction` outright (`"Document type 'JavaActions$JavaAction' cannot be created."`) — a Java action is backed by a `.java` source file Studio Pro generates/manages, so the model document can't be created standalone. `CREATE/ALTER JAVA ACTION` are rejected with an actionable error (`mdl/backend/mcp/javaaction.go`); *calling* a Java action from a microflow is unaffected. | Watch for a Java-action create tool. | | **No mapping document ops** | PED's `ped_read/find/update_document` reject `ImportMappings$ImportMapping` and `ExportMappings$ExportMapping` ("Unknown document type" / "not supported"), while `ped_get_schema` returns full `element` **and** `constructor` schemas and `ped_list_folder` lists them — the security-document shape exactly. The constructor is `{ $Type, name }` only, so even a whitelisted create would yield an empty shell with no way to populate it. mxcli's MCP backend matches: all ten mapping methods are `unsupportedBackend` stubs (`unsupported_gen.go`). `JsonStructures$JsonStructure` — the mapping's *source* — **is** fully supported, and `ped_check_errors` **does** accept a mapping by name, so an on-disk-authored mapping can still be error-checked by Studio Pro. Re-probed 11.13.0, 2026-08-24. | Watch for the mapping types appearing in `ped_read_document`. | -| **No security document ops** | PED's `ped_find/read/update/create_document` reject every security type (`Security$ModuleSecurity`, `Security$ProjectSecurity`, …) as "Unknown document type" — only `ped_get_schema` knows them as nested *elements*. Concord exposes only security *reads* (`audit_security`, `read_entity_access_rules`, `read_microflow_security`, `read_security_info`). So the security **documents** cannot be authored via MCP (module/user roles, demo users, project security level) — neither server has a write path. **Entity access rules are the exception and are authorable**: a `DomainModels$AccessRule` lives on the domain model, not the security document, so `GRANT` works (ADD-only — PED refuses to remove a rule or a member access, so `REVOKE` and replacing a rule in place are rejected). Security **reads** are served from the local `.mpr` by the backend's reader, so `SHOW MODULE ROLES` / `USER ROLES` / `PROJECT SECURITY` work over `--mcp` and `GRANT` can validate the role it references — routing that read to PED instead is what made every `GRANT --mcp` abort before its first tool call (#900). Determine support with `ped_read_document`, NOT `ped_find_document`: `find` also reports the (supported) nameless `DomainModels$DomainModel` as "Unknown", so it is not a reliable probe. | **11.14 — still sealed, and now actively mis-advertised.** The tool descriptions name `Settings$ProjectSettings`, `Security$ProjectSecurity` and `Security$ModuleSecurity` as supported and `ped_get_schema` returns ten fully documented security element types, but every document-API call is refused (`Unknown document type` / `Document type … is not supported.`), against a `Navigation$NavigationDocument` control that reaches the operation layer in the same probe. No security type declares a **method**, so the new `call` op opens nothing. **One thing did move:** replacing an existing `DomainModels$AccessRule` in place no longer needs member removal — 11.14's `set` replaces a non-null element, and the probe clears shape validation and fails only at reference resolution. So re-GRANT looks reachable while **REVOKE is still unmeasured** (bounds and writability checks both mask removability). See the 11.14 delta for the probes and the control. | +| **No security document ops** | PED's `ped_find/read/update/create_document` reject every security type (`Security$ModuleSecurity`, `Security$ProjectSecurity`, …) as "Unknown document type" — only `ped_get_schema` knows them as nested *elements*. So the security **documents** cannot be authored via MCP (module/user roles, demo users, project security level) — there is no write path. **Entity access rules are the exception and are authorable**: a `DomainModels$AccessRule` lives on the domain model, not the security document, so `GRANT` works (ADD-only — PED refuses to remove a rule or a member access, so `REVOKE` and replacing a rule in place are rejected). Security **reads** are served from the local `.mpr` by the backend's reader, so `SHOW MODULE ROLES` / `USER ROLES` / `PROJECT SECURITY` work over `--mcp` and `GRANT` can validate the role it references — routing that read to PED instead is what made every `GRANT --mcp` abort before its first tool call (#900). Determine support with `ped_read_document`, NOT `ped_find_document`: `find` also reports the (supported) nameless `DomainModels$DomainModel` as "Unknown", so it is not a reliable probe. | **11.14 — still sealed, and now actively mis-advertised.** The tool descriptions name `Settings$ProjectSettings`, `Security$ProjectSecurity` and `Security$ModuleSecurity` as supported and `ped_get_schema` returns ten fully documented security element types, but every document-API call is refused (`Unknown document type` / `Document type … is not supported.`), against a `Navigation$NavigationDocument` control that reaches the operation layer in the same probe. No security type declares a **method**, so the new `call` op opens nothing. **One thing did move:** replacing an existing `DomainModels$AccessRule` in place no longer needs member removal — 11.14's `set` replaces a non-null element, and the probe clears shape validation and fails only at reference resolution. So re-GRANT looks reachable while **REVOKE is still unmeasured** (bounds and writability checks both mask removability). See the 11.14 delta for the probes and the control. | ## Transport (per environment, not per version) @@ -731,8 +731,7 @@ documentation field** (dropped). CREATE OR MODIFY sets the `defaultValue` and `exposedToClient` leaves in place; a *type* change is rejected because the model's `type` is a nested `DataTypes$*Type` element PED can't set directly (same constraint as an attribute's type — `UpdateConstant` reads the live type and refuses a -mismatch). DROP goes through Concord's `delete_document` (no PED delete tool), like -enumerations. +mismatch). DROP uses `ped_delete_document` (Studio Pro 11.16+), like enumerations. **CREATE MICROFLOW** maps a broad set of actions (variable/object/list changes, create/commit/delete/rollback, aggregate, list operations, retrieve, call @@ -1040,159 +1039,25 @@ the duplicate source-document create. See `mdl/backend/mcp/` and the return a clear "not supported by the MCP backend" error via the generated `unsupportedBackend` base. -## Concord (optional second client — gap-filler) - -> **Concord is Windows-only — it does not run on macOS** (confirmed 2026-08-11 on -> the 11.13 Mac host). On macOS every Concord-backed capability is simply absent, -> so `DROP` of a standalone document (enumeration, microflow, page) has **no path -> at all**: PED has no delete tool and Concord is the only gap-filler. `check_model` -> is likewise unavailable; `ped_check_errors` remains for per-document validation. -> This also means MCP-authored test documents cannot be cleaned up on macOS — -> remove them in Studio Pro, or close without saving. - -Some deployments run a second MCP server, **Concord** (a Studio Pro extension; -`concord-mcp`), alongside the built-in PED server. Concord is **not** an authoring -server — it has none of the `ped_*`/`pg_*` create tools — but it provides -operational/refactor capabilities PED lacks. The backend uses **PED for authoring -by default** and reaches for Concord **only** for these gaps. Configure it with -`--mcp-concord` / `--mcp-concord-dial` (a second `Client`, dialed independently); -it stays `nil` when not configured, and every Concord-backed op errors clearly if -it's missing. - -Wired so far: -- **`delete_document`** — real `DROP` of standalone documents (enumeration, - microflow, page), which PED cannot delete at all. `DROP ENUMERATION/MICROFLOW/ - PAGE` resolves the document's module + name and calls - `delete_document {module_name, document_name}`. Unlike `save_all` this is - **model-based, not keystroke automation**, so it is robust. (Entities and - associations still delete via PED's array-element removal — no Concord needed.) -- **`check_model`** (`--mcp-check`) — domain-model consistency check after writes - (PED has no validation for the live model). Parses `{success, healthy, summary - {errorCount, warningCount, …}, errors[], warnings[]}` and prints a report to - stderr on Disconnect. Model-based (robust). Note: `healthy: true` means zero - *errors*, not zero warnings — the report shows both. -- **`save_all`** (`--mcp-save`) — PED has no save tool, so writes live only in - Studio Pro's in-memory model until the user saves. `--mcp-save` flushes via - Concord's `save_all` on Disconnect. **Unreliable — keystroke automation.** - Concord's `save_all` synthesizes a macOS Cmd+S (osascript → System Events), with - two failure modes observed against 11.11 Beta: - 1. **Permission.** Needs macOS **Accessibility** on the *responsible* process. - If Studio Pro is launched from a shell that exec's the binary directly, the - responsible process is the **terminal**, not Studio Pro — relaunch via - `open -n -a "" --args …` (launchd → app is its own responsible process) - and grant Studio Pro Accessibility. Otherwise it fails `osascript is not - allowed to send keystrokes (1002)`. - 2. **The keystroke does not save, and hangs Studio Pro — confirmed broken in - 11.11 Beta (2026-06-08).** First observed from a devcontainer - (`{"status":"save_command_sent"}`, no disk change). Then re-tested the - authoritative way — from Claude Code in the Concord terminal, **single active - Studio Pro instance, app frontmost** — and it still did **not** persist *and* - **hung Studio Pro** while Concord tried to drive the save. So it is **not** - the two-instance ambiguity; it is a genuine **Concord/Studio Pro `save_all` - bug** (the synthetic Cmd-S hangs the IDE). **Do not rely on `--mcp-save`; - save manually (Cmd-S) in Studio Pro, and report `save_all` upstream to the - Concord/Studio Pro team.** The backend wiring is correct and will work - unchanged once `save_all` is fixed; a silently-no-op `save_command_sent` it - cannot detect. Model-based gap-closers (`delete_document`, `check_model`) are - unaffected — they do not use keystroke automation. - -- **`get_app_status`** — the API call works and returns well-formed - `{data:{running, runningUrl, projectName, …}}` (exposed as `GetAppStatus()`, - printed by `--mcp-run`). **But `running`/`runningUrl` are effectively a - port/process probe**, not the current session's console-managed runtime: it - reported `running | :8080` while the Studio Pro runtime console was empty — - because an **orphaned runtime from a previous run** was still bound to `:8080` - (restarting Studio Pro doesn't kill the separate runtime process; `:8080` was - confirmed listening). So trust the API shape, but treat `running: running` as - "something is bound to the runtime port," which may be stale. -- **`run_app` / `stop_app`** (`--mcp-run` starts the app) — ⚠️ **same - UI-automation failure as `save_all`.** They are "click the Run/Stop button" - automations: `stop_app` returned `{"status":"command_sent"}` but the app stayed - running across repeated `get_app_status` polls (2026-06-08, 11.11 Beta). So like - `save_all` they report success without taking effect; `run_app` almost certainly - behaves the same. Wired correctly (will work once Concord's UI automation is - fixed), but **don't rely on `--mcp-run`/`stop_app` — start/stop the app manually - in Studio Pro.** Report upstream. - -**Pattern (important):** Concord's **model-editing** tools work (`delete_document`, -`check_model`); its **UI-automation** tools that synthesize button clicks / -keystrokes do **not** in this environment (`save_all`, `stop_app`, `run_app` all -return a `*_command_sent` status with no actual effect, and `save_all` can hang -Studio Pro). `get_app_status` is read-only and returns valid data but reflects raw -port state (can be a stale runtime). Net: only `delete_document` and -`check_model` are dependable today. - -### Full tool inventory (44 tools, captured 2026-06-15) - -Concord identity: `concord-mcp` (proto `2025-03-26`), port 7783 (directly -container-reachable; no socat). **Two behaviors to know:** - -- **`tools/list` is context-curated.** After `concord__set_task_context`, a - subsequent `tools/list` returns only a *curated subset* relevant to the declared - task. The full 44 below come from a fresh `tools/list` with no task context set. - Always re-probe without a task context to see everything. -- **No Maia / agent-invoke tool.** Like PED, Concord exposes *tools an agent uses*, - not a "run the Maia agent" entrypoint. The `concord__*` tools are agent **self- - management** helpers (calibration, friction journal, tool discovery), not Maia. - -Wired/tested status: ✅ dependable · ⚠️ UI-automation, unreliable (see above) · -○ present, not yet wired/vetted. `R` = readOnlyHint. - -| Category | Tool | R/W | Status / note | -|----------|------|-----|---------------| -| Domain model | `rename_entity` | W | ○ in-place, identity-preserving rename (PED only does `set /name`) | -| Domain model | `rename_attribute` | W | ○ identity-preserving | -| Domain model | `rename_association` | W | ○ identity-preserving | -| Domain model | `rename_module` | W | ○ identity-preserving | -| Domain model | `rename_document` | W | ○ identity-preserving | -| Domain model | `set_documentation` | W | ○ set element documentation | -| Domain model | `arrange_domain_model` | W | ○ auto-layout entities/associations | -| Domain model | `delete_model_element` | W | ○ delete entity/attribute/association (PED already removes these via array op) | -| Constants/enums | `rename_enumeration_value` | W | ○ identity-preserving | -| Microflows | `modify_microflow_activity` | W | ○ mutate an activity by 1-based position | -| Microflows | `insert_before_activity` | W | ○ sequence an activity | -| Microflows | `set_microflow_url` | W | ○ | -| Pages | `delete_document` | W | ✅ real `DROP` of standalone docs (wired) | -| Pages | `exclude_document` | W | ○ | -| Pages | `generate_overview_pages` | W | ○ CRUD list + new-edit pages | -| Navigation | `manage_navigation` | W | ○ menu structure, home page, role gating (Concord convenience; mxcli now authors web-profile nav directly via generic `ped_update_document` on `Navigation$NavigationDocument` — see below) | -| Project settings | `read_configurations` | R | ○ | -| Project settings | `set_configuration` | W | ○ | -| Project settings | `read_runtime_settings` | R | ○ | -| Project settings | `set_runtime_settings` | W | ○ | -| Project settings | `get_active_run_configuration` | R | ○ which run config will run | -| Validation/diag | `check_model` | R | ✅ domain-model consistency check (wired, `--mcp-check`) | -| Validation/diag | `check_project_errors` | R | ○ full-project "Check All Errors" — reported stubbed in Concord | -| Validation/diag | `analyze_project_patterns` | R | ○ architectural pattern/anti-pattern scan | -| Validation/diag | `get_last_error` | R | ○ | -| Validation/diag | `get_studio_pro_logs` | R | ○ | -| Security (read) | `audit_security` | R | ○ full security audit (intended) | -| Security (read) | `read_entity_access_rules` | R | ○ | -| Security (read) | `read_microflow_security` | R | ○ | -| Security (read) | `read_security_info` | R | ○ project security overview | -| App runtime | `get_app_status` | R | ✅ valid data, but raw port probe (can be a stale runtime) | -| App runtime | `run_app` | W | ⚠️ UI automation — `command_sent`, no effect | -| App runtime | `stop_app` | W | ⚠️ UI automation — `command_sent`, app stays running | -| App runtime | `save_all` | W | ⚠️ UI automation — does not persist and **hangs Studio Pro**; save manually | -| Project | `refresh_project` | W | ○ force re-scan of project dir for external file changes | -| Concord meta | `concord__session_context` | R | ○ session-start briefing (call once) | -| Concord meta | `concord__set_task_context` | W | ○ declares task → curates subsequent `tools/list` | -| Concord meta | `concord__find_tool` | R | ○ free-text "which tool for this task" | -| Concord meta | `concord__verify` | R | ○ verify a high-level workflow (e.g. `entity_created`) actually applied | -| Concord meta | `concord__diagnose_project` | R | ○ parallel project-state probes (SP running? .mpr locked?) | -| Concord meta | `concord__calibration_summary` | R | ○ C4 calibration journal summary | -| Concord meta | `concord__record_calibration` | W | ○ append (claimed, actual) calibration entry | -| Concord meta | `concord__silence_friction` | W | ○ suppress friction-wire emission for a tool | -| Concord meta | `concord__tried_approaches` | R | ○ prior friction occurrences for a pattern/tool | - -**Highest-value unwired gap-closers** (would fill PED limits the matrix currently -marks unsupported, pending probe + identity-preservation vetting per -[ADR-0002](../13-decisions/0002-backend-abstraction.md) and the "don't fake -identity ops" rule): the `rename_*` family (true renames vs PED's `set /name`). -(Navigation is no longer in this list — mxcli authors web-profile navigation -directly over PED; see the navigation note below.) `check_project_errors` and -`refresh_project` are also candidates. `delete_model_element` is low priority (PED -already removes entities/associations). +## Delete, check and run + +mxcli once paired PED with a second MCP server, Concord (a Studio Pro extension), +for delete, save, validate and run. That support is removed: Studio Pro's own +server covers each of them, and Concord's save and run were keystroke and button +automations that reported success without taking effect. The Concord measurements +are in this file's git history. + +| Operation | Tool | Studio Pro | Notes | +|---|---|---|---| +| Save | — | all | Nothing to do. PED writes each change to disk as it lands (measured on 11.12.0 and 11.16.0; Save All then writes 0 files). | +| `DROP` of a standalone document | `ped_delete_document` | 11.16+ | `deleteDocument` (`delete.go`), gated on the tool's presence. An older server gets an error naming 11.16+. **On 11.16.0 the tool fails for every caller outside Studio Pro's own agent session** (`Error executing tool handler: Session 'mcp-server' not found`, `isError:false`), so its success reply is unmeasured and an unrecognised reply is settled with `ped_find_document`. | +| `--mcp-check` | `ped_check_errors`, no filters | 11.15+ | `CheckProject` (`check_errors.go`): the whole project's listing, every page. Up to 11.14 the tool requires `documents`, so there is no project-wide form. | +| `--mcp-run` | `execute_handler` `mendix_app` `{action: run, timeoutSeconds: 120}` | 11.16+ | `RunApp` (`app.go`). Presence is asked once with `query_handlers {operation: getDetails, names: [mendix_app]}`; 11.12 has neither tool. `status` answers `{"running":false,"launching":false,"appUrl":"http://localhost:8080/"}` from outside Studio Pro (measured 2026-11-01). The `run` reply is unmeasured: one that is not a status is settled by asking `status`. A project with model errors is refused, not run. Runtime log: `read_file /app-logs/app.log`. | + +**Playwright** is reachable the same way: `playwright_status` answered "Playwright +is available; use the browser_* handlers to check the UI." on 11.16.0, and the +`mcp_studio-pro-playwright_browser_*` handlers (navigate, snapshot, click, type, +wait_for, …) drive the running app. mxcli does not call them yet. ## Onboarding a new Studio Pro version diff --git a/mdl/backend/mcp/app.go b/mdl/backend/mcp/app.go new file mode 100644 index 000000000..031b47713 --- /dev/null +++ b/mdl/backend/mcp/app.go @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "encoding/json" + "fmt" + "io" + "strings" +) + +// Running the app goes through a Studio Pro 11.16 handler, not a tool: the +// server advertises execute_handler and query_handlers, and mendix_app (run, +// stop, status) is one of the handlers behind them. Studio Pro 11.12 has +// neither tool, so every call here is gated on their presence. + +const ( + mendixAppHandler = "mendix_app" + // appRunTimeoutSeconds is the longest wait mendix_app accepts (schema max + // 120). A timeout does not cancel the run; status still reports it. + appRunTimeoutSeconds = 120 + // appLogPath is where Studio Pro serves the runtime log (read_file). + appLogPath = "/app-logs/app.log" +) + +// AppStatus is mendix_app's status reply, measured on 11.16.0: +// {"running":false,"launching":false,"appUrl":"http://localhost:8080/"}. +type AppStatus struct { + Running bool `json:"running"` + Launching bool `json:"launching"` + AppURL string `json:"appUrl"` +} + +// requireHandler reports an actionable error unless the server exposes the +// named handler. Presence is asked once per session with query_handlers +// getDetails, which execute_handler's own description requires before a call. +func (b *Backend) requireHandler(name string) error { + if b.client == nil || !b.client.HasTool("execute_handler") || !b.client.HasTool("query_handlers") { + return fmt.Errorf("%s needs Studio Pro 11.16+ (execute_handler); this Studio Pro's MCP server has no handlers", name) + } + if ok, cached := b.handlers[name]; cached { + if !ok { + return fmt.Errorf("this Studio Pro offers no %s handler", name) + } + return nil + } + res, err := b.client.CallTool("query_handlers", map[string]any{ + "operation": "getDetails", + "names": []string{name}, + }) + if err != nil { + return err + } + text := pedStripReminder(res.Text) + if res.IsError { + return fmt.Errorf("query_handlers: %s", text) + } + var details struct { + Handlers []struct { + Name string `json:"name"` + } `json:"handlers"` + } + if err := json.Unmarshal([]byte(strings.TrimSpace(text)), &details); err != nil { + return fmt.Errorf("query_handlers: parsing result: %w", err) + } + found := false + for _, h := range details.Handlers { + found = found || h.Name == name + } + if b.handlers == nil { + b.handlers = map[string]bool{} + } + b.handlers[name] = found + if !found { + return fmt.Errorf("this Studio Pro offers no %s handler", name) + } + return nil +} + +// executeHandler runs a handler and returns its reply text. A handler that +// throws answers "Error executing tool handler: …", which CallTool already +// classifies as an error. +func (b *Backend) executeHandler(name string, payload map[string]any) (string, error) { + if err := b.requireHandler(name); err != nil { + return "", err + } + res, err := b.client.CallTool("execute_handler", map[string]any{"name": name, "payload": payload}) + if err != nil { + return "", err + } + text := pedStripReminder(res.Text) + if res.IsError { + return "", fmt.Errorf("%s: %s", name, strings.TrimSpace(text)) + } + return text, nil +} + +func parseAppStatus(text string) (*AppStatus, bool) { + var s AppStatus + if err := json.Unmarshal([]byte(strings.TrimSpace(text)), &s); err != nil { + return nil, false + } + return &s, true +} + +// GetAppStatus reports whether the app is running or launching, and its URL, +// without disturbing a running session. +func (b *Backend) GetAppStatus() (*AppStatus, error) { + text, err := b.executeHandler(mendixAppHandler, map[string]any{"action": "status"}) + if err != nil { + return nil, err + } + s, ok := parseAppStatus(text) + if !ok { + return nil, fmt.Errorf("%s status: unexpected reply: %s", mendixAppHandler, strings.TrimSpace(text)) + } + return s, nil +} + +// RunApp starts (or restarts) the app — Studio Pro's Run button — and waits up +// to timeoutSeconds (0 means appRunTimeoutSeconds) for it to become ready. +// +// mendix_app refuses to run a project with model errors and lists them instead. +// Its run reply has not been observed live, so a reply that is not a status is +// settled by looking: the run took iff status then says running or launching. +func (b *Backend) RunApp(timeoutSeconds int) (*AppStatus, error) { + if timeoutSeconds <= 0 { + timeoutSeconds = appRunTimeoutSeconds + } + text, err := b.executeHandler(mendixAppHandler, map[string]any{"action": "run", "timeoutSeconds": timeoutSeconds}) + if err != nil { + return nil, fmt.Errorf("%w (runtime log: read_file %s)", err, appLogPath) + } + if s, ok := parseAppStatus(text); ok { + return s, nil + } + if s, serr := b.GetAppStatus(); serr == nil && (s.Running || s.Launching) { + return s, nil + } + return nil, fmt.Errorf("%s run: %s (runtime log: read_file %s)", mendixAppHandler, strings.TrimSpace(text), appLogPath) +} + +// StopApp stops the running app — Studio Pro's Stop button. +func (b *Backend) StopApp() error { + _, err := b.executeHandler(mendixAppHandler, map[string]any{"action": "stop"}) + return err +} + +// writeAppStatus prints one line on where the app stands after a run. +func writeAppStatus(w io.Writer, s *AppStatus) { + switch { + case s.Running: + fmt.Fprintf(w, "app running at %s\n", s.AppURL) + case s.Launching: + fmt.Fprintf(w, "app still launching at %s (check again with mendix_app status)\n", s.AppURL) + default: + fmt.Fprintf(w, "app is not running (runtime log: read_file %s)\n", appLogPath) + } +} diff --git a/mdl/backend/mcp/app_test.go b/mdl/backend/mcp/app_test.go new file mode 100644 index 000000000..d27692b62 --- /dev/null +++ b/mdl/backend/mcp/app_test.go @@ -0,0 +1,202 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "strings" + "testing" +) + +// The 11.16 surface that carries handlers; 11.12 advertises neither tool. +var handlerTools = map[string][]string{ + "query_handlers": {"operation", "keyword", "names"}, + "execute_handler": {"name", "payload"}, +} + +// mendixAppDetails is the shape of 11.16.0's getDetails reply (handlers-detail +// probe), trimmed to what requireHandler reads. +const mendixAppDetails = `{"handlers":[{"name":"mendix_app","description":"Controls the Mendix application runtime."}],"errors":[]}` + +// fakeMendixApp answers query_handlers and execute_handler mendix_app with the +// given reply per action. +func fakeMendixApp(t *testing.T, replies map[string]string) *fakePED { + f := newFakePED(t, func(name string, args map[string]any) (string, bool) { + switch name { + case "query_handlers": + return mendixAppDetails, false + case "execute_handler": + if args["name"] != "mendix_app" { + return "Error executing tool handler: Handler '" + args["name"].(string) + "' not found.", false + } + payload, _ := args["payload"].(map[string]any) + action, _ := payload["action"].(string) + if r, ok := replies[action]; ok { + return r, false + } + } + return "unexpected " + name, true + }) + f.tools = handlerTools + return f +} + +func TestGetAppStatus_ParsesMeasuredReply(t *testing.T) { + f := fakeMendixApp(t, map[string]string{ + "status": `{"running":false,"launching":false,"appUrl":"http://localhost:8080/"}`, + }) + b := &Backend{client: f.connectClient(t)} + + s, err := b.GetAppStatus() + if err != nil { + t.Fatalf("GetAppStatus: %v", err) + } + if s.Running || s.Launching || s.AppURL != "http://localhost:8080/" { + t.Errorf("status = %+v", s) + } + q, ok := f.callByName("query_handlers") + if !ok || q.Args["operation"] != "getDetails" { + t.Fatalf("handler presence must be asked with getDetails first, got %+v", q) + } + call, _ := f.callByName("execute_handler") + if payload, _ := call.Args["payload"].(map[string]any); payload["action"] != "status" { + t.Errorf("payload = %v, want action status", call.Args["payload"]) + } +} + +func TestRunApp_SendsTimeoutAndReportsURL(t *testing.T) { + f := fakeMendixApp(t, map[string]string{ + "run": `{"running":true,"launching":false,"appUrl":"http://localhost:8080/"}`, + }) + b := &Backend{client: f.connectClient(t)} + + s, err := b.RunApp(0) + if err != nil { + t.Fatalf("RunApp: %v", err) + } + if !s.Running { + t.Errorf("status = %+v, want running", s) + } + call, _ := f.callByName("execute_handler") + payload, _ := call.Args["payload"].(map[string]any) + if payload["action"] != "run" || payload["timeoutSeconds"] != float64(appRunTimeoutSeconds) { + t.Errorf("payload = %v, want run with timeoutSeconds %d", payload, appRunTimeoutSeconds) + } + var sb strings.Builder + writeAppStatus(&sb, s) + if sb.String() != "app running at http://localhost:8080/\n" { + t.Errorf("report = %q", sb.String()) + } +} + +// mendix_app refuses to run a project with model errors and lists them instead; +// that text is not a status, and status then shows nothing running. +func TestRunApp_RefusalIsAnError(t *testing.T) { + f := fakeMendixApp(t, map[string]string{ + "run": "The app cannot be run: the project has 2 errors.", + "status": `{"running":false,"launching":false,"appUrl":"http://localhost:8080/"}`, + }) + b := &Backend{client: f.connectClient(t)} + + _, err := b.RunApp(0) + if err == nil || !strings.Contains(err.Error(), "the project has 2 errors") || !strings.Contains(err.Error(), appLogPath) { + t.Fatalf("a refused run should carry the reply and point at the log, got: %v", err) + } +} + +// A run reply that is not a status but did start the app is settled by status. +func TestRunApp_UnrecognisedReplySettledByStatus(t *testing.T) { + f := fakeMendixApp(t, map[string]string{ + "run": "Started.", + "status": `{"running":false,"launching":true,"appUrl":"http://localhost:8080/"}`, + }) + b := &Backend{client: f.connectClient(t)} + + s, err := b.RunApp(0) + if err != nil { + t.Fatalf("RunApp: %v", err) + } + if !s.Launching { + t.Errorf("status = %+v, want launching", s) + } +} + +func TestStopApp_HandlerFailureSurfaces(t *testing.T) { + f := fakeMendixApp(t, map[string]string{"stop": handlerSessionError}) + b := &Backend{client: f.connectClient(t)} + + if err := b.StopApp(); err == nil || !strings.Contains(err.Error(), "Session 'mcp-server' not found") { + t.Fatalf("a handler that throws must be an error, got: %v", err) + } +} + +func TestAppHandler_AbsentOn1112(t *testing.T) { + f := newFakePED(t, nil) + f.tools = map[string][]string{"ped_create_document": {"documents"}} // 11.12 surface + b := &Backend{client: f.connectClient(t)} + + if _, err := b.RunApp(0); err == nil || !strings.Contains(err.Error(), "needs Studio Pro 11.16+") { + t.Fatalf("without execute_handler RunApp should name the version, got: %v", err) + } + if len(f.calls) != 0 { + t.Errorf("no tool may be called on a server without handlers, got %+v", f.calls) + } +} + +func TestAppHandler_NotOffered(t *testing.T) { + f := newFakePED(t, func(name string, _ map[string]any) (string, bool) { + return `{"handlers":[],"errors":[]}`, false + }) + f.tools = handlerTools + b := &Backend{client: f.connectClient(t)} + + if _, err := b.GetAppStatus(); err == nil || !strings.Contains(err.Error(), "no mendix_app handler") { + t.Fatalf("got: %v", err) + } + if _, ok := f.callByName("execute_handler"); ok { + t.Error("execute_handler must not be called for a handler query_handlers did not list") + } + // Cached: a second call asks nothing more. + n := len(f.calls) + _, _ = b.GetAppStatus() + if len(f.calls) != n { + t.Errorf("handler presence should be cached for the session") + } +} + +func TestCheckProject(t *testing.T) { + listing := "Listing problems 1-2 (out of 2). Check ID: 6\n" + + "'MyFirstModule.Zz_BrokenEnum' (Enumerations$Enumeration):\n" + + "- [error] Missing enumeration values. (at locations: -)\n" + + "'MyFirstModule' (DomainModels$DomainModel):\n" + + "- [error] Duplicate name\n" + f := newFakePED(t, func(string, map[string]any) (string, bool) { return listing, false }) + f.tools = map[string][]string{"ped_check_errors": {"filters", "pagination"}} + b := &Backend{client: f.connectClient(t)} + + got, err := b.CheckProject() + if err != nil { + t.Fatalf("CheckProject: %v", err) + } + if len(got) != 4 || got[0] != "'MyFirstModule.Zz_BrokenEnum' (Enumerations$Enumeration):" { + t.Fatalf("listing = %q", got) + } + call, _ := f.callByName("ped_check_errors") + if len(call.Args) != 0 { + t.Errorf("a project-wide check sends no filters, got %v", call.Args) + } + var sb strings.Builder + writeCheckReport(&sb, got) + if !strings.Contains(sb.String(), " - [error] Duplicate name") { + t.Errorf("report:\n%s", sb.String()) + } +} + +func TestCheckProject_NeedsFilters(t *testing.T) { + f := newFakePED(t, nil) + f.tools = map[string][]string{"ped_check_errors": {"documents"}} // up to 11.14 + b := &Backend{client: f.connectClient(t)} + + if _, err := b.CheckProject(); err == nil || !strings.Contains(err.Error(), "11.15+") { + t.Fatalf("got: %v", err) + } +} diff --git a/mdl/backend/mcp/backend.go b/mdl/backend/mcp/backend.go index 2964535b7..819c2497f 100644 --- a/mdl/backend/mcp/backend.go +++ b/mdl/backend/mcp/backend.go @@ -45,20 +45,13 @@ type Backend struct { path string server ServerInfo - // concord is an optional second MCP client to the Concord extension server, - // used ONLY for capabilities the built-in PED server lacks (delete, save, - // validate, run). nil unless configured via WithConcord. PED stays the - // authoring path; Concord is the gap-filler. - concordURL string - concordDial string - concord *Client - // On-Disconnect Concord actions (the gap-fillers PED lacks). saveOnExit flushes - // via save_all (--mcp-save), checkOnExit validates via check_model - // (--mcp-check), runOnExit starts the app via run_app and prints its URL - // (--mcp-run). - saveOnExit bool + // On-Disconnect actions. checkOnExit runs a project-wide ped_check_errors + // (--mcp-check); runOnExit starts the app through the mendix_app handler and + // prints its URL (--mcp-run). checkOnExit bool runOnExit bool + // handlers caches query_handlers getDetails answers by handler name. + handlers map[string]bool // schemaFetched records element types already fetched via ped_get_schema // this session (the contract asks for a schema fetch before create/add). @@ -173,14 +166,10 @@ func New(mcpURL, dial string) *Backend { } } -// ConcordConfig configures the optional second MCP client to the Concord -// extension server (gap-filler for capabilities PED lacks) and which of its -// on-Disconnect actions to run. -type ConcordConfig struct { - URL, Dial string - SaveOnExit bool // run save_all (--mcp-save) - CheckOnExit bool // run check_model (--mcp-check) - RunOnExit bool // run run_app (--mcp-run) +// ExitActions selects what Disconnect runs after the command's writes. +type ExitActions struct { + Check bool // project-wide ped_check_errors (--mcp-check) + Run bool // execute_handler mendix_app run (--mcp-run) } // WithTracer attaches a tool-call tracer (--mcp-verbose / --mcp-trace). The same @@ -191,13 +180,10 @@ func (b *Backend) WithTracer(t *backend.Tracer) *Backend { return b } -// WithConcord enables the Concord client. Returns the receiver for chaining. -func (b *Backend) WithConcord(cfg ConcordConfig) *Backend { - b.concordURL = cfg.URL - b.concordDial = cfg.Dial - b.saveOnExit = cfg.SaveOnExit - b.checkOnExit = cfg.CheckOnExit - b.runOnExit = cfg.RunOnExit +// WithExitActions sets what Disconnect runs. Returns the receiver for chaining. +func (b *Backend) WithExitActions(a ExitActions) *Backend { + b.checkOnExit = a.Check + b.runOnExit = a.Run return b } @@ -229,19 +215,6 @@ func (b *Backend) Connect(path string) error { b.server = si b.path = path - // Optional Concord gap-filler client. A failure here is fatal only because the - // user explicitly asked for it (--mcp-concord); PED authoring already succeeded. - if b.concordURL != "" { - cc, err := NewClient(ClientOptions{URL: b.concordURL, Dial: b.concordDial}) - if err != nil { - return fmt.Errorf("create Concord client: %w", err) - } - cc.trace = b.tracer - if _, err := cc.Initialize(); err != nil { - return fmt.Errorf("connect to Concord MCP server %q: %w", b.concordURL, err) - } - b.concord = cc - } return nil } @@ -253,37 +226,31 @@ func (b *Backend) ServerInfo() ServerInfo { return b.server } // --------------------------------------------------------------------------- func (b *Backend) Disconnect() error { - // Persist PED's in-memory writes via Concord before tearing down (PED has no - // save tool). A save failure must not be silent — surface it on stderr — but - // it does not block teardown (the writes still live in Studio Pro's memory). - if b.saveOnExit && b.concord != nil { - if err := b.SaveAll(); err != nil { - fmt.Fprintf(os.Stderr, "warning: --mcp-save failed, changes remain unsaved in Studio Pro: %v\n", err) - } - } - // Validate the (in-memory) model after writes if requested. Diagnostic, so it - // prints to stderr and never blocks teardown. - if b.checkOnExit && b.concord != nil { - if r, err := b.CheckModel(""); err != nil { + // PED persists every write to disk as it lands (measured on 11.12.0 and + // 11.16.0), so there is nothing to save here. + // + // Validate the model after writes if requested. Diagnostic, so it prints to + // stderr and never blocks teardown. + if b.checkOnExit && b.client != nil { + if problems, err := b.CheckProject(); err != nil { fmt.Fprintf(os.Stderr, "warning: --mcp-check failed: %v\n", err) } else { - writeCheckReport(os.Stderr, r) + writeCheckReport(os.Stderr, problems) } } // Start the app (build + deploy current model) and report its URL, for a // change-then-run loop. - if b.runOnExit && b.concord != nil { - if err := b.RunApp(); err != nil { + if b.runOnExit && b.client != nil { + if s, err := b.RunApp(0); err != nil { fmt.Fprintf(os.Stderr, "warning: --mcp-run failed: %v\n", err) - } else if s, err := b.GetAppStatus(); err == nil && s.Data.RunningURL != "" { - fmt.Fprintf(os.Stderr, "app running at %s\n", s.Data.RunningURL) + } else { + writeAppStatus(os.Stderr, s) } } if b.reader != nil { err := b.reader.Disconnect() b.reader = nil b.client = nil - b.concord = nil b.path = "" return err } diff --git a/mdl/backend/mcp/businessevent.go b/mdl/backend/mcp/businessevent.go index cb8324d69..33e83ffab 100644 --- a/mdl/backend/mcp/businessevent.go +++ b/mdl/backend/mcp/businessevent.go @@ -12,8 +12,8 @@ import ( // the document type outright ("Document type 'BusinessEvents$BusinessEventService' // cannot be created.") — it is off the create whitelist (and the schema is a // $element, not a $constructor). So CREATE / CREATE OR MODIFY are rejected; reads -// (SHOW / DESCRIBE) delegate to the local .mpr; DROP of an existing service goes -// through Concord, like other standalone documents. +// (SHOW / DESCRIBE) delegate to the local .mpr; DROP of an existing service uses +// ped_delete_document (11.16+), like other standalone documents. // // The domain model a business-events setup relies on — a published-event entity // (`extends BusinessEvents.PublishedBusinessEvent`, generalization is supported by @@ -43,8 +43,8 @@ func (b *Backend) UpdateBusinessEventService(svc *model.BusinessEventService) er return errCreatePathUnbuilt("business event service", svc.Name) } -// DeleteBusinessEventService drops an existing service via Concord's delete_document -// (PED has no delete tool), like other standalone documents. Requires --mcp-concord. +// DeleteBusinessEventService drops an existing service via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeleteBusinessEventService(id model.ID) error { services, err := b.reader.ListBusinessEventServices() if err != nil { @@ -56,7 +56,7 @@ func (b *Backend) DeleteBusinessEventService(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for business event service %q: %w", s.Name, err) } - return b.concordDeleteDocument(modName, s.Name) + return b.deleteDocument(businessEventServiceDocType, modName, s.Name) } } return fmt.Errorf("business event service %s not found", id) diff --git a/mdl/backend/mcp/capabilities.go b/mdl/backend/mcp/capabilities.go index df00c2805..f069f2efb 100644 --- a/mdl/backend/mcp/capabilities.go +++ b/mdl/backend/mcp/capabilities.go @@ -5,6 +5,7 @@ package mcp import ( _ "embed" "fmt" + "slices" "sort" "strconv" "strings" @@ -51,10 +52,9 @@ type Capability struct { // the user's Studio Pro preferences and configured MCP servers, not just with // versions (ADR-0006 Revision). type Capabilities struct { - ServerName string - ServerVersion string - ProjectVersion string - ConcordConnected bool + ServerName string + ServerVersion string + ProjectVersion string // Tools are the Studio Pro tools present, from the live probe. Tools []string // FederatedTools are tools Studio Pro proxies from MCP servers the user has @@ -183,7 +183,7 @@ func errCreatePathUnbuilt(kind, name string) error { // capabilities builds the effective capability set: the table resolved against // the project's Mendix version, narrowed by the live tool probe, plus live -// identity/Concord. +// identity. // // Cached for the session: every authoring gate calls this, and the probe is a // network round-trip. tools.listChanged means the surface *can* move mid-session, @@ -194,9 +194,8 @@ func (b *Backend) capabilities() Capabilities { return *b.capsCache } caps := Capabilities{ - ServerName: b.server.Name, - ServerVersion: b.server.Version, - ConcordConnected: b.concord != nil, + ServerName: b.server.Name, + ServerVersion: b.server.Version, } var pv *types.ProjectVersion if b.reader != nil { @@ -241,11 +240,21 @@ func (b *Backend) CapabilityReport() string { sb.WriteString("========================\n") fmt.Fprintf(&sb, "Studio Pro MCP server : %s %s\n", orUnknown(caps.ServerName), orUnknown(caps.ServerVersion)) fmt.Fprintf(&sb, "Project Mendix version: %s\n", orUnknown(caps.ProjectVersion)) - concord := "not connected — DROP of standalone docs (enum/microflow/page/…) is unavailable" - if caps.ConcordConnected { - concord = "connected" + drop := "unavailable — needs Studio Pro 11.16+ (ped_delete_document)" + if slices.Contains(caps.Tools, "ped_delete_document") { + drop = "ped_delete_document" } - fmt.Fprintf(&sb, "Concord gap-filler : %s\n\n", concord) + fmt.Fprintf(&sb, "DROP of documents : %s\n", drop) + check := "unavailable — needs Studio Pro 11.15+ (ped_check_errors filters)" + if b.client != nil && b.client.SupportsToolArg("ped_check_errors", "filters") { + check = "ped_check_errors, project-wide" + } + fmt.Fprintf(&sb, "--mcp-check : %s\n", check) + run := "unavailable — needs Studio Pro 11.16+ (execute_handler mendix_app)" + if slices.Contains(caps.Tools, "execute_handler") { + run = "execute_handler mendix_app" + } + fmt.Fprintf(&sb, "--mcp-run : %s\n\n", run) sb.WriteString("Authorable over MCP:\n") for _, c := range caps.Features { diff --git a/mdl/backend/mcp/capabilities.yaml b/mdl/backend/mcp/capabilities.yaml index 959a530a8..1542273b9 100644 --- a/mdl/backend/mcp/capabilities.yaml +++ b/mdl/backend/mcp/capabilities.yaml @@ -51,7 +51,7 @@ features: feature: "Enumerations" available: true requires_tools: [ped_create_document] - note: "CREATE (DROP via Concord)" + note: "CREATE / DROP (DROP needs Studio Pro 11.16+, ped_delete_document)" - key: constants feature: "Constants" available: true diff --git a/mdl/backend/mcp/check_errors.go b/mdl/backend/mcp/check_errors.go index 385923189..17d22264a 100644 --- a/mdl/backend/mcp/check_errors.go +++ b/mdl/backend/mcp/check_errors.go @@ -4,6 +4,7 @@ package mcp import ( "fmt" + "io" "regexp" "strconv" "strings" @@ -68,7 +69,9 @@ func (b *Backend) checkDocumentNow(docType, docName string) error { } for attempt := 0; ; attempt++ { - problems, stale, err := b.collectCheckErrors(base, docType, docName) + problems, stale, err := b.collectCheckErrors(base, docName, func(text string) []string { + return scopeCheckListing(text, docType, docName) + }) if err != nil { return err } @@ -86,9 +89,10 @@ func (b *Backend) checkDocumentNow(docType, docName string) error { } // collectCheckErrors runs one complete check (first response plus every page) and -// returns the problems listed for the document itself. stale reports that a page -// window went stale and the whole check must be asked again. -func (b *Backend) collectCheckErrors(base map[string]any, docType, docName string) (problems []string, stale bool, err error) { +// returns the problem lines scope keeps from each listing page. docName names what +// was checked in errors. stale reports that a page window went stale and the +// whole check must be asked again. +func (b *Backend) collectCheckErrors(base map[string]any, docName string, scope func(string) []string) (problems []string, stale bool, err error) { res, err := b.client.CallTool("ped_check_errors", base) if err != nil { return nil, false, err @@ -111,7 +115,7 @@ func (b *Backend) collectCheckErrors(base map[string]any, docType, docName strin to, _ := strconv.Atoi(m[2]) total, _ := strconv.Atoi(m[3]) checkID, _ := strconv.Atoi(m[4]) - problems = scopeCheckListing(trimmed, docType, docName) + problems = scope(trimmed) for offset := to; offset < total; { args := make(map[string]any, len(base)+1) @@ -137,7 +141,7 @@ func (b *Backend) collectCheckErrors(base map[string]any, docType, docName strin break } next, _ := strconv.Atoi(pm[2]) - problems = append(problems, scopeCheckListing(page, docType, docName)...) + problems = append(problems, scope(page)...) if next <= offset { break // no progress; never loop } @@ -171,3 +175,50 @@ func scopeCheckListing(text, docType, docName string) []string { } return out } + +// CheckProject runs ped_check_errors over the whole project (--mcp-check) and +// returns its listing: each unit header followed by that unit's problems. Only +// the 11.15+ shape can ask project-wide — the older `documents` argument is +// required and names the documents to check. +func (b *Backend) CheckProject() ([]string, error) { + if b.client == nil || !b.client.SupportsToolArg("ped_check_errors", "filters") { + return nil, fmt.Errorf("a project-wide check needs Studio Pro 11.15+ (ped_check_errors filters)") + } + for attempt := 0; ; attempt++ { + problems, stale, err := b.collectCheckErrors(map[string]any{}, "the project", projectCheckListing) + if err != nil { + return nil, err + } + if !stale { + return problems, nil + } + if attempt >= checkErrorsMaxRefetch { + return nil, fmt.Errorf("the project check did not settle: Studio Pro's error list kept changing between pages") + } + } +} + +// projectCheckListing keeps every line of an 11.15 listing but its page header. +func projectCheckListing(text string) []string { + var out []string + for _, line := range strings.Split(text, "\n") { + line = strings.TrimRight(line, "\r") + if checkListingHeader.MatchString(line) || strings.TrimSpace(line) == "" { + continue + } + out = append(out, line) + } + return out +} + +// writeCheckReport prints a project check's listing, or that it is clean. +func writeCheckReport(w io.Writer, listing []string) { + if len(listing) == 0 { + fmt.Fprintln(w, "model check: no errors found") + return + } + fmt.Fprintln(w, "model check:") + for _, line := range listing { + fmt.Fprintf(w, " %s\n", line) + } +} diff --git a/mdl/backend/mcp/client.go b/mdl/backend/mcp/client.go index 5e8d345d7..5385d9c0b 100644 --- a/mdl/backend/mcp/client.go +++ b/mdl/backend/mcp/client.go @@ -288,6 +288,19 @@ func (c *Client) SupportsToolArg(tool, arg string) bool { return c.toolArgs[tool][arg] } +// HasTool reports whether the connected server advertises the named tool, from +// the same cached tools/list probe as SupportsToolArg (false when the probe fails). +func (c *Client) HasTool(tool string) bool { + if c.toolArgs == nil { + if _, err := c.ListTools(); err != nil { + c.toolArgs = map[string]map[string]bool{} + return false + } + } + _, ok := c.toolArgs[tool] + return ok +} + func (c *Client) CallTool(name string, arguments any) (*ToolResult, error) { if c.trace.Enabled() { target, detail := summarizeToolCall(name, arguments) @@ -314,7 +327,18 @@ func (c *Client) CallTool(name string, arguments any) (*ToolResult, error) { } sb.WriteString(blk.Text) } - return &ToolResult{IsError: tc.IsError, Text: sb.String()}, nil + text := sb.String() + return &ToolResult{IsError: tc.IsError || handlerFailed(text), Text: text}, nil +} + +// handlerFailedPrefix opens the text Studio Pro returns when a tool's handler +// throws. It arrives with isError:false (measured on 11.16.0, ped_delete_document: +// "Error executing tool handler: Session 'mcp-server' not found"), so a caller +// that trusts the flag would read a failed call as a success. +const handlerFailedPrefix = "Error executing tool handler:" + +func handlerFailed(text string) bool { + return strings.HasPrefix(strings.TrimSpace(text), handlerFailedPrefix) } // call sends a JSON-RPC request and returns the matching response, handling diff --git a/mdl/backend/mcp/concord.go b/mdl/backend/mcp/concord.go deleted file mode 100644 index af1f0ad32..000000000 --- a/mdl/backend/mcp/concord.go +++ /dev/null @@ -1,178 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package mcp - -import ( - "encoding/json" - "fmt" - "io" - "strings" - - "github.com/mendixlabs/mxcli/model" -) - -// Concord is a Studio Pro extension whose MCP server provides capabilities the -// built-in PED server lacks (delete, save, validate, run). The MCP backend uses -// PED for authoring by default and reaches for Concord ONLY for these gaps. The -// Concord client is optional (nil unless configured via WithConcord); every -// Concord-backed operation errors clearly when it is not configured. - -// concordCall invokes a Concord tool, returning the result text. It errors if -// Concord is not configured or the tool reports failure. Concord tools report -// failure either via the MCP isError flag or a JSON body with "error"/"success": -// false in the result text. -func (b *Backend) concordCall(tool string, args map[string]any) (string, error) { - if b.concord == nil { - return "", fmt.Errorf("%s requires the Concord MCP server — pass --mcp-concord (Concord provides capabilities the built-in server lacks)", tool) - } - res, err := b.concord.CallTool(tool, args) - if err != nil { - return "", err - } - text := pedStripReminder(res.Text) - if res.IsError || concordFailed(text) { - return "", fmt.Errorf("%s: %s", tool, text) - } - return text, nil -} - -// concordFailed reports whether a Concord result body signals failure. Concord -// returns JSON like {"error":"..."} or {"success":false,...} on failure and -// {"success":true,...}/{"status":"ok",...} on success. -func concordFailed(text string) bool { - t := strings.TrimSpace(text) - return strings.Contains(t, `"error"`) || strings.Contains(t, `"success":false`) || strings.Contains(t, `"success": false`) -} - -// SaveAll persists every unsaved change in Studio Pro (Concord save_all, the -// equivalent of Ctrl+S). PED has no save tool, so this is the only way to flush -// PED-authored in-memory writes to disk from mxcli. -func (b *Backend) SaveAll() error { - _, err := b.concordCall("save_all", map[string]any{}) - return err -} - -// concordDeleteDocument removes a standalone document (enumeration, microflow, -// page) via Concord's delete_document. PED has no delete tool, so DROP of these -// document types is only possible through Concord. -func (b *Backend) concordDeleteDocument(moduleName, docName string) error { - _, err := b.concordCall("delete_document", map[string]any{ - "module_name": moduleName, - "document_name": docName, - }) - return err -} - -// CheckItem is one domain-model consistency finding from check_model. -type CheckItem struct { - Module string `json:"module"` - Entity string `json:"entity"` - Code string `json:"code"` - Message string `json:"message"` -} - -// CheckResult is the parsed result of Concord's check_model. -type CheckResult struct { - Success bool `json:"success"` - Healthy bool `json:"healthy"` // true == zero errors (NOT zero warnings) - Summary struct { - TotalItems int `json:"totalItems"` - ErrorCount int `json:"errorCount"` - WarningCount int `json:"warningCount"` - InfoCount int `json:"infoCount"` - } `json:"summary"` - Errors []CheckItem `json:"errors"` - Warnings []CheckItem `json:"warnings"` -} - -// CheckModel runs Concord's domain-model consistency checker, optionally scoped to -// one module (""=whole project). A non-empty Errors slice is a *result*, not a -// tool failure, so this bypasses the flat success/error heuristic and parses the -// structured body directly. -func (b *Backend) CheckModel(moduleName string) (*CheckResult, error) { - if b.concord == nil { - return nil, fmt.Errorf("check_model requires the Concord MCP server — pass --mcp-concord") - } - args := map[string]any{} - if moduleName != "" { - args["module"] = moduleName - } - res, err := b.concord.CallTool("check_model", args) - if err != nil { - return nil, err - } - text := pedStripReminder(res.Text) - if res.IsError { - return nil, fmt.Errorf("check_model: %s", text) - } - var r CheckResult - if err := json.Unmarshal([]byte(text), &r); err != nil { - return nil, fmt.Errorf("check_model: parsing result: %w", err) - } - return &r, nil -} - -// writeCheckReport prints a concise consistency report. Errors and warnings are -// both shown — "healthy" (zero errors) does NOT mean zero warnings. -func writeCheckReport(w io.Writer, r *CheckResult) { - fmt.Fprintf(w, "model check: %d error(s), %d warning(s)\n", r.Summary.ErrorCount, r.Summary.WarningCount) - for _, it := range r.Errors { - fmt.Fprintf(w, " ERROR %s.%s [%s]: %s\n", it.Module, it.Entity, it.Code, it.Message) - } - for _, it := range r.Warnings { - fmt.Fprintf(w, " warn %s.%s [%s]: %s\n", it.Module, it.Entity, it.Code, it.Message) - } -} - -// AppStatus is the parsed result of Concord's get_app_status. -type AppStatus struct { - Status string `json:"status"` - Data struct { - ProjectPath string `json:"projectPath"` - ProjectName string `json:"projectName"` - Running string `json:"running"` - RunningURL string `json:"runningUrl"` - } `json:"data"` -} - -// GetAppStatus returns whether the loaded app is running and its runtime URL -// (Concord get_app_status — read-only, the "what state is the project in" probe). -func (b *Backend) GetAppStatus() (*AppStatus, error) { - if b.concord == nil { - return nil, fmt.Errorf("get_app_status requires the Concord MCP server — pass --mcp-concord") - } - res, err := b.concord.CallTool("get_app_status", map[string]any{}) - if err != nil { - return nil, err - } - text := pedStripReminder(res.Text) - if res.IsError { - return nil, fmt.Errorf("get_app_status: %s", text) - } - var s AppStatus - if err := json.Unmarshal([]byte(text), &s); err != nil { - return nil, fmt.Errorf("get_app_status: parsing result: %w", err) - } - return &s, nil -} - -// RunApp starts the loaded Mendix app on its run profile (Concord run_app, -// equivalent to the green Run button — builds + deploys the current model). -func (b *Backend) RunApp() error { - _, err := b.concordCall("run_app", map[string]any{}) - return err -} - -// StopApp stops the running Mendix runtime (Concord stop_app, the red Stop button). -func (b *Backend) StopApp() error { - _, err := b.concordCall("stop_app", map[string]any{}) - return err -} - -// moduleNameForContainer resolves a container (module) ID to its module name, -// session-aware so freshly created modules resolve too. -func (b *Backend) moduleNameForContainer(containerID model.ID) (string, error) { - // Folder-aware: a document's container may be a folder, not the module directly. - name, _, err := b.resolveDocContainer(containerID) - return name, err -} diff --git a/mdl/backend/mcp/concord_test.go b/mdl/backend/mcp/concord_test.go deleted file mode 100644 index a39b02f64..000000000 --- a/mdl/backend/mcp/concord_test.go +++ /dev/null @@ -1,62 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package mcp - -import ( - "strings" - "testing" -) - -func TestConcordFailed(t *testing.T) { - cases := map[string]bool{ - `{"success":true,"message":"saved"}`: false, - `{"status":"ok","data":{}}`: false, - `{"error":"not allowed"}`: true, - `{"success":false,"message":"nope"}`: true, - `{"success": false }`: true, - `Module 'X' created successfully.`: false, - } - for text, want := range cases { - if got := concordFailed(text); got != want { - t.Errorf("concordFailed(%q) = %v, want %v", text, got, want) - } - } -} - -func TestConcordCall_NotConfigured(t *testing.T) { - b := &Backend{} // no Concord client - if err := b.SaveAll(); err == nil { - t.Fatal("SaveAll without Concord should error") - } - if _, err := b.concordCall("save_all", nil); err == nil { - t.Fatal("concordCall without Concord should error with an actionable message") - } - if _, err := b.CheckModel(""); err == nil { - t.Fatal("CheckModel without Concord should error") - } - if _, err := b.GetAppStatus(); err == nil { - t.Fatal("GetAppStatus without Concord should error") - } - if err := b.RunApp(); err == nil { - t.Fatal("RunApp without Concord should error") - } - if err := b.StopApp(); err == nil { - t.Fatal("StopApp without Concord should error") - } -} - -func TestWriteCheckReport(t *testing.T) { - r := &CheckResult{Healthy: true} - r.Summary.ErrorCount = 1 - r.Summary.WarningCount = 1 - r.Errors = []CheckItem{{Module: "M", Entity: "E", Code: "bad_ref", Message: "broken"}} - r.Warnings = []CheckItem{{Module: "M", Entity: "W", Code: "no_attributes", Message: "empty"}} - var sb strings.Builder - writeCheckReport(&sb, r) - out := sb.String() - if !strings.Contains(out, "1 error(s), 1 warning(s)") || - !strings.Contains(out, "ERROR M.E [bad_ref]: broken") || - !strings.Contains(out, "warn M.W [no_attributes]: empty") { - t.Fatalf("report:\n%s", out) - } -} diff --git a/mdl/backend/mcp/constant.go b/mdl/backend/mcp/constant.go index 870384f92..eaa860b18 100644 --- a/mdl/backend/mcp/constant.go +++ b/mdl/backend/mcp/constant.go @@ -102,8 +102,8 @@ func (b *Backend) liveConstantType(qn string) (string, error) { return k, nil } -// DeleteConstant drops a constant via Concord's delete_document (PED has no delete -// tool). Requires --mcp-concord; errors clearly otherwise. +// DeleteConstant drops a constant via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeleteConstant(id model.ID) error { c, err := b.GetConstant(id) if err != nil { @@ -113,7 +113,7 @@ func (b *Backend) DeleteConstant(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for constant %q: %w", c.Name, err) } - if err := b.concordDeleteDocument(modName, c.Name); err != nil { + if err := b.deleteDocument(constantDocType, modName, c.Name); err != nil { return err } b.forgetSessionConstant(modName, c.Name) diff --git a/mdl/backend/mcp/delete.go b/mdl/backend/mcp/delete.go new file mode 100644 index 000000000..c1eb35650 --- /dev/null +++ b/mdl/backend/mcp/delete.go @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "fmt" + "strings" + "time" + + "github.com/mendixlabs/mxcli/model" +) + +// The document types DROP addresses that no create path names. +const ( + nanoflowDocType = "Microflows$Nanoflow" + pageDocType = "Pages$Page" + businessEventServiceDocType = "BusinessEvents$BusinessEventService" +) + +// deleteDocument drops a standalone named document (constant, enumeration, +// microflow, nanoflow, page, workflow, business-event service). +// +// Studio Pro 11.16 added ped_delete_document; older servers have no delete tool, +// so DROP over --mcp needs 11.16+. On 11.16.0 the tool fails for every call made +// outside Studio Pro's own agent session ("Session 'mcp-server' not found"), +// measured live — that failure is surfaced, not retried elsewhere. +func (b *Backend) deleteDocument(docType, moduleName, docName string) error { + if b.client == nil || !b.client.HasTool("ped_delete_document") { + return fmt.Errorf("DROP of %s.%s needs Studio Pro 11.16+ (ped_delete_document); this Studio Pro's MCP server has no delete tool — delete it in Studio Pro", moduleName, docName) + } + return b.pedDeleteDocument(docType, moduleName, docName) +} + +// moduleNameForContainer resolves a container (module) ID to its module name, +// session-aware so freshly created modules resolve too. +func (b *Backend) moduleNameForContainer(containerID model.ID) (string, error) { + // Folder-aware: a document's container may be a folder, not the module directly. + name, _, err := b.resolveDocContainer(containerID) + return name, err +} + +// pedDeleteDocument calls ped_delete_document for one document. Its success +// reply has not been observed live, so a reply that is neither PED's usual +// "SUCCESS…" nor an error is settled by looking: the delete happened iff the +// document is gone. A timeout is settled the same way, as for creates. +func (b *Backend) pedDeleteDocument(docType, moduleName, docName string) error { + target := moduleName + "." + docName + res, err := b.client.CallTool("ped_delete_document", map[string]any{ + "documents": []map[string]any{{ + "documentType": docType, + "moduleName": moduleName, + "documentName": docName, + }}, + }) + if isTimeoutErr(err) { + time.Sleep(timeoutVerifyDelay) + if exists, verr := b.pedDocumentExists(moduleName, docType, docName); verr == nil && !exists { + timeoutNotice(fmt.Sprintf("%s %s", docType, target)) + return nil + } + return timeoutUnverified(err, fmt.Sprintf("%s %s", docType, target)) + } + if err != nil { + return err + } + opErr := pedOpError("ped_delete_document", target, res) + if opErr == nil || res.IsError || pedReportsFailure(res.Text) { + return opErr + } + exists, verr := b.pedDocumentExists(moduleName, docType, docName) + if verr != nil { + return fmt.Errorf("%w (and confirming the delete failed: %v)", opErr, verr) + } + if exists { + return opErr + } + return nil +} + +// pedReportsFailure recognises PED's failure texts, which arrive with +// isError:false ("Deleting documents failed (1 of 1):\nERROR: …"). +func pedReportsFailure(text string) bool { + return strings.Contains(text, "failed (") || strings.Contains(text, "\nERROR:") || strings.HasPrefix(strings.TrimSpace(text), "ERROR") +} diff --git a/mdl/backend/mcp/delete_test.go b/mdl/backend/mcp/delete_test.go new file mode 100644 index 000000000..e59900f23 --- /dev/null +++ b/mdl/backend/mcp/delete_test.go @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "encoding/json" + "strings" + "testing" +) + +func TestDeleteDocument_NativeWhenAdvertised(t *testing.T) { + f := newFakePED(t, func(name string, _ map[string]any) (string, bool) { + if name == "ped_delete_document" { + return "SUCCESS: Deleting documents (1)", false + } + return "unexpected", true + }) + f.tools = map[string][]string{"ped_delete_document": {"documents"}} + b := &Backend{client: f.connectClient(t)} + + if err := b.deleteDocument(constantDocType, "MyFirstModule", "Gone"); err != nil { + t.Fatalf("deleteDocument: %v", err) + } + call, ok := f.callByName("ped_delete_document") + if !ok { + t.Fatal("ped_delete_document was not called") + } + raw, _ := json.Marshal(call.Args["documents"]) + want := `[{"documentName":"Gone","documentType":"Constants$Constant","moduleName":"MyFirstModule"}]` + if string(raw) != want { + t.Errorf("documents = %s, want %s", raw, want) + } +} + +// The success text of ped_delete_document has not been observed live (the tool +// failed on every 11.16.0 attempt), so an unrecognised reply is settled by +// looking: the delete succeeded iff the document is gone. +func TestDeleteDocument_UnrecognisedReplyVerifiedByLookup(t *testing.T) { + for _, tc := range []struct { + name string + found string + wantErr bool + }{ + {"gone", `{"foundDocuments":[]}`, false}, + {"still there", `{"foundDocuments":[{"qualifiedName":"MyFirstModule.Gone"}]}`, true}, + } { + t.Run(tc.name, func(t *testing.T) { + f := newFakePED(t, func(name string, _ map[string]any) (string, bool) { + if name == "ped_find_document" { + return tc.found, false + } + return "Deleted 1 document.", false + }) + f.tools = map[string][]string{"ped_delete_document": {"documents"}} + b := &Backend{client: f.connectClient(t)} + err := b.deleteDocument(constantDocType, "MyFirstModule", "Gone") + if (err != nil) != tc.wantErr { + t.Fatalf("deleteDocument err = %v, wantErr %v", err, tc.wantErr) + } + }) + } +} + +func TestDeleteDocument_NativeFailureSurfaces(t *testing.T) { + f := newFakePED(t, func(name string, _ map[string]any) (string, bool) { + if name == "ped_find_document" { + return `{"foundDocuments":[{"qualifiedName":"MyFirstModule.Gone"}]}`, false + } + return handlerSessionError, false + }) + f.tools = map[string][]string{"ped_delete_document": {"documents"}} + b := &Backend{client: f.connectClient(t)} + + err := b.deleteDocument(constantDocType, "MyFirstModule", "Gone") + if err == nil { + t.Fatal("a failed native delete must not report success") + } + if !strings.Contains(err.Error(), "Session 'mcp-server' not found") { + t.Errorf("error should carry Studio Pro's message, got: %v", err) + } +} + +func TestDeleteDocument_NoNativeToolNeeds1116(t *testing.T) { + f := newFakePED(t, nil) + f.tools = map[string][]string{"ped_create_document": {"documents"}} // pre-11.16 surface + b := &Backend{client: f.connectClient(t)} + + err := b.deleteDocument(constantDocType, "MyFirstModule", "Gone") + if err == nil || !strings.Contains(err.Error(), "needs Studio Pro 11.16+ (ped_delete_document)") { + t.Fatalf("without ped_delete_document the error should name the version it needs, got: %v", err) + } + if _, ok := f.callByName("ped_delete_document"); ok { + t.Error("ped_delete_document must not be called when the server does not advertise it") + } +} diff --git a/mdl/backend/mcp/enumeration.go b/mdl/backend/mcp/enumeration.go index 74220766d..cc1f2885e 100644 --- a/mdl/backend/mcp/enumeration.go +++ b/mdl/backend/mcp/enumeration.go @@ -43,8 +43,8 @@ func (b *Backend) UpdateEnumeration(enum *model.Enumeration) error { return fmt.Errorf("modifying enumeration %q is not yet supported by the MCP backend (create a new one, or edit it in Studio Pro)", enum.Name) } -// DeleteEnumeration drops an enumeration via Concord's delete_document (PED has -// no delete tool). Requires --mcp-concord; errors clearly otherwise. +// DeleteEnumeration drops an enumeration via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeleteEnumeration(id model.ID) error { enum, err := b.GetEnumeration(id) if err != nil { @@ -54,7 +54,7 @@ func (b *Backend) DeleteEnumeration(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for enumeration %q: %w", enum.Name, err) } - if err := b.concordDeleteDocument(modName, enum.Name); err != nil { + if err := b.deleteDocument(enumerationDocType, modName, enum.Name); err != nil { return err } b.forgetSessionEnum(modName, enum.Name) diff --git a/mdl/backend/mcp/enumeration_test.go b/mdl/backend/mcp/enumeration_test.go index 5c5041e19..92ae20b63 100644 --- a/mdl/backend/mcp/enumeration_test.go +++ b/mdl/backend/mcp/enumeration_test.go @@ -84,9 +84,9 @@ func TestListEnumerations_SessionTakesPrecedence(t *testing.T) { } } -func TestDeleteEnumeration_RequiresConcord(t *testing.T) { - // DROP routes to Concord's delete_document; without --mcp-concord it must - // give an actionable error (not "no delete tool ever"). Use a session module +func TestDeleteEnumeration_NeedsNativeDelete(t *testing.T) { + // DROP routes to ped_delete_document; without a connected server that has it, + // it must give an actionable error naming the version. Use a session module // + enum so the resolve path needs no local reader. b := &Backend{} mod := &model.Module{Name: "M"} @@ -99,7 +99,7 @@ func TestDeleteEnumeration_RequiresConcord(t *testing.T) { b.sessionEnums = append(b.sessionEnums, enum) err := b.DeleteEnumeration(enum.ID) - if err == nil || !strings.Contains(err.Error(), "Concord") { - t.Fatalf("DROP without Concord should error pointing at --mcp-concord, got: %v", err) + if err == nil || !strings.Contains(err.Error(), "11.16+") { + t.Fatalf("DROP without ped_delete_document should name Studio Pro 11.16+, got: %v", err) } } diff --git a/mdl/backend/mcp/handler_error_test.go b/mdl/backend/mcp/handler_error_test.go new file mode 100644 index 000000000..47f69f814 --- /dev/null +++ b/mdl/backend/mcp/handler_error_test.go @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import "testing" + +// handlerSessionError is Studio Pro 11.16.0's verbatim ped_delete_document +// failure, measured live: the tool ran, failed, and still said isError:false. +const handlerSessionError = "Error executing tool handler: Session 'mcp-server' not found" + +func TestCallTool_HandlerErrorIsAnError(t *testing.T) { + f := newFakePED(t, func(string, map[string]any) (string, bool) { return handlerSessionError, false }) + res, err := f.connectClient(t).CallTool("ped_delete_document", map[string]any{}) + if err != nil { + t.Fatalf("CallTool: %v", err) + } + if !res.IsError { + t.Fatalf("a %q result must be reported as an error, got IsError=false", handlerSessionError) + } +} diff --git a/mdl/backend/mcp/microflow.go b/mdl/backend/mcp/microflow.go index c6c2050f0..4d7c18c4a 100644 --- a/mdl/backend/mcp/microflow.go +++ b/mdl/backend/mcp/microflow.go @@ -372,8 +372,8 @@ func (b *Backend) ListMicroflows() ([]*microflows.Microflow, error) { return out, nil } -// DeleteMicroflow drops a microflow via Concord's delete_document (PED has no -// delete tool). Requires --mcp-concord; errors clearly otherwise. +// DeleteMicroflow drops a microflow via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeleteMicroflow(id model.ID) error { mf, err := b.GetMicroflow(id) if err != nil { @@ -383,7 +383,7 @@ func (b *Backend) DeleteMicroflow(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for microflow %q: %w", mf.Name, err) } - return b.concordDeleteDocument(modName, mf.Name) + return b.deleteDocument(microflowDocType, modName, mf.Name) } // GetMicroflow resolves by ID, preferring session-created microflows. diff --git a/mdl/backend/mcp/nanoflow.go b/mdl/backend/mcp/nanoflow.go index c4cd07e75..5d0372ada 100644 --- a/mdl/backend/mcp/nanoflow.go +++ b/mdl/backend/mcp/nanoflow.go @@ -14,7 +14,7 @@ import ( // Did you mean: Microflows$Microflow?") — its create whitelist covers microflows // but not nanoflows, despite the two sharing a structure. CREATE / CREATE OR MODIFY // are therefore rejected with an actionable error. DROP of an existing nanoflow -// still works via Concord's delete_document, like microflows; reads delegate to the +// uses ped_delete_document (11.16+), like microflows; reads delegate to the // local .mpr. // CreateNanoflow is gated on the capability model (capabilities.yaml / @@ -35,8 +35,8 @@ func (b *Backend) UpdateNanoflow(nf *microflows.Nanoflow) error { return errCreatePathUnbuilt("nanoflow", nf.Name) } -// DeleteNanoflow drops an existing nanoflow via Concord's delete_document (PED has -// no delete tool), like microflows. Requires --mcp-concord. +// DeleteNanoflow drops an existing nanoflow via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeleteNanoflow(id model.ID) error { nf, err := b.GetNanoflow(id) if err != nil { @@ -46,7 +46,7 @@ func (b *Backend) DeleteNanoflow(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for nanoflow %q: %w", nf.Name, err) } - return b.concordDeleteDocument(modName, nf.Name) + return b.deleteDocument(nanoflowDocType, modName, nf.Name) } // Nanoflow reads delegate to the local reader (nanoflows can't be created over MCP, diff --git a/mdl/backend/mcp/page.go b/mdl/backend/mcp/page.go index 45f2630cf..d497fc109 100644 --- a/mdl/backend/mcp/page.go +++ b/mdl/backend/mcp/page.go @@ -105,8 +105,8 @@ func (b *Backend) ListPages() ([]*pages.Page, error) { return out, nil } -// DeletePage drops a page via Concord's delete_document (PED has no delete tool). -// Requires --mcp-concord; errors clearly otherwise. +// DeletePage drops a page via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeletePage(id model.ID) error { page, err := b.GetPage(id) if err != nil { @@ -116,7 +116,7 @@ func (b *Backend) DeletePage(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for page %q: %w", page.Name, err) } - return b.concordDeleteDocument(modName, page.Name) + return b.deleteDocument(pageDocType, modName, page.Name) } // Page-related reads delegate to the local reader (the executor's page builder diff --git a/mdl/backend/mcp/workflow.go b/mdl/backend/mcp/workflow.go index c43612236..793e163cb 100644 --- a/mdl/backend/mcp/workflow.go +++ b/mdl/backend/mcp/workflow.go @@ -281,8 +281,8 @@ func (b *Backend) upsertSessionWorkflow(wf *workflows.Workflow) { b.sessionWorkflows = append(b.sessionWorkflows, wf) } -// DeleteWorkflow drops a workflow via Concord's delete_document (PED has no delete -// tool). Requires --mcp-concord. +// DeleteWorkflow drops a workflow via deleteDocument: ped_delete_document +// (Studio Pro 11.16+). func (b *Backend) DeleteWorkflow(id model.ID) error { wf, err := b.GetWorkflow(id) if err != nil { @@ -292,7 +292,7 @@ func (b *Backend) DeleteWorkflow(id model.ID) error { if err != nil { return fmt.Errorf("resolve module for workflow %q: %w", wf.Name, err) } - return b.concordDeleteDocument(modName, wf.Name) + return b.deleteDocument(workflowDocType, modName, wf.Name) } // ListWorkflows merges local (on-disk) workflows with those created this session. From 475d08dc182836b336d29904905d264db4e4be9d Mon Sep 17 00:00:00 2001 From: Ako Date: Sun, 1 Nov 2026 16:14:39 +0000 Subject: [PATCH 52/61] docs(proposals): playwright check through Studio Pro's MCP browser Proposes 'mxcli playwright check --mcp ': the same verdict and assertions, driven through Studio Pro 11.16's browser_* handlers instead of a local Chromium. Maps each PageSignals field to its handler source, records the absent evaluate/screenshot handlers, and leaves the handler input schemas as an open measurement (query_handlers timed out). Co-Authored-By: Claude Opus 5.5 --- .../PROPOSAL_playwright_check_via_mcp.md | 325 ++++++++++++++++++ 1 file changed, 325 insertions(+) create mode 100644 docs/11-proposals/PROPOSAL_playwright_check_via_mcp.md diff --git a/docs/11-proposals/PROPOSAL_playwright_check_via_mcp.md b/docs/11-proposals/PROPOSAL_playwright_check_via_mcp.md new file mode 100644 index 000000000..b7ee8adb2 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_playwright_check_via_mcp.md @@ -0,0 +1,325 @@ +--- +title: Playwright Check Through Studio Pro's MCP Browser +status: draft +date: 2026-11-01 +related: + - PROPOSAL_mcp_backend + - PROPOSAL_playwright_session_reuse + - proposal-playwright-cli +--- + +# Proposal: Playwright Check Through Studio Pro's MCP Browser + +**Status:** Draft +**Date:** 2026-11-01 + +## Problem Statement + +`mxcli playwright check` gives a one-line text verdict per page of a running app. +It covers rendering, rows, error banners, console errors, failed requests, the +sign-in form, access denied and wrong-page redirects. To get there it launches a +**local** headless Chromium through Node and `playwright` (`cmd/mxcli/docker/pagecheck.go`). +That only works when three things are true of the machine mxcli runs on: + +1. Node, Playwright and a Chromium are installed there. +2. The app is reachable from there. For `run --local` that is localhost. For an + app that **Studio Pro** runs on the host, a devcontainer has to forward the + port and the check has to be pointed at it with `--base-url`. +3. The app it checks is the one the model edits went into. + +Under `--mcp`, all three fail together. mxcli writes the model into a live Studio +Pro, and from 11.16 `--mcp-run` starts the app there too, through the +`mendix_app` handler. The app then runs **on Studio Pro's machine**, and the +local check either cannot reach it or has no browser to reach it with. So the +loop "edit through MCP → run in Studio Pro → check the page" breaks at the last +step. The agent falls back to driving the `browser_*` handlers by hand. That is +the hand-rolled navigate / snapshot / read sequence `playwright check` was built +to replace, and here every accessibility snapshot lands in the agent's context. + +Studio Pro 11.16 ships its own Playwright behind `execute_handler`. On +2026-11-01, `playwright_status` on 11.16.0 answered *"Playwright is available; +use the browser_\* handlers to check the UI."* Its browser runs next to the app, +so neither the reachability problem nor the install problem comes up. + +**Who benefits:** agents and contributors working with `--mcp` from a +devcontainer or another machine, which is the setup the MCP backend exists for. + +## Decision 1: A Mode of `playwright check`, Not a Flag on `--mcp-run` + +Recommendation: **`mxcli playwright check --mcp `**. It is the existing +command with the existing assertions and verdict format, and the browser is +Studio Pro's. It is **not** a `--mcp-check-page` added to `--mcp-run`. + +- **Building and checking are separate steps with separate failure modes.** + `--mcp-run` answers "did the app start" with exit 0 or an error. A check + answers "is this page right" with exit 0 for pass, 1 for fail and 2 for could + not check. One exit code cannot carry both answers, and agents read the exit + code. +- **A check is repeated far more often than a run.** The usual loop is: run + once, then check, fix through MCP, check again, and so on. Tied to `--mcp-run`, + every check would restart the app (`mendix_app run` restarts a running app), + which costs a minute each time and loses the session. +- **One command, one verdict.** Agents and `verify-in-runtime.md` already know + `playwright check`. A second command with the same output would split the + docs, the tests and the habits built on them. The browser is a transport + detail, the same way `--mcp` is for model writes. +- **They compose.** `mxcli exec fix.mdl --mcp URL --mcp-run` followed by + `mxcli playwright check /p/orders --mcp URL` is two commands with two clear + results. + +**Spelling.** `--mcp` is already a root **persistent string flag**: the server +URL, with `--mcp-dial` beside it (`cmd/mxcli/main.go`). A boolean `--mcp` on +`playwright check` would collide with it. So the mode is selected by the existing +flag carrying its URL, exactly as model writes are: + +```bash +mxcli playwright check /p/customers /p/orders -p app.mpr \ + --mcp http://localhost:7782/mcp --mcp-dial host.docker.internal:7782 +``` + +The `--mcp` help text ("Route model writes…") is widened to cover this use. + +**Base URL.** Under `--mcp`, the pages are resolved against `mendix_app status`'s +`appUrl` (measured as `http://localhost:8080/`), not `.docker/.env` or a +`run --local` handshake. That `localhost` is Studio Pro's, which is the point. +An explicit `--base-url` still wins. The check **never starts the app**. If +status reports not running, it exits 2 with "start it with `--mcp-run`". + +## What the Check Needs from `execute_handler` + +### Handlers, Measured 2026-11-01 on 11.16.0 + +`query_handlers {}` (captured earlier the same day in +`mx-test-projects/runs/mcp-surface/query_handlers.txt`) lists these browser +handlers: + +| Handler (`mcp_studio-pro-playwright_` prefix omitted) | Description as served | Used by the check | +|---|---|---| +| `playwright_status` (no prefix) | whether Playwright is available, and why not | **gate**: first call | +| `browser_navigate` | Navigate to a URL | each page | +| `browser_snapshot` | accessibility snapshot of the current page | each page: the main signal source | +| `browser_find` | search the snapshot for text or a regexp, with context | `--assert-text`, login-form and alert detection without a whole snapshot | +| `browser_wait_for` | wait for text to appear or disappear, or for a time | settling | +| `browser_console_messages` | all console messages | `console-errors=` | +| `browser_network_requests` / `browser_network_request` | numbered list of requests since load / one request's details | `http=`, failed same-origin requests, settling | +| `browser_fill_form`, `browser_type`, `browser_click`, `browser_press_key` | form input | sign-in | +| `browser_handle_dialog` | handle a dialog | a native dialog blocking the page | +| `browser_close` | close the page | `--sign-out` cleanup | +| `browser_navigate_back`, `browser_hover`, `browser_select_option` | — | not needed | + +**Absent from the list, and load-bearing for the design:** +`browser_evaluate`, `browser_take_screenshot`, `browser_tabs`, and any +storage-state handler. Upstream `@playwright/mcp` has these. Studio Pro serves a +curated subset (and adds `browser_find`, which upstream does not have). + +### Input Schemas: Not Yet Captured + +`getDetails` is the only source of the payload shapes, and `execute_handler`'s +description requires it before a call. It **could not be read**. From 15:3x to +15:58 UTC on 2026-11-01, every `query_handlers` call on the 11.16.0 server +(`host.docker.internal:7785`, Host: localhost) failed with JSON-RPC `-32000 +"Request timed out"` after 30 s. That includes `{operation: list}`, a +`keyword: browser` list, and `getDetails` for `mendix_app` alone, which had +answered earlier that day. Other tools on the same connection (`tools/list`, +`glob`) answered in about 0.2 s. So the handler registry, not the connection, +was stuck. My guess, not measured, is that it waits on the Playwright subserver +while listing. + +Two consequences for the design, whatever the schemas turn out to be: + +1. **A `query_handlers` timeout is "could not check" (exit 2)**, with that + cause named. `requireHandler` currently caches only answers. It must not + cache a timeout as "no such handler", and it must not retry it the way + mutation timeouts are retried (`timeout.go`). +2. The schemas are **re-measured before implementation starts** (Open + Question 1). The payload field names below follow upstream `@playwright/mcp` + and are placeholders until then: `navigate {url}`, `click {element, ref}`, + `type {element, ref, text, submit?}`, `fill_form {fields[]}`, + `wait_for {text?, textGone?, time?}`, `console_messages {level?}`. The + implementation reads nothing from them until measured. Each call site cites + the captured `getDetails` reply, the way `AppStatus` cites its measured reply. + +## How Each Verdict Signal Maps + +The verdict is built from `PageSignals` (`pagecheck.go`), which `EvaluatePage` +and `formatPageVerdict` turn into the line. The MCP mode must **produce the same +`PageSignals`** so that evaluation and formatting are shared, not reimplemented. +The local probe collects them with `page.evaluate` and CSS selectors. Neither is +available here, so each signal comes from a different source: + +| Signal | Local probe | Through MCP | Fidelity | +|---|---|---|---| +| `finalUrl`, `title` | `p.url()`, `p.title()` | the "Page URL" / "Page Title" lines of the navigate/snapshot reply | same, if the reply carries them (measure) | +| `status` | `goto` response | the document entry in `browser_network_requests` | same | +| `headings` | `h1,h2` outside the demo-user switcher | snapshot `heading [level=1/2]` nodes outside the switcher's subtree | same | +| `loginForm` | `#usernameInput` | snapshot textbox named like the sign-in field, or `browser_find` | same, keyed on the accessible name (measure on 11.16's login.html) | +| `alerts`, `errorAlerts` | `.alert-danger`, `.mx-dialog-error`, … | snapshot `alert` / `dialog` nodes | **weaker**: the danger/warning/info split relies on classes the tree may not show. Measure, and fall back to "any alert fails" if no split survives | +| `rows`, `dataWidgets` | grid/list-view selectors | `row` nodes with `gridcell` children; `listitem` under a list | close for Data Grid 2 (role=row); **list views to measure** | +| `textLen`, `textFound` | `body.innerText` | the snapshot's text content; `--assert-text` via `browser_find` | close | +| `consoleErrors` | `console` + `pageerror` events | `browser_console_messages`, errors only, minus same-origin "Failed to load resource" | same, if level is reported | +| `httpErrors` | `response` events ≥ 400 | `browser_network_requests` entries ≥ 400, same filters | same | +| `unsettled`, `pendingRequests` | in-flight map + text stable ×3 | poll `browser_network_requests` and `browser_find` until both are unchanged across polls, within `--wait` | **weaker**: requests still open may not be visible (#845 is the case where it matters). Measure whether the list shows pending entries | +| `counts` (`--assert-count`) | `locator(css).count()` | **CSS cannot be evaluated** | see below | +| `screenshot` | `p.screenshot` | **no handler** | refused | + +**`--assert-count`.** With no evaluate, a CSS selector such as +`.mx-name-dgCustomers [role=row]>=2` has nothing to run against. In MCP mode the +check accepts an **accessibility selector**, which is `role` with an optional +`"name"`, counted in the snapshot: `row>=2`, `button "Save"==1`. It refuses a CSS +selector with exit 2 and names the role form. It never silently passes. Widget +names (`mx-name-*`) do not appear in the accessibility tree. Whether a scoped +form (`within grid "Customers"`) is worth adding is Open Question 3. + +**`--screenshot`** is refused under `--mcp` with exit 2: "Studio Pro's browser +has no screenshot handler; drop --mcp for a visual check". `verify-in-runtime.md` +already makes a screenshot the exception, so this costs least where it matters +least. + +## Sign-in and Sessions + +- **Credentials are resolved as today**: `--user/--password`, `--role R` and the + demo-user fallback read project security from `-p`. The sign-in itself is + `navigate login.html` → fill the two fields → click Sign in, and the result is + confirmed by the next snapshot showing no sign-in form. A refusal still reads + the runtime log for the cause, but through `read_file /app-logs/app.log` + (`appLogPath` in `app.go`), not `/.mxcli/runtime.log`. +- **No storage-state file.** The session lives in Studio Pro's browser context, + so it persists across checks only as long as that browser does. Without a + storage handler, mxcli cannot save it or restore it. The check therefore + starts each run by asking whether the first page already shows the sign-in + form. If it does not, it treats the browser as signed in, and the verdict + names the user only when this run signed in. `--fresh-login` signs out first. + `--sign-out` signs out and closes the page at the end. +- **The shared browser is the user's too.** An agent working alongside a human in + Studio Pro drives the same browser. The check navigates away from whatever was + open, and the help text says so. +- **Passwords cross the MCP connection** into Studio Pro, the same connection + model writes use. Demo-user passwords are generated per project, but a + `--password` for a real account may land in a Studio Pro log. This goes in the + help text, measured if possible (Open Question 4). + +## Fit with `verify-in-runtime.md` and `playwright check` + +- **`playwright check`**: one command, two probes. `RunPageCheck` already + isolates its browser behind seams (`probePagesFn`, `loginFn`, `signOutFn` in + `pagecheck_check.go`). The MCP mode becomes a second implementation of those + three, selected when `--mcp` is set. Everything above them is unchanged and + shared: credential resolution, the access-denied and wrong-page rules, + `EvaluatePage` and the verdict line. A new failure rule then reaches both + modes at once. +- **`verify-in-runtime.md`**: the table "Symptom lives in → Sufficient proof" + gains no row. The tier is the same, and it is still the running app. The + procedure gains one branch at step 4/5 ("Provision and boot", "Assert in the + browser"): *when the model was written with `--mcp` into Studio Pro 11.16+, + boot with `--mcp-run` and assert with `playwright check --mcp`, not + `run --local`*. Step 6, the A/B against the broken build, works the same way. + Two limits are stated there: no screenshot, and `--assert-count` takes the + role form. The skill's "assert in text" rule fits better here than locally, + because the alternative under MCP is snapshots in the agent's context. +- **`live-edit-with-studio-pro` skill**: its "after `--mcp-run`" step points to + `playwright check --mcp` instead of browser handlers called by hand. +- **`PROPOSAL_playwright_session_reuse`**: does not apply to this mode. There is + no session file to reuse, and the browser's lifetime belongs to Studio Pro. + +## BSON Structure + +Not applicable. The check reads nothing from the model and writes nothing to +it. It talks only to the running app, through Studio Pro's browser. + +## Proposed CLI + +No MDL syntax. The command, flags and exit codes are those of +`playwright check`, with: + +| Flag | Under `--mcp` | +|---|---| +| `--mcp URL`, `--mcp-dial ADDR` | select Studio Pro's browser (root flags, existing) | +| `--base-url` | defaults to `mendix_app status` `appUrl` | +| `--assert-text`, `--wait`, `--user/--password/--role`, `--fresh-login`, `--sign-out`, `--allow-console-errors` | same meaning | +| `--assert-count` | accessibility selector (`row>=2`); a CSS selector is exit 2 | +| `--screenshot` | exit 2 | + +Exit 2 is also returned when the server lacks `execute_handler` (before 11.16), +when `playwright_status` reports unavailable (its reason is printed verbatim), when +`query_handlers` times out, and when the app is not running. + +## Implementation Plan + +**Prerequisite:** `mdl/backend/mcp/app.go` (`requireHandler`, `executeHandler`, +`GetAppStatus`) exists today only on branch `claude/festive-driscoll-aa5aa4`, +commit 41698502e ("drop Concord"), and is not on `main`. This work starts after +that lands. + +### Files to modify/create + +| File | Change | +|---|---| +| `mdl/backend/mcp/browser.go` (new) | `PlaywrightStatus()`, and thin wrappers over `executeHandler` for the handlers in the table above, using the measured payload shapes, with the reply returned as text | +| `mdl/backend/mcp/app.go` | `requireHandler`: a `-32000` timeout is returned as its own error (`ErrHandlersUnavailable`), never cached as "absent" | +| `cmd/mxcli/docker/pagecheck_snapshot.go` (new) | pure parser from snapshot text, console text and request-list text to `PageSignals`, plus the accessibility-selector counter. It has no I/O and is where most of the tests go | +| `cmd/mxcli/docker/pagecheck_mcp.go` (new) | an MCP implementation of probe, login and sign-out behind a small interface, with no import of `mdl/backend/mcp`. `cmd/mxcli` passes in an adapter, so `docker` stays free of the backend | +| `cmd/mxcli/docker/pagecheck_check.go` | `PageCheckOptions` gains `Browser` (nil means the local probe). The three seams dispatch on it | +| `cmd/mxcli/docker/pagecheck_assert.go` | `ParseCountAssertion` gains the role form, tagged so each mode refuses the other's form | +| `cmd/mxcli/cmd_playwright.go` | read `--mcp`/`--mcp-dial`, build the backend and adapter, resolve the base URL from status, refuse `--screenshot`, extend the help text | +| `cmd/mxcli/main.go` | widen the `--mcp` flag description | +| `.claude/skills/verify-in-runtime.md`, `.claude/skills/mendix/live-edit-with-studio-pro/SKILL.md`, `docs-site/src/tools/mcp-connect.md` | the branch and limits described above | + +Order: measure (Open Question 1) → snapshot parser with fixtures → `browser.go` → +dispatch in `RunPageCheck` → CLI → docs. + +## Version Compatibility + +Studio Pro **11.16+**, gated on **presence, not version**: the `execute_handler` +and `query_handlers` tools, then `playwright_status` by `getDetails`, and then +`playwright_status` itself must answer "available". Presence also depends on +Studio Pro's preferences and the servers connected to it +(`mcp-1116-surface.md`), so a version number alone would be wrong in both +directions. Before 11.16, the error names the version, as `--mcp-run` does. + +## Test Plan + +- **Snapshot parser (unit, the bulk).** Fixtures are snapshot, console and + request-list replies **captured from 11.16.0**, not hand-written, saved under + `cmd/mxcli/docker/testdata/mcp-snapshots/`. They cover: a page with a Data + Grid 2 that has rows; an empty list view; the sign-in form; an + `alert-danger`; an error dialog; the 403 landing; an unknown `/p/` that + redirects home with an info dialog (#998); and the demo-user switcher's + heading. Each asserts the `PageSignals` and the verdict line. +- **Backend wrappers (unit).** `newFakePED` in `mdl/backend/mcp/client_test.go` + answers `query_handlers` and `execute_handler`. The cases: the handler is + absent; `playwright_status` reports unavailable, and its reason is surfaced; a + `-32000` timeout gives `ErrHandlersUnavailable` and is not cached; + "Error executing tool handler: …" with `isError:false` is an error. +- **`RunPageCheck` dispatch (unit).** A fake `Browser` covers the sign-in + sequence, `--fresh-login`, `--sign-out`, an already-signed-in browser, CSS + `--assert-count` refused under MCP, and `--screenshot` refused. +- **Parity control (live, manual or `-tags integration` with an env-provided + URL).** One app, run by Studio Pro, with a passing page and a page that has a + seeded error banner. Check both pages with both probes, the local one through + a forwarded port. The verdicts must agree: the passing page must pass under + both and the banner page must fail under both. This is the control. A mode + that passes everything cannot pass it. +- **Fix-proven step.** For the settling rule, revert it and confirm that a page + whose list-view microflow sleeps 3 s reports `rows=0` instead of + STILL LOADING or its real rows. This is the #845 symptom, made to happen + through MCP. + +## Open Questions + +1. **Handler input schemas.** Re-run `query_handlers {operation: getDetails, + names: [playwright_status, mcp_studio-pro-playwright_browser_*]}` once the + registry answers again (a Studio Pro restart may be needed), and commit the + reply to `mx-test-projects/runs/mcp-surface/`. Also measure: does the navigate + or snapshot reply carry URL and title? Does `browser_network_requests` show + pending requests and the status of the document request? Does + `browser_console_messages` report the level? +2. **Why `query_handlers` hung.** Is it reproducible, does it follow the + Playwright subserver's start, and does it recover without a restart? If it is + the subserver, an exit 2 "could not check" is right, but the message should + say to restart Studio Pro. +3. **A scoped count.** Is `row>=2` enough without a "within grid X" scope? Pages + with two grids are common. +4. **Where the password goes.** Does Studio Pro log `execute_handler` payloads? +5. **Concurrent users of the browser.** Does Studio Pro's Maia chat share this + browser? If it does, a check run while Maia is browsing interleaves with it, + and the help text should say so. From 418e6fb60b3f0f45e63e57fe99a139444116bcb7 Mon Sep 17 00:00:00 2001 From: Ako Date: Sun, 1 Nov 2026 17:25:04 +0000 Subject: [PATCH 53/61] Report every widget measured as CE0582 under the React client (MPR012) MPR012 named only the static and dynamic image. The classic drop-down, the legacy data grid and the reference selector are CE0582 under the React client too, and lint said nothing about the ones a project already holds. The set is measured, not taken from the documentation: mxbuild 11.16.0 on a project switched to the React client reported CE0582 on the data grid and the reference selector (also with a select page) beside both image widgets, and 11.14.0 on the drop-down. The controls stay silent and are pinned by the test: navigation tree, menu bar, simple menu bar, file manager and scroll container (11.16.0), image uploader (11.13.0), and the common input and container widgets (11.15.0). Template grid and the reference set selectors were on no web page available to measure and are left out. The rule is renamed ReactUnsupportedWidget (ID unchanged), and each finding names its replacement. Closes #564 Co-Authored-By: Claude Opus 5.5 --- .claude/commands/mendix/lint.md | 2 +- CHANGELOG.md | 1 + cmd/mxcli/cmd_lint.go | 3 +- docs-site/src/language/widget-types.md | 6 +- mdl/executor/cmd_lint.go | 4 +- ..._widget.go => react_unsupported_widget.go} | 113 ++++++++++------- .../react_unsupported_widget_564_test.go | 114 ++++++++++++++++++ ...ct_unsupported_widget_client_1287_test.go} | 4 +- ...st.go => react_unsupported_widget_test.go} | 20 +-- 9 files changed, 207 insertions(+), 60 deletions(-) rename mdl/linter/rules/{legacy_image_widget.go => react_unsupported_widget.go} (50%) create mode 100644 mdl/linter/rules/react_unsupported_widget_564_test.go rename mdl/linter/rules/{legacy_image_widget_client_1287_test.go => react_unsupported_widget_client_1287_test.go} (95%) rename mdl/linter/rules/{legacy_image_widget_test.go => react_unsupported_widget_test.go} (88%) diff --git a/.claude/commands/mendix/lint.md b/.claude/commands/mendix/lint.md index ce4e60102..37ad36a46 100644 --- a/.claude/commands/mendix/lint.md +++ b/.claude/commands/mendix/lint.md @@ -44,7 +44,7 @@ mxcli lint -p app.mpr --baseline lint-baseline.json | MPR005 | quality | ImageSource - IMAGE widgets with no source configured | | MPR006 | quality | EmptyContainer - Empty layout containers | | MPR007 | security | PageNavigationSecurity - Navigation pages need allowed roles (CE0557); every user role must be able to open its home page (at Production security, else CE2729 per widget) | -| MPR012 | correctness | LegacyImageWidget - staticimage/dynamicimage are unsupported by the React client (CE0582; info on the classic client) | +| MPR012 | correctness | ReactUnsupportedWidget - static/dynamic image, drop-down, legacy data grid and reference selector are unsupported by the React client (CE0582; info on the classic client) | | MPR013 | correctness | SnippetCallAppearance - Class/Style/DynamicClasses/design properties on a snippet call never reach the page (it renders no element) | | SEC001 | security | NoEntityAccessRules - Persistent entities need access rules | | SEC002 | security | WeakPasswordPolicy - Password minimum length should be 8+ | diff --git a/CHANGELOG.md b/CHANGELOG.md index 695075aa0..829cac26c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **MPR012 covers every widget measured as CE0582 under the React client** (ako/mxcli#564) — besides the static and dynamic image it now reports the classic drop-down, the legacy data grid and the reference selector a project already holds, each with its replacement (`combobox`, Data grid 2). The set is measured on 11.16 with mxbuild, controls included: the navigation tree, menu bar, file manager, scroll container, image uploader and the common input widgets build clean and stay silent. The rule is renamed `ReactUnsupportedWidget`; its ID is unchanged. - **`mxcli lint --baseline ` reports only new findings** (ako/mxcli#1328, mendixlabs/mxcli#1455) — record a baseline with `--update-baseline`, commit it, and later runs report only the findings it does not have, with an exit code that follows them alone and a `N new, M already known, K fixed` summary on stderr. Findings are matched on rule and document, not message, so rewording a rule does not make its findings new; a scoped run (`-m`, `-d`, `-r`) reports new findings but cannot update the baseline. - **Lint permission rows name their access rule** (ako/mxcli#1437, mendixlabs/mxcli#1539) — `permissions()` and `permissions_for()` rows carry `access_rule_id` and `access_rule_index`, so a rule can count an entity's access rules per setting or list one rule's module roles exactly, instead of guessing by grouping on XPath, default and rights. The catalog schema moves to version 31, so a cached catalog is rebuilt once. - **Lint rules can read the data guideline rules asked for** (ako/mxcli#1453, ako/mxcli#1301) — a microflow's `excluded` and `apply_entity_access`, an enumeration's `values` with their captions (so a rule can resolve `getCaption(Module.Enum.Value)`), `constant_values()` for the run-configuration values, and `project_settings()` (default language, OQL version, Java version, hash algorithm, time zone, rounding, decimal scale, optimistic locking, foreign keys, web client). The catalog schema moves to version 30, so a cached catalog is rebuilt once. diff --git a/cmd/mxcli/cmd_lint.go b/cmd/mxcli/cmd_lint.go index 66a2ab5d1..45a01330c 100644 --- a/cmd/mxcli/cmd_lint.go +++ b/cmd/mxcli/cmd_lint.go @@ -30,6 +30,7 @@ Built-in rules check for: - Empty containers (MPR006) - layout containers with no children - Navigation page security (MPR007) - pages in navigation need allowed roles, and every user role must be able to open its home page - Gallery selection listener (MPR009) - DataView 'DataSource: selection X' needs gallery 'ItemSelectionMode: toggle' (Studio Pro CE3637) + - React-unsupported widgets (MPR012) - static/dynamic image, drop-down, legacy data grid, reference selector are CE0582 under the React client - Entity access rules (SEC001) - persistent entities need access rules - Password policy (SEC002) - password minimum length should be 8+ - Demo users (SEC003) - demo users should be off at Production security @@ -475,7 +476,7 @@ func builtinLintRules() []linter.Rule { rules.NewDomainModelSizeRule(), rules.NewValidationFeedbackRule(), rules.NewImageSourceRule(), - rules.NewLegacyImageWidgetRule(), + rules.NewReactUnsupportedWidgetRule(), rules.NewSnippetCallAppearanceRule(), rules.NewEmptyContainerRule(), rules.NewGallerySelectionListenerRule(), diff --git a/docs-site/src/language/widget-types.md b/docs-site/src/language/widget-types.md index 7921989c2..bb5efac42 100644 --- a/docs-site/src/language/widget-types.md +++ b/docs-site/src/language/widget-types.md @@ -352,7 +352,11 @@ The classic drop-down (`DROPDOWN`, stored as `Forms$DropDown`) is not supported the React client: on a project whose Web UI setting is `OptimizedClient: Yes` (`show settings`), mxbuild reports CE0582 and `check --references` reports **MDL-WIDGET40**. Use `COMBOBOX` over the same attribute. Under `No` and -`MigrationMode` the drop-down builds, and nothing is reported. +`MigrationMode` the drop-down builds, and nothing is reported. A drop-down the +project already holds is reported by `mxcli lint` as **MPR012**, together with +the other built-in widgets that are CE0582 under the React client: the static and +dynamic image, the legacy data grid and the reference selector (measured on +11.16; the template grid and the reference set selectors are not yet measured). ### TEXTBOX diff --git a/mdl/executor/cmd_lint.go b/mdl/executor/cmd_lint.go index d19b7dc29..9dfeb00c6 100644 --- a/mdl/executor/cmd_lint.go +++ b/mdl/executor/cmd_lint.go @@ -35,7 +35,7 @@ func execLint(ctx *ExecContext, s *ast.LintStmt) error { rules.NewDomainModelSizeRule(), rules.NewValidationFeedbackRule(), rules.NewImageSourceRule(), - rules.NewLegacyImageWidgetRule(), + rules.NewReactUnsupportedWidgetRule(), rules.NewSnippetCallAppearanceRule(), rules.NewMissingTranslationsRule(), rules.NewRequiredCaptionDefaultLanguageRule(), @@ -153,7 +153,7 @@ func listLintRules(ctx *ExecContext) error { lint.AddRule(rules.NewDomainModelSizeRule()) lint.AddRule(rules.NewValidationFeedbackRule()) lint.AddRule(rules.NewImageSourceRule()) - lint.AddRule(rules.NewLegacyImageWidgetRule()) + lint.AddRule(rules.NewReactUnsupportedWidgetRule()) lint.AddRule(rules.NewSnippetCallAppearanceRule()) lint.AddRule(rules.NewMissingTranslationsRule()) lint.AddRule(rules.NewRequiredCaptionDefaultLanguageRule()) diff --git a/mdl/linter/rules/legacy_image_widget.go b/mdl/linter/rules/react_unsupported_widget.go similarity index 50% rename from mdl/linter/rules/legacy_image_widget.go rename to mdl/linter/rules/react_unsupported_widget.go index 03f2e4d81..a7a774993 100644 --- a/mdl/linter/rules/legacy_image_widget.go +++ b/mdl/linter/rules/react_unsupported_widget.go @@ -8,30 +8,35 @@ import ( "github.com/mendixlabs/mxcli/mdl/linter" ) -// LegacyImageWidgetRule reports Mendix's two legacy image widgets, neither of -// which the React client supports. +// ReactUnsupportedWidgetRule reports the built-in widgets the React client does +// not support: the two legacy image widgets, the classic drop-down, the legacy +// data grid and the reference selector (ako/mxcli#564). // // # Why this is correctness, not style // // The React client arrived in Mendix 10.7, and mxbuild reports CE0582 on each -// of these widgets whenever it is enabled — -// measured on 11.12.1, where a page carrying one is a build ERROR: +// of these widgets whenever it is enabled — measured on 11.12.1, where a page +// carrying one is a build ERROR: // // [error] [CE0582] "Widget static image is not supported in React client. // Right-click this error to convert it to an alternative // widget type." at Static image 'imgLogo' // -// Studio Pro offers the conversion from that error's context menu; the -// replacement is the pluggable Image widget, which mxcli spells `image` and -// which takes the same `Image:` reference. +// Studio Pro offers the conversion from that error's context menu. Which +// widgets are in the set is measured, not taken from the documentation (see +// reactUnsupportedWidgets). // // # Why mxcli writes them at all // // Because a project being converted UP already contains them, and mxcli's job -// there is to round-trip the model rather than quietly rewrite it. Both widgets -// are authorable and both round-trip through DESCRIBE. What was missing was -// anyone saying so at author time: `mxcli check` is silent on them, and the only -// signal was CE0582 at the far end of a build. +// there is to round-trip the model rather than quietly rewrite it. The image +// widgets and the drop-down are authorable and round-trip through DESCRIBE; the +// legacy data grid and the reference selector come from Studio Pro. What was +// missing was anyone saying so: `mxcli check` is silent on them, and the only +// signal was CE0582 at the far end of a build. The drop-down is the exception +// at author time — `check --references` refuses it in a script (MDL-WIDGET40) +// — but not for the drop-downs a project already holds, which only this rule +// reports. // // # Why the LINTER and not `mxcli check` // @@ -50,54 +55,78 @@ import ( // (notPlatformModule excludes any module with a Source), which is what keeps the // rule off the Studio Pro-authored static images a blank app inherits from // FeedbackModule — content the reader cannot fix and an update would replace. -type LegacyImageWidgetRule struct{} +type ReactUnsupportedWidgetRule struct{} -// NewLegacyImageWidgetRule creates a new legacy image widget rule. -func NewLegacyImageWidgetRule() *LegacyImageWidgetRule { - return &LegacyImageWidgetRule{} +// NewReactUnsupportedWidgetRule creates the MPR012 rule. +func NewReactUnsupportedWidgetRule() *ReactUnsupportedWidgetRule { + return &ReactUnsupportedWidgetRule{} } -func (r *LegacyImageWidgetRule) ID() string { return "MPR012" } -func (r *LegacyImageWidgetRule) Name() string { return "LegacyImageWidget" } -func (r *LegacyImageWidgetRule) Category() string { return "correctness" } -func (r *LegacyImageWidgetRule) DefaultSeverity() linter.Severity { return linter.SeverityWarning } +func (r *ReactUnsupportedWidgetRule) ID() string { return "MPR012" } +func (r *ReactUnsupportedWidgetRule) Name() string { return "ReactUnsupportedWidget" } +func (r *ReactUnsupportedWidgetRule) Category() string { return "correctness" } +func (r *ReactUnsupportedWidgetRule) DefaultSeverity() linter.Severity { return linter.SeverityWarning } // RequiredCatalogMode: ctx.Widgets() reads widgets_data, which only a full // catalog build fills; under the default fast build the rule found nothing. -func (r *LegacyImageWidgetRule) RequiredCatalogMode() linter.CatalogMode { return linter.CatalogFull } +func (r *ReactUnsupportedWidgetRule) RequiredCatalogMode() linter.CatalogMode { + return linter.CatalogFull +} -func (r *LegacyImageWidgetRule) Description() string { - return "Checks for the legacy static/dynamic image widgets, which the React client does not support (CE0582)" +func (r *ReactUnsupportedWidgetRule) Description() string { + return "Checks for built-in widgets the React client does not support — static/dynamic image, drop-down, legacy data grid, reference selector (CE0582)" } -// LegacyImage describes one of the two widgets, in the words the author will -// meet elsewhere: Mendix's own term (which CE0582 uses) and the MDL keyword. -type LegacyImage struct { - MendixTerm string - MDLKeyword string +// ReactUnsupported describes one widget of the set, in the words the author +// will meet elsewhere: Mendix's own term (which CE0582 uses), the MDL keyword +// (empty for a widget MDL cannot write), and what to use instead. +type ReactUnsupported struct { + MendixTerm string + MDLKeyword string + Replacement string } -// legacyImageWidgets is deliberately a deny-list of exactly two storage names. -// An allow-list would make every unrecognised widget a violation, and the one -// widget it must never fire on is the pluggable Image — the replacement. +// reactUnsupportedWidgets is deliberately a deny-list of measured storage +// names. An allow-list would make every unrecognised widget a violation, and +// the widgets it must never fire on are the pluggable replacements. +// +// Measured with mxbuild on projects set to the React client: +// - 11.12.1: static image; 11.14.0: drop-down (MDL-WIDGET40); +// - 11.16.0 (FactoryManagement): data grid, reference selector, and the +// reference selector with a select page — the same Forms$ReferenceSelector; +// static and dynamic image again. +// +// And measured silent, so left out: navigation tree, menu bar, simple menu bar, +// file manager, scroll container (11.16.0); image uploader (11.13.0); text box, +// text area, check box, date picker, radio buttons, list view, group box, +// navigation list (11.15.0). The template grid and the reference set selectors +// were on no web page available to measure, so they are not claimed either way. // // DocumentTemplates$StaticImageViewer is a different type in a different // document kind and is not covered: document templates are not pages, the React // client does not render them, and CE0582 does not mention them. -var legacyImageWidgets = map[string]LegacyImage{ - "Forms$StaticImageViewer": {MendixTerm: "static image", MDLKeyword: "staticimage"}, - "Forms$ImageViewer": {MendixTerm: "dynamic image", MDLKeyword: "dynamicimage"}, +var reactUnsupportedWidgets = map[string]ReactUnsupported{ + "Forms$StaticImageViewer": {"static image", "staticimage", + "Replace `staticimage` with the pluggable `image` widget (same `Image:` reference)"}, + "Forms$ImageViewer": {"dynamic image", "dynamicimage", + "Replace `dynamicimage` with the pluggable `image` widget (same `Image:` reference)"}, + "Forms$DropDown": {"drop-down", "dropdown", + "Replace `dropdown` with `combobox` over the same attribute"}, + "Forms$DataGrid": {"data grid", "", + "Replace the legacy data grid with Data grid 2 (`datagrid`)"}, + "Forms$ReferenceSelector": {"reference selector", "", + "Replace the reference selector with `combobox` over the same association"}, } -// LegacyImageWidget reports whether widgetType is one of the two legacy image -// widgets, and how to name it. -func LegacyImageWidget(widgetType string) (LegacyImage, bool) { - got, ok := legacyImageWidgets[widgetType] +// ReactUnsupportedWidget reports whether widgetType is one the React client +// does not support, and how to name and replace it. +func ReactUnsupportedWidget(widgetType string) (ReactUnsupported, bool) { + got, ok := reactUnsupportedWidgets[widgetType] return got, ok } -// Check reports one violation per legacy image widget found. -func (r *LegacyImageWidgetRule) Check(ctx *linter.LintContext) []linter.Violation { +// Check reports one violation per React-unsupported widget found. +func (r *ReactUnsupportedWidgetRule) Check(ctx *linter.LintContext) []linter.Violation { var violations []linter.Violation native := ctx.NativePages() classic := runsClassicClient(ctx) @@ -106,7 +135,7 @@ func (r *LegacyImageWidgetRule) Check(ctx *linter.LintContext) []linter.Violatio if ctx.IsExcluded(w.ModuleName) { continue } - legacy, ok := LegacyImageWidget(w.WidgetType) + legacy, ok := ReactUnsupportedWidget(w.WidgetType) if !ok { continue } @@ -141,9 +170,7 @@ func (r *LegacyImageWidgetRule) Check(ctx *linter.LintContext) []linter.Violatio DocumentName: docNameFromQualified(w.ContainerQualifiedName), DocumentID: w.ContainerID, }, - Suggestion: fmt.Sprintf( - "Replace `%s` with the pluggable `image` widget (same `Image:` reference), or convert it in Studio Pro from the CE0582 error's context menu", - legacy.MDLKeyword), + Suggestion: legacy.Replacement + ", or convert it in Studio Pro from the CE0582 error's context menu", }) } diff --git a/mdl/linter/rules/react_unsupported_widget_564_test.go b/mdl/linter/rules/react_unsupported_widget_564_test.go new file mode 100644 index 000000000..1dfd85e15 --- /dev/null +++ b/mdl/linter/rules/react_unsupported_widget_564_test.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package rules + +import ( + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/catalog" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/model" +) + +// ako/mxcli#564: MPR012 named only the two legacy image widgets, but CE0582 is +// a whole set. Measured with mxbuild on a project switched to the React client +// (Web UI → UseOptimizedClient = Yes): +// +// - 11.16.0, FactoryManagement: CE0582 on "data grid", "reference selector" +// and "reference selector (with select page)" — both are +// Forms$ReferenceSelector — beside the static and dynamic image; +// - 11.14.0: CE0582 on the classic drop-down (MDL-WIDGET40's measurement). +// +// The controls are measured too, and must stay silent: in the same 11.16 build +// the navigation tree, menu bar, simple menu bar, file manager and scroll +// container raised nothing; on 11.13.0 (JTSBootLogboek) neither did the image +// uploader; on a fresh 11.15.0 app neither did a text box, text area, check +// box, date picker, radio buttons, file manager, list view, group box or +// navigation list. The template grid and the reference set selectors were on +// no web page available to measure, so they are not claimed either way. +func TestReactUnsupportedWidgets_CoverTheMeasuredCE0582Set(t *testing.T) { + cat, err := catalog.New() + if err != nil { + t.Fatal(err) + } + defer cat.Close() + + unsupported := map[string]string{ // storage type -> mxbuild's term in CE0582 + "Forms$StaticImageViewer": "static image", + "Forms$ImageViewer": "dynamic image", + "Forms$DropDown": "drop-down", + "Forms$DataGrid": "data grid", + "Forms$ReferenceSelector": "reference selector", + } + supported := []string{ + "Forms$NavigationTree", "Forms$MenuBar", "Forms$SimpleMenuBar", "Forms$FileManager", + "Forms$ScrollContainer", "Forms$ImageUploader", "Forms$TextBox", "Forms$TextArea", + "Forms$CheckBox", "Forms$DatePicker", "Forms$RadioButtonGroup", "Forms$ListView", + "Forms$GroupBox", "Forms$NavigationList", + "CustomWidgets$CustomWidget", // the replacements: image, combobox, datagrid + // unmeasured, so not claimed + "Forms$TemplateGrid", "Forms$ReferenceSetSelector", "Forms$InputReferenceSetSelector", + } + + stmts := []string{ + `INSERT INTO modules_data (Id, Name, QualifiedName, ModuleName, Source) VALUES + ('m1', 'MyFirstModule', 'MyFirstModule', 'MyFirstModule', '')`, + `INSERT INTO pages_data (Id, Name, QualifiedName, ModuleName, LayoutRef) VALUES + ('p1', 'P_Legacy', 'MyFirstModule.P_Legacy', 'MyFirstModule', 'Atlas_Core.Atlas_Default')`, + } + n := 0 + add := func(widgetType string) { + n++ + stmts = append(stmts, `INSERT INTO widgets_data (Id, Name, WidgetType, ContainerId, ContainerQualifiedName, ContainerType, ModuleName) + VALUES ('w`+strings.Repeat("x", n)+`', '`+widgetType+`', '`+widgetType+`', 'p1', 'MyFirstModule.P_Legacy', 'PAGE', 'MyFirstModule')`) + } + for wt := range unsupported { + add(wt) + } + for _, wt := range supported { + add(wt) + } + for _, stmt := range stmts { + if _, err := cat.CatalogDB().Exec(stmt); err != nil { + t.Fatalf("%s: %v", stmt, err) + } + } + + reader := settingsReader{ps: &model.ProjectSettings{WebUI: &model.WebUISettings{UseOptimizedClient: "Yes"}}} + vs := NewReactUnsupportedWidgetRule().Check(linter.NewLintContext(cat, reader)) + + var reported []string + for _, v := range vs { + // The widget is named after its storage type in this fixture, so the + // message says which one it was. + hit := "" + for wt := range unsupported { + if strings.Contains(v.Message, "'"+wt+"'") { + hit = wt + } + } + if hit == "" { + t.Errorf("reported a widget outside the measured CE0582 set: %q", v.Message) + continue + } + reported = append(reported, hit) + // mxbuild's own words, so the lint finding and the build error match. + if !strings.HasPrefix(v.Message, unsupported[hit]+" ") { + t.Errorf("%s: message %q does not open with mxbuild's term %q", hit, v.Message, unsupported[hit]) + } + if v.Suggestion == "" { + t.Errorf("%s: no suggestion — the replacement is the useful half of the finding", hit) + } + } + sort.Strings(reported) + var want []string + for wt := range unsupported { + want = append(want, wt) + } + sort.Strings(want) + if strings.Join(reported, ",") != strings.Join(want, ",") { + t.Errorf("reported %v, want exactly %v", reported, want) + } +} diff --git a/mdl/linter/rules/legacy_image_widget_client_1287_test.go b/mdl/linter/rules/react_unsupported_widget_client_1287_test.go similarity index 95% rename from mdl/linter/rules/legacy_image_widget_client_1287_test.go rename to mdl/linter/rules/react_unsupported_widget_client_1287_test.go index a133b9a19..0a503629a 100644 --- a/mdl/linter/rules/legacy_image_widget_client_1287_test.go +++ b/mdl/linter/rules/react_unsupported_widget_client_1287_test.go @@ -25,7 +25,7 @@ func (r settingsReader) GetProjectSettings() (*model.ProjectSettings, error) { r // on an app at 0 errors (mendixlabs/mxcli#1226, ako/mxcli#1287). Measured on 11.14.0 for the same CE0582 on a classic // drop-down: an error under "Yes", clean under "No" and "MigrationMode" // (usesReactClient in mdl/executor). -func TestLegacyImageWidgetRule_FollowsTheProjectsWebClient(t *testing.T) { +func TestReactUnsupportedWidgetRule_FollowsTheProjectsWebClient(t *testing.T) { cat, err := catalog.New() if err != nil { t.Fatal(err) @@ -45,7 +45,7 @@ func TestLegacyImageWidgetRule_FollowsTheProjectsWebClient(t *testing.T) { } check := func(reader linter.LintReader) linter.Violation { t.Helper() - vs := NewLegacyImageWidgetRule().Check(linter.NewLintContext(cat, reader)) + vs := NewReactUnsupportedWidgetRule().Check(linter.NewLintContext(cat, reader)) if len(vs) != 1 { t.Fatalf("want one MPR012 for img1, got %d: %v", len(vs), vs) } diff --git a/mdl/linter/rules/legacy_image_widget_test.go b/mdl/linter/rules/react_unsupported_widget_test.go similarity index 88% rename from mdl/linter/rules/legacy_image_widget_test.go rename to mdl/linter/rules/react_unsupported_widget_test.go index 18a3cf89f..9b96083c0 100644 --- a/mdl/linter/rules/legacy_image_widget_test.go +++ b/mdl/linter/rules/react_unsupported_widget_test.go @@ -22,7 +22,7 @@ import ( // mxcli can author both (it must: a project being converted up already contains // them), and nothing told the author they were reaching for a widget their app // cannot build. This rule is that telling. -func TestLegacyImageWidget_NamesBothTypesAndTheReplacement(t *testing.T) { +func TestReactUnsupportedWidget_NamesTheImageTypesAndTheReplacement(t *testing.T) { cases := []struct { widgetType string wantTerm string @@ -33,7 +33,7 @@ func TestLegacyImageWidget_NamesBothTypesAndTheReplacement(t *testing.T) { } for _, c := range cases { t.Run(c.widgetType, func(t *testing.T) { - got, ok := LegacyImageWidget(c.widgetType) + got, ok := ReactUnsupportedWidget(c.widgetType) if !ok { t.Fatalf("%s not recognised as a legacy image widget", c.widgetType) } @@ -49,11 +49,11 @@ func TestLegacyImageWidget_NamesBothTypesAndTheReplacement(t *testing.T) { } } -// The CONTROL, and the reason this is a deny-list of two rather than anything +// The CONTROL, and the reason this is a deny-list rather than anything // broader: every other widget must be silent. A rule that fires on the // PLUGGABLE image — the widget it tells people to move TO — would be worse than // no rule at all. -func TestLegacyImageWidget_IsSilentOnEverythingElse(t *testing.T) { +func TestReactUnsupportedWidget_IsSilentOnTheReplacements(t *testing.T) { for _, widgetType := range []string{ "CustomWidgets$CustomWidget", // the pluggable image lives here "Forms$DynamicText", @@ -61,8 +61,8 @@ func TestLegacyImageWidget_IsSilentOnEverythingElse(t *testing.T) { "DocumentTemplates$StaticImageViewer", // a document template, not a page "", } { - if got, ok := LegacyImageWidget(widgetType); ok { - t.Errorf("%q was reported as a legacy image widget (%+v)", widgetType, got) + if got, ok := ReactUnsupportedWidget(widgetType); ok { + t.Errorf("%q was reported as React-unsupported (%+v)", widgetType, got) } } } @@ -70,8 +70,8 @@ func TestLegacyImageWidget_IsSilentOnEverythingElse(t *testing.T) { // The rule's identity is part of its contract: an ID that collides with another // rule's silently shadows it in the config, and the category decides what // `--category` filters it into. -func TestLegacyImageWidgetRule_Identity(t *testing.T) { - r := NewLegacyImageWidgetRule() +func TestReactUnsupportedWidgetRule_Identity(t *testing.T) { + r := NewReactUnsupportedWidgetRule() if r.ID() != "MPR012" { t.Errorf("ID = %q, want MPR012", r.ID()) } @@ -90,7 +90,7 @@ func TestLegacyImageWidgetRule_Identity(t *testing.T) { // platform is what tells them apart — note that the native layout lives in a // Marketplace module (Atlas_Core), so the lookup must not apply the // platform-module filter the iterators use. The web page is the control. -func TestLegacyImageWidgetRule_SkipsNativePages(t *testing.T) { +func TestReactUnsupportedWidgetRule_SkipsNativePages(t *testing.T) { db, err := sql.Open("sqlite", ":memory:") if err != nil { t.Fatal(err) @@ -122,7 +122,7 @@ func TestLegacyImageWidgetRule_SkipsNativePages(t *testing.T) { t.Fatalf("%s: %v", stmt, err) } } - violations := NewLegacyImageWidgetRule().Check(linter.NewLintContextFromDB(catalog.WrapSqlDB(db))) + violations := NewReactUnsupportedWidgetRule().Check(linter.NewLintContextFromDB(catalog.WrapSqlDB(db))) if len(violations) != 1 || violations[0].Location.DocumentName != "Logboek_Images" { var got []string for _, v := range violations { From d910b4d8835f3789f390ad00aec89ac6064291cb Mon Sep 17 00:00:00 2001 From: Ako Date: Sun, 1 Nov 2026 17:42:45 +0000 Subject: [PATCH 54/61] Write an empty header for an uncaptioned custom-content data grid column Through the generic `pluggablewidget 'com.mendix.widget.web.datagrid.Datagrid'` form, a custom-content column with `Caption: ''` (or none) got its MDL column name as the header text, and end users saw "colTriage". Mendix stores no name for a column, and Data grid 2 renders the header text as is (`header.trim() || ' '` in Datagrid.mjs). The name was a stand-in for an empty header, which ledger #54 round 2 took to be CE0463. It is not: that CE0463 was a null TextTemplate, and the empty Forms$ClientTemplate that update-widgets writes builds clean (ChipCoV8 #11; 0 errors on 11.15.0, where the keyword `datagrid` form already writes it). Verified in the running app (11.15.0): the page written by the old build renders "colTriage" in its header, the one written by this build does not. Closes #535 Co-Authored-By: Claude Opus 5.5 --- ...stom-content-column-header-shows-name.json | 1 + CHANGELOG.md | 1 + ...535-custom-content-column-empty-header.mdl | 36 +++++++++++++ .../ledger-54-custom-content-column.mdl | 19 ++++--- mdl/executor/widget_defs_test.go | 44 ++++++++------- mdl/executor/widget_engine.go | 54 +++++++++---------- 6 files changed, 99 insertions(+), 56 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-11-01-custom-content-column-header-shows-name.json create mode 100644 mdl-examples/bug-tests/535-custom-content-column-empty-header.mdl diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-11-01-custom-content-column-header-shows-name.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-01-custom-content-column-header-shows-name.json new file mode 100644 index 000000000..85bc8554b --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-11-01-custom-content-column-header-shows-name.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-11-01", "symptom": "A Data grid 2 custom-content column written with `Caption: ''` (or no caption) through the generic `pluggablewidget 'com.mendix.widget.web.datagrid.Datagrid'` path showed its MDL column name (`colTriage`) as the header to end users; mx check, check and lint all passed.", "cause": "applyColumnHeaderFallback filled an empty header on an attribute-less column with the column's NAME, a name Mendix does not store for a column. It was written on the belief that an empty header is CE0463; that CE0463 was a null TextTemplate, and the empty Forms$ClientTemplate (ChipCoV8 #11) builds clean. The keyword `datagrid` path already wrote the empty header.", "file": "mdl/executor/widget_engine.go", "insight": "Read the widget's own bundle before reasoning about a render: Datagrid.mjs renders `header.trim() || ' '`, so no runtime fallback exists and the text had to come from the writer. Grepping the stored unit for the name settled which path wrote it. A CE0463 'fix' that substitutes a value should be re-measured once the null-vs-empty shape is known; this one outlived the shape fix by three months. Separately: create or replace does not clear a column caption to '' on either path (Unchanged page), so an app already holding the name needs the page dropped and recreated.", "issue": "ako/mxcli#535"} diff --git a/CHANGELOG.md b/CHANGELOG.md index 695075aa0..1683ffcec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A custom-content data grid column with an empty caption no longer shows its MDL name as the header** (ako/mxcli#535) — through the generic `pluggablewidget 'com.mendix.widget.web.datagrid.Datagrid'` form, `column colTriage (Caption: '') { … }` (or no caption) wrote `colTriage` as the header text, which end users saw; Mendix stores no name for a column. It now writes an empty header, as the keyword `datagrid` form already did; it builds clean (11.15.0) and Data grid 2 renders it blank. A page that already holds the name keeps it until the page is dropped and created again: `create or replace` does not yet clear a caption to `''`. - **`describe` and the source catalog no longer decode every document of a type to find one** (ako/mxcli#1850) — describing a snippet decoded every snippet of the project, a nanoflow every microflow and nanoflow, a page every page, and so did resolving the snippets, pages, layouts and flows a description names. A source catalog build describes every document, so it paid that once per document: on FactoryManagement a first `refresh catalog full source` took 33 s and 145 s of CPU, and now takes 19 s and 38 s, with identical text for all 3,718 documents. `create nanoflow` likewise decodes only the nanoflows of its own name. - **Refreshing the catalog after a model edit no longer spends seconds on widget definitions** (ako/mxcli#1848) — `refresh catalog` re-read every widget package on every run, and on FactoryManagement it took about 3 s of a 5 s refresh on Windows. The step is now skipped when no `.mpk` or `.def.json` file has changed since the last refresh by the same mxcli build, and reading a widget's editor rules is no longer quadratic in the size of its `editorConfig.js` (Maps' took 0.5 s and now takes 15 ms, with identical rules on 256 widgets). Two widgets that derive the same keyword, such as `KeyboardShortcut.mpk` and `itvisors.KeyboardShortcut.mpk`, no longer overwrite each other's definition: the pluggable one keeps `KEYBOARDSHORTCUT` and the other becomes `CUSTOMKEYBOARDSHORTCUT`. Measured on a Mac, a refresh after one edit went from 2.8–3.2 s to 1.0–1.4 s. - **MPR012 follows the app's Web UI client** (ako/mxcli#1287, mendixlabs/mxcli#1226) — a Mendix 11 app can still run the classic client, where the legacy static and dynamic image widgets build clean, but MPR012 warned on each of them that "mxbuild reports CE0582". On the classic client or in migration mode it is now an info-level note that the widget needs converting before the app switches; under the React client it stays a warning. diff --git a/mdl-examples/bug-tests/535-custom-content-column-empty-header.mdl b/mdl-examples/bug-tests/535-custom-content-column-empty-header.mdl new file mode 100644 index 000000000..edbeab8b9 --- /dev/null +++ b/mdl-examples/bug-tests/535-custom-content-column-empty-header.mdl @@ -0,0 +1,36 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#535: a custom-content column with an empty caption showed its MDL +-- column name as the header +-- ============================================================================ +-- +-- Symptom: `column colTriage (Caption: '') { … }` in a Data grid 2 written +-- through the generic `pluggablewidget` form rendered "colTriage" in the column +-- header. mx check, check and lint all passed; only a browser showed it. +-- +-- Cause: the writer filled an empty header on an attribute-less column with the +-- column's name. Mendix stores no name for a column, and Data grid 2 renders the +-- header text as is (`header.trim() || ' '` in Datagrid.mjs, 11.15.0). +-- +-- The generic form needs a name on every column (MDL-WIDGET35), which is how a +-- name was there to leak. +-- +-- After fix: both custom-content columns below store an empty header, as the keyword +-- `datagrid` path already did; `describe page` prints `Caption: ''` for each, +-- and mx check reports 0 errors (11.15.0). + +create module Bug535; + +create persistent entity Bug535.Ticket (Title: String(100)); + +create page Bug535.Tickets (Title: 'Tickets', Layout: Atlas_Core.Atlas_Default) { + pluggablewidget 'com.mendix.widget.web.datagrid.Datagrid' dgTickets (DataSource: database from Bug535.Ticket) { + column colTitle (Attribute: Title, Caption: 'Title') + column colTriage (Caption: '') { + dynamictext dtTriage (Content: 'triage') + } + column colNoCaption { + dynamictext dtNoCaption (Content: 'nocap') + } + } +}; diff --git a/mdl-examples/bug-tests/ledger-54-custom-content-column.mdl b/mdl-examples/bug-tests/ledger-54-custom-content-column.mdl index 09a0220ff..bcb78fc88 100644 --- a/mdl-examples/bug-tests/ledger-54-custom-content-column.mdl +++ b/mdl-examples/bug-tests/ledger-54-custom-content-column.mdl @@ -9,13 +9,16 @@ mdl 1; -- still failed `mx check` with CE0463. A custom-content column requires a -- non-empty header, and the first fix had no attribute name to fall back to. -- --- After fix: the header falls back to the COLUMN'S OWN NAME when there is no --- bound attribute. The fallback is gated on the item template having a `header` --- slot, so header-less object-list items (chart series, accordion groups) are --- never given a spurious header. +-- After fix: the header falls back to the bound attribute's name; a column with +-- no attribute gets an EMPTY header (an empty Forms$ClientTemplate, the shape +-- update-widgets writes), which Data grid 2 renders blank. Round 2 used the +-- column's own name instead, which showed end users "colActionsEmpty" — Mendix +-- stores no name for a column (ako/mxcli#535). The fallback is gated on the item +-- template having a `header` slot, so header-less object-list items (chart +-- series, accordion groups) are never given a spurious header. -- --- Verified: `mx check` → 0 errors on Mendix 11.12.1 for both the empty-caption --- and no-caption custom-content columns (previously CE0463). +-- Verified: `mx check` → 0 errors on Mendix 11.12.1 (column-name header) and +-- 11.15.0 (empty header) for both the empty-caption and no-caption columns. -- -- NOTE: requires the project's generated widget defs (`mxcli widget init`), so -- this is exercised via exec/mx-check, not just syntax check. @@ -43,11 +46,11 @@ create or modify page L54c.Grid ( DataSource: database from L54c.Thing ) { column colName (Attribute: Name, Caption: 'Name') - -- Custom-content column, empty caption → header falls back to the column name + -- Custom-content column, empty caption → empty header (#535) column colActionsEmpty (Caption: '') { actionbutton btnA (Caption: 'Go', Action: call microflow L54c.RowAction(Thing = $currentObject)) } - -- Custom-content column, no caption → header falls back to the column name + -- Custom-content column, no caption → empty header (#535) column colActionsBare { actionbutton btnB (Caption: 'Go2', Action: call microflow L54c.RowAction(Thing = $currentObject)) } diff --git a/mdl/executor/widget_defs_test.go b/mdl/executor/widget_defs_test.go index 49a97d7e2..242ed78b7 100644 --- a/mdl/executor/widget_defs_test.go +++ b/mdl/executor/widget_defs_test.go @@ -729,7 +729,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "attribute", Operation: "attribute", AttributePath: "Mod.Ent.Foo"}, }, } - applyColumnHeaderFallback(spec1, "col1", true) + applyColumnHeaderFallback(spec1, true) if len(spec1.Properties) != 2 { t.Errorf("Case 1 (header present): expected 2 properties, got %d", len(spec1.Properties)) } @@ -741,7 +741,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "staticName", Operation: "primitive", PrimitiveVal: "x"}, }, } - applyColumnHeaderFallback(spec2, "series1", false) + applyColumnHeaderFallback(spec2, false) if len(spec2.Properties) != 1 { t.Errorf("Case 2 (no header slot): expected 1 property, got %d", len(spec2.Properties)) } @@ -752,7 +752,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "attribute", Operation: "attribute", AttributePath: "Mod.Ent.OrderNumber"}, }, } - applyColumnHeaderFallback(spec3, "col3", true) + applyColumnHeaderFallback(spec3, true) if len(spec3.Properties) != 2 { t.Fatalf("Case 3 (fallback): expected 2 properties, got %d", len(spec3.Properties)) } @@ -767,7 +767,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "attribute", Operation: "attribute", AttributePath: "BareName"}, }, } - applyColumnHeaderFallback(spec4, "col4", true) + applyColumnHeaderFallback(spec4, true) if len(spec4.Properties) != 2 || spec4.Properties[1].TextTemplate != "BareName" { t.Errorf("Case 4: fallback for unqualified path = %v", spec4.Properties) } @@ -783,7 +783,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "attribute", Operation: "attribute", AttributePath: "Mod.Ent.Amount"}, }, } - applyColumnHeaderFallback(spec5, "col5", true) + applyColumnHeaderFallback(spec5, true) if len(spec5.Properties) != 2 { t.Fatalf("Case 5 (empty caption): expected 2 properties (no duplicate), got %d", len(spec5.Properties)) } @@ -799,38 +799,42 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "attribute", Operation: "attribute", AttributePath: "Mod.Ent.Amount"}, }, } - applyColumnHeaderFallback(spec6, "col6", true) + applyColumnHeaderFallback(spec6, true) if spec6.Properties[0].TextTemplate != "" { t.Errorf("Case 6: param-bearing header must not be overwritten, got TextTemplate=%q", spec6.Properties[0].TextTemplate) } - // Case 7 (ledger #54, custom-content column): no attribute, empty header — - // fall back to the COLUMN NAME in place (a custom-content column has nothing - // to derive a header from, and an empty header is CE0463). + // Case 7 (ako/mxcli#535): a custom-content column with `Caption: ''` and no + // attribute gets an EMPTY header, never the column's name. Mendix stores no + // name for a column, and Data grid 2 renders an empty header blank — so + // writing the name put an internal identifier ("colTriage") in front of end + // users. The empty header is the empty Forms$ClientTemplate update-widgets + // writes (ChipCoV8 #11), which mx check accepts (0 errors on 11.15.0, the + // keyword `datagrid` path has always written it). spec7 := &backend.ObjectListItemSpec{ Properties: []backend.ObjectListItemProperty{ {PropertyKey: "header", Operation: "texttemplate", TextTemplate: ""}, {PropertyKey: "showContentAs", Operation: "primitive", PrimitiveVal: "customContent"}, }, } - applyColumnHeaderFallback(spec7, "colActions", true) - if h := spec7.Properties[0]; h.PropertyKey != "header" || h.TextTemplate != "colActions" { - t.Errorf("Case 7: empty custom-content header not filled with column name: %+v", spec7.Properties[0]) + applyColumnHeaderFallback(spec7, true) + if h := spec7.Properties[0]; h.PropertyKey != "header" || h.TextTemplate != "" || !h.EmptyTemplate { + t.Errorf("Case 7: empty custom-content header = %+v, want an empty texttemplate with EmptyTemplate=true (not the column name)", spec7.Properties[0]) } - // Case 8 (ledger #54): custom-content column with NO header prop at all — - // append a header carrying the column name (absent header is CE0463 too). + // Case 8 (ako/mxcli#535): custom-content column with NO header prop at all — + // append an empty header, the same as Case 7. spec8 := &backend.ObjectListItemSpec{ Properties: []backend.ObjectListItemProperty{ {PropertyKey: "showContentAs", Operation: "primitive", PrimitiveVal: "customContent"}, }, } - applyColumnHeaderFallback(spec8, "colActions", true) + applyColumnHeaderFallback(spec8, true) if len(spec8.Properties) != 2 { t.Fatalf("Case 8 (absent custom-content header): expected 2 properties, got %d", len(spec8.Properties)) } - if h := spec8.Properties[1]; h.PropertyKey != "header" || h.TextTemplate != "colActions" { - t.Errorf("Case 8: appended header = %+v, want TextTemplate=colActions", spec8.Properties[1]) + if h := spec8.Properties[1]; h.PropertyKey != "header" || h.TextTemplate != "" || !h.EmptyTemplate { + t.Errorf("Case 8: appended header = %+v, want an empty texttemplate with EmptyTemplate=true", spec8.Properties[1]) } // Case 9 (ChipCoV8 #11): an UNNAMED column with no caption and no attribute @@ -844,7 +848,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "showContentAs", Operation: "primitive", PrimitiveVal: "customContent"}, }, } - applyColumnHeaderFallback(spec9, "", true) + applyColumnHeaderFallback(spec9, true) if len(spec9.Properties) != 2 { t.Fatalf("Case 9 (unnamed, empty caption): expected 2 properties (no duplicate), got %d", len(spec9.Properties)) } @@ -859,7 +863,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "showContentAs", Operation: "primitive", PrimitiveVal: "customContent"}, }, } - applyColumnHeaderFallback(spec10, "", true) + applyColumnHeaderFallback(spec10, true) if len(spec10.Properties) != 2 { t.Fatalf("Case 10 (unnamed, absent header): expected 2 properties, got %d", len(spec10.Properties)) } @@ -874,7 +878,7 @@ func TestApplyColumnHeaderFallback(t *testing.T) { {PropertyKey: "staticName", Operation: "texttemplate", TextTemplate: ""}, }, } - applyColumnHeaderFallback(spec11, "", false) + applyColumnHeaderFallback(spec11, false) if len(spec11.Properties) != 1 || spec11.Properties[0].EmptyTemplate { t.Errorf("Case 11 (no header slot): must be untouched, got %+v", spec11.Properties) } diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index cfb0770f1..0b30c6b52 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -2299,11 +2299,10 @@ func (e *PluggableWidgetEngine) buildObjectListItem(mapping *ObjectListMapping, // DataGrid column header convention: when the user provides no Caption, // fall back to the bound attribute name — or, for a custom-content column - // with no attribute, the column's own name. Mirrors buildDataGrid2ColumnObject - // in datagrid_builder.go. Without this, an empty/absent column header trips - // Studio Pro's CE0463 (ledger #54). Only applies to items whose template has - // a `header` slot (datagrid columns), never to header-less object-list items - // like chart series. + // with no attribute, an empty header. Mirrors the keyword path + // (datagrid_column.go). Only applies to items whose template has a `header` + // slot (datagrid columns), never to header-less object-list items like chart + // series. hasHeaderSlot := false for _, ip := range mapping.ItemProperties { if strings.EqualFold(ip.PropertyKey, "header") { @@ -2311,7 +2310,7 @@ func (e *PluggableWidgetEngine) buildObjectListItem(mapping *ObjectListMapping, break } } - applyColumnHeaderFallback(&spec, child.Name, hasHeaderSlot) + applyColumnHeaderFallback(&spec, hasHeaderSlot) // DataGrid column sortable convention: attribute-less columns (typical // "Actions" custom-content columns) default to sortable=false, since @@ -2400,12 +2399,20 @@ func (e *PluggableWidgetEngine) buildObjectListItem(mapping *ObjectListMapping, // item: the slot keyed "content" if present, otherwise the first slot. // Returns "" if there are no widgets-typed slots. // applyColumnHeaderFallback synthesizes a texttemplate spec for the `header` -// property when the spec has an attribute binding but no header text. Mirrors -// the keyword path's `if caption == "" { caption = col.Attribute }` fallback -// in datagrid_builder.go. The check is conservative: only fires when the -// header slot is genuinely empty, so it's safe to call for any object-list -// item kind. -func applyColumnHeaderFallback(spec *backend.ObjectListItemSpec, columnName string, hasHeaderSlot bool) { +// property when the spec has no header text: the bound attribute's leaf name, or +// — for a column with no attribute — an empty header. Mirrors the keyword path's +// `if caption == "" { caption = col.Attribute }` in datagrid_column.go. The check +// is conservative: only fires when the header slot is genuinely empty, so it's +// safe to call for any object-list item kind. +// +// A custom-content column used to fall back to the column's own NAME (ledger +// #54 round 2), on the belief that an empty header is CE0463. It is not: the +// CE0463 was a null TextTemplate, and the empty Forms$ClientTemplate that +// update-widgets writes builds clean (ChipCoV8 #11; 0 errors on 11.15.0, where +// the keyword path writes exactly that). The name, meanwhile, is one Mendix +// does not store for a column, and Data grid 2 renders the header text as is — +// so `Caption: ”` showed end users "colTriage" (ako/mxcli#535). +func applyColumnHeaderFallback(spec *backend.ObjectListItemSpec, hasHeaderSlot bool) { // Only items whose template has a `header` slot (datagrid columns) get a // header fallback; header-less object-list items (chart series, accordion // groups) are left untouched. @@ -2420,10 +2427,7 @@ func applyColumnHeaderFallback(spec *backend.ObjectListItemSpec, columnName stri case "header": headerIdx = i // An explicit `Caption: ''` reaches here as a texttemplate with no - // text and no params. Studio Pro rejects an empty column header with - // CE0463 "widget definition changed" (ledger #54) — the same failure - // an absent header would cause without this fallback. Treat empty as - // absent so the fallback applies either way. + // text and no params; treat it like an absent header. headerEmpty = p.Operation == "texttemplate" && p.TextTemplate == "" && len(p.Parameters) == 0 case "attribute": attrPath = p.AttributePath @@ -2433,23 +2437,17 @@ func applyColumnHeaderFallback(spec *backend.ObjectListItemSpec, columnName stri if headerIdx >= 0 && !headerEmpty { return } - // Fallback header text: the bound attribute's leaf name (Module.Entity.Attr → - // Attr), or — for a custom-content / action column with no attribute — the - // column's own name. Every datagrid column needs a NON-empty header; an empty - // or absent one is CE0463 (ledger #54), and a custom-content column has no - // attribute to derive one from, so the column name is the sensible default. + // Fallback header text: the bound attribute's leaf name + // (Module.Entity.Attr → Attr). fallback := attrPath if idx := strings.LastIndex(fallback, "."); idx >= 0 { fallback = fallback[idx+1:] } if fallback == "" { - fallback = columnName - } - if fallback == "" { - // An unnamed column with no caption and no attribute has nothing to - // derive a header from. It must still not serialize as TextTemplate: - // null — `mx update-widgets` writes an empty Forms$ClientTemplate for - // the header, so null is CE0463 (ChipCoV8 #11). Flag the header + // No attribute to name the column after: an empty header, which Data + // grid 2 renders blank. It must not serialize as TextTemplate: null — + // `mx update-widgets` writes an empty Forms$ClientTemplate for the + // header, so null is CE0463 (ChipCoV8 #11). Flag the header // EmptyTemplate; scoped by the hasHeaderSlot gate above to datagrid // columns, never to other texttemplates. if headerIdx >= 0 { From 3bf444a5d005575fa162d36ae6b96c987086fca1 Mon Sep 17 00:00:00 2001 From: Ako Date: Sun, 1 Nov 2026 19:06:28 +0000 Subject: [PATCH 55/61] Add SEC011: a page XPath through an association the page's roles cannot read At Production security the runtime checks member access on every association step of a page data source's XPath, on the entity the step starts from, and refuses the retrieve: the page opens to an error dialog, "No access rights for Shop$Account/Shop.Order_Account". mxbuild, check and lint were all clean. Measured on 11.15.0 with run --local and playwright check, as a user role holding one module role: - owner Both, traversed from the TO entity, member at None: refused; - default owner, traversed from the FROM entity, member at None: refused; - default owner, traversed from the TO entity (not a member there): opens; - the same XPath inside an entity access rule: opens; - control, members granted ReadOnly: every page opens. SEC011 reports exactly the refused cases, per user role that can open the page and reads the starting entity, as an error naming the member to grant. Live, it flags the two refused pages of the measured app and nothing on the control; on FactoryManagement (Production, 27 page XPaths with paths) it reports none. Closes #945 Closes #1337 Co-Authored-By: Claude Opus 5.5 --- .claude/commands/mendix/lint.md | 1 + .claude/skills/mendix/assess-quality/SKILL.md | 21 +- .../skills/mendix/manage-security/SKILL.md | 15 +- CHANGELOG.md | 1 + cmd/mxcli/cmd_lint.go | 2 + docs-site/src/tools/builtin-rules.md | 1 + docs-site/src/tools/linting.md | 2 +- mdl/linter/report.go | 1 + mdl/linter/rules/xpath_association_access.go | 278 ++++++++++++++++++ .../rules/xpath_association_access_test.go | 201 +++++++++++++ 10 files changed, 509 insertions(+), 14 deletions(-) create mode 100644 mdl/linter/rules/xpath_association_access.go create mode 100644 mdl/linter/rules/xpath_association_access_test.go diff --git a/.claude/commands/mendix/lint.md b/.claude/commands/mendix/lint.md index ce4e60102..8fff31735 100644 --- a/.claude/commands/mendix/lint.md +++ b/.claude/commands/mendix/lint.md @@ -50,6 +50,7 @@ mxcli lint -p app.mpr --baseline lint-baseline.json | SEC002 | security | WeakPasswordPolicy - Password minimum length should be 8+ | | SEC003 | security | DemoUsersActive - Demo users should be off at Production security | | SEC010 | security | PageEntityNotReadable - A page data widget lists an entity the user roles that open the page cannot read; at Production it renders empty, silently (info when only row-scoped) | +| SEC011 | security | XPathAssociationNotReadable - A page data source's XPath crosses an association a role that opens the page may not read; at Production the page fails to open ("No access rights for …") | | CONV011 | performance | NoCommitInLoop - Commits inside loops (commit actions, or create/change with commit) cause N+1 issues | | CONV012 | quality | ExclusiveSplitCaption - Exclusive splits need meaningful captions | | CONV013 | quality | ErrorHandlingOnCalls - External calls (REST/WS/Java) need custom error handling | diff --git a/.claude/skills/mendix/assess-quality/SKILL.md b/.claude/skills/mendix/assess-quality/SKILL.md index 6a879c8ca..d2deb96d8 100644 --- a/.claude/skills/mendix/assess-quality/SKILL.md +++ b/.claude/skills/mendix/assess-quality/SKILL.md @@ -12,14 +12,14 @@ description: "Audit a Mendix project against best practices for naming, security |---|---| | 29-36 | When to Use This Skill | | 37-96 | Assessment Workflow | -| 97-354 | Quality Guidelines Reference | -| 355-365 | Assessment Report Template | -| 366-378 | Executive Summary | -| 379-389 | project Metrics | -| 390-402 | Automated Findings | -| 403-422 | Manual Review Findings | -| 423-438 | Recommendations | -| 439-449 | Sources | +| 97-355 | Quality Guidelines Reference | +| 356-366 | Assessment Report Template | +| 367-379 | Executive Summary | +| 380-390 | project Metrics | +| 391-403 | Automated Findings | +| 404-423 | Manual Review Findings | +| 424-439 | Recommendations | +| 440-450 | Sources | # Assess Mendix Project Quality @@ -152,7 +152,7 @@ After reviewing automated results, assess the following areas manually. The guid ### C. Security -**Enforced by rules:** SEC001-SEC010, CONV006, CONV007, CONV008, MPR007 +**Enforced by rules:** SEC001-SEC011, CONV006, CONV007, CONV008, MPR007 #### C.1 Entity Access @@ -160,6 +160,7 @@ After reviewing automated results, assess the following areas manually. The guid |-----------|------|----------| | All persistent entities must have access rules | SEC001 | Critical | | Every role that opens a page can read the entities its grids, galleries and combo boxes list (else it renders empty at Production) | SEC010 | High | +| Every association a page data source's XPath crosses is readable by the roles that open the page, on the entity the step starts from (else the page fails to open at Production) | SEC011 | High | | Do not grant Create or Delete rights on entity access rules | CONV006 | High | | All entity access rules should have XPath constraints | CONV007 | High | | Set default access rights to None | Manual check | High | @@ -441,7 +442,7 @@ Overall Score: [X]/100 (from `mxcli report`) This skill incorporates guidelines from: - **Conventions.pdf** — Squad Apps internal best practices (14 categories, 80+ guidelines) - **CONV001-CONV017 lint rules** — Automated checks derived from Conventions.pdf -- **MPR001-MPR007, SEC001-SEC010** — Built-in linter rules +- **MPR001-MPR007, SEC001-SEC011** — Built-in linter rules - **ARCH, DESIGN, QUAL series** — Starlark architecture/quality rules - **Mendix Performance Best Practices** — Official Mendix documentation on performance optimization - **Mendix Security Best Practices** — Official Mendix documentation on security configuration diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index e75335f11..bdb44f8d8 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -15,9 +15,9 @@ description: "Configure Mendix security in MDL — module and user roles, access | 44-120 | The System-module ceiling — decide the role model around this FIRST | | 121-560 | Syntax Reference | | 561-575 | Starlark Lint Rule APIs | -| 576-628 | Common Workflow: Setting Up Module Security | -| 629-638 | Common Mistakes | -| 639-648 | Validation | +| 576-637 | Common Workflow: Setting Up Module Security | +| 638-647 | Common Mistakes | +| 648-657 | Validation | # Security Management Skill @@ -626,6 +626,15 @@ Audit members are the one case where naming a member cannot change its rights: member access to write for it and a rule that carries one fails CE0066. Let the rule's default cover it, or change the default. +**A page data source's XPath needs the association member too.** At Production the +runtime checks member access on every association step of a data-source XPath, on the +entity the step starts from: `database from M.Account where [M.Order_Account/M.Order/IsActive += true()]` with `M.Order_Account` at None in Account's rule (owner Both) opens to an error +dialog, `No access rights for M$Account/M.Order_Account`, while mxbuild reports nothing. +A step from an entity the association is not a member of (the TO side of an `OWNER +Default` association) is not checked, and neither is an XPath inside an access rule +(measured, 11.15). `mxcli lint` reports it as SEC011. + ## Common Mistakes 1. **Creating module roles before the module exists** — `create module` must come first diff --git a/CHANGELOG.md b/CHANGELOG.md index c0344d3bb..9df7b833c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **SEC011: a page data source whose XPath crosses an association the page's roles cannot read** (ako/mxcli#945, #1337, mendixlabs/mxcli#1471) — at Production security such a page opens to an error dialog (`No access rights for Shop$Account/Shop.Order_Account`) while mxbuild, `check` and `lint` were all clean. The rule checks each association step from the entity it starts on, per user role that can open the page, and reports it as an error with the member to grant. Measured on 11.15: an association is checked only where it is a member (its FROM entity, and its TO entity when the owner is Both), and an XPath inside an entity access rule is not checked at all, so neither is reported. - **DROP over `--mcp` uses Studio Pro 11.16's `ped_delete_document`** — DROP of a constant, enumeration, microflow, nanoflow, page, workflow or business-event service no longer needs `--mcp-concord` when Studio Pro advertises the native delete tool; Concord stays as the fallback for older servers and for a native call that fails. - **`mxcli lint --baseline ` reports only new findings** (ako/mxcli#1328, mendixlabs/mxcli#1455) — record a baseline with `--update-baseline`, commit it, and later runs report only the findings it does not have, with an exit code that follows them alone and a `N new, M already known, K fixed` summary on stderr. Findings are matched on rule and document, not message, so rewording a rule does not make its findings new; a scoped run (`-m`, `-d`, `-r`) reports new findings but cannot update the baseline. - **Lint permission rows name their access rule** (ako/mxcli#1437, mendixlabs/mxcli#1539) — `permissions()` and `permissions_for()` rows carry `access_rule_id` and `access_rule_index`, so a rule can count an entity's access rules per setting or list one rule's module roles exactly, instead of guessing by grouping on XPath, default and rights. The catalog schema moves to version 31, so a cached catalog is rebuilt once. diff --git a/cmd/mxcli/cmd_lint.go b/cmd/mxcli/cmd_lint.go index 66a2ab5d1..835455eed 100644 --- a/cmd/mxcli/cmd_lint.go +++ b/cmd/mxcli/cmd_lint.go @@ -34,6 +34,7 @@ Built-in rules check for: - Password policy (SEC002) - password minimum length should be 8+ - Demo users (SEC003) - demo users should be off at Production security - Unreadable page data (SEC010) - a data widget's entity must be readable by the roles that open its page (at Production it renders empty) + - XPath through an unreadable association (SEC011) - a page data source's XPath may cross only associations the roles opening the page can read (at Production the page fails) - Un-describable branch structure (MDL-FLOW01) - decision branches that do not nest, so DESCRIBE cannot render them faithfully Bundled Starlark rules (in .claude/lint-rules/): @@ -486,6 +487,7 @@ func builtinLintRules() []linter.Rule { rules.NewWeakPasswordPolicyRule(), rules.NewDemoUsersActiveRule(), rules.NewPageEntityReadAccessRule(), // SEC010 - full catalog (widgets, permissions) + rules.NewXPathAssociationAccessRule(), // SEC011 - full catalog (xpath_expressions, permissions) rules.NewOverlappingActivitiesRule(), // MPR008 - requires BSON inspection rules.NewLoopChildContainmentRule(), // MPR011 - requires BSON inspection rules.NewNoCommitInLoopRule(), // CONV011-CONV014 - require BSON inspection diff --git a/docs-site/src/tools/builtin-rules.md b/docs-site/src/tools/builtin-rules.md index ba8990a4e..1caa05338 100644 --- a/docs-site/src/tools/builtin-rules.md +++ b/docs-site/src/tools/builtin-rules.md @@ -30,6 +30,7 @@ The **lint rules** below run with `mxcli lint`. There is also a separate group o | **SEC002** | Password policy -- Checks for secure password configuration | | **SEC003** | Demo users -- Warns about demo users in production security level | | **SEC010** | Unreadable page data -- A data grid, gallery, list view or combo box lists an entity that a user role allowed on the page cannot read (including its generalizations and specializations). At Production security it renders empty for that role and nothing else reports it. Info when the role reads only through an XPath-constrained rule; System entities use access measured from a deployed model, so a combo box over System.User gets its own message (it lists only the current user) | +| **SEC011** | XPath through an unreadable association -- A page data source's XPath crosses an association that a user role allowed on the page may not read on the entity the step starts from (an association is a member of its FROM entity's rules, and of its TO entity's when the owner is Both). At Production security the page fails to open with "No access rights for Module$Entity/Module.Association", while mxbuild reports nothing. Error. Entity access-rule XPaths are not checked: the runtime does not apply member access there (measured, 11.15) | ## Convention Rules diff --git a/docs-site/src/tools/linting.md b/docs-site/src/tools/linting.md index 4183f83e4..eb2c49c4a 100644 --- a/docs-site/src/tools/linting.md +++ b/docs-site/src/tools/linting.md @@ -17,7 +17,7 @@ The linting system provides: | Category | Prefix | Focus | |----------|--------|-------| | MDL | MDL001-MDL007 | Naming conventions, empty microflows, domain model size | -| Security | SEC001-SEC010 | Access rules, password policy, demo users, PII exposure | +| Security | SEC001-SEC011 | Access rules, password policy, demo users, PII exposure | | Convention | CONV001-CONV021 | Best practice conventions, error handling | | Quality | QUAL001-QUAL004 | Complexity, documentation, long microflows | | Architecture | ARCH001-ARCH003 | Cross-module data, entity business keys | diff --git a/mdl/linter/report.go b/mdl/linter/report.go index 104c12f2c..96d663003 100644 --- a/mdl/linter/report.go +++ b/mdl/linter/report.go @@ -56,6 +56,7 @@ var categoryMapping = map[string]string{ "SEC008": "Security", "SEC009": "Security", "SEC010": "Security", + "SEC011": "Security", "CONV006": "Security", "CONV007": "Security", "CONV008": "Security", diff --git a/mdl/linter/rules/xpath_association_access.go b/mdl/linter/rules/xpath_association_access.go new file mode 100644 index 000000000..2462908eb --- /dev/null +++ b/mdl/linter/rules/xpath_association_access.go @@ -0,0 +1,278 @@ +// SPDX-License-Identifier: Apache-2.0 + +package rules + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// XPathAssociationAccessRule (SEC011) reports a page data source whose XPath +// crosses an association that a user role able to open the page may not read +// (ako/mxcli#945, #1337). +// +// At Production security the runtime checks member access on every association +// step of a data-source XPath, on the entity the step starts from, and refuses +// the whole retrieve when the role cannot read it — the page opens to an error +// dialog and the log says +// +// SecurityRuntimeException: No access rights for Shop$Account/Shop.Order_Account +// +// mxbuild reports 0 errors (it checks bound paths, CE2729, not paths inside an +// XPath filter), and so did check, exec and lint. +// +// # What was measured (11.15.0, Production, a user role holding one module role) +// +// - An association is a member of its FROM entity's access rules, and of its +// TO entity's too when its owner is Both. Traversed from an entity it is a +// member of, with that member at None: refused — from the TO side of an +// owner-Both association and from the FROM side of a default one alike. +// - Traversed from an entity it is NOT a member of (the TO side of a default +// association): the page opens. Nothing to check, so nothing is reported. +// - The same XPath inside an entity ACCESS RULE whose own rule leaves the +// member at None: the page opens. Access-rule XPaths are left out. +// - Control: with the members granted ReadOnly, every page opens. +// +// Left out because unmeasured: a step whose membership comes from a +// generalization of the entity it starts on; snippets (which roles open them +// depends on the pages that call them); microflow retrieves, which apply entity +// access only when asked to and are not page data sources. +// +// A role that may not read the step's starting entity at all is not reported +// here: for the widget's own entity that is SEC010's finding, and the step +// itself is not where the runtime stops. +type XPathAssociationAccessRule struct{} + +// NewXPathAssociationAccessRule creates the SEC011 rule. +func NewXPathAssociationAccessRule() *XPathAssociationAccessRule { + return &XPathAssociationAccessRule{} +} + +func (r *XPathAssociationAccessRule) ID() string { return "SEC011" } +func (r *XPathAssociationAccessRule) Name() string { return "XPathAssociationNotReadable" } +func (r *XPathAssociationAccessRule) Category() string { return "security" } +func (r *XPathAssociationAccessRule) DefaultSeverity() linter.Severity { return linter.SeverityError } + +func (r *XPathAssociationAccessRule) Description() string { + return "Checks that a page data source's XPath crosses only associations the roles opening the page may read (at Production the page fails to open: \"No access rights for …\")" +} + +// RequiredCatalogMode: xpath_expressions, widgets and permissions are written +// by REFRESH CATALOG FULL. +func (r *XPathAssociationAccessRule) RequiredCatalogMode() linter.CatalogMode { + return linter.CatalogFull +} + +// assocEnds is one association's ends and owner, as the catalog records them. +type assocEnds struct { + from, to, owner string +} + +// memberOf reports whether the association is a member of entity's access +// rules: of its FROM entity always, of its TO entity when the owner is Both. +func (a assocEnds) memberOf(entity string) bool { + return entity == a.from || (strings.EqualFold(a.owner, "Both") && entity == a.to) +} + +// other returns the entity a step from entity lands on. +func (a assocEnds) other(entity string) string { + if entity == a.from { + return a.to + } + if entity == a.to { + return a.from + } + return "" +} + +// hop is one association step and the entity it starts from. +type hop struct { + entity, association string +} + +// Check reports each (page data source, association step, user role) where the +// role opens the page, reads the step's starting entity, and not the member. +func (r *XPathAssociationAccessRule) Check(ctx *linter.LintContext) []linter.Violation { + reader := ctx.Reader() + if reader == nil { + return nil + } + ps, err := reader.GetProjectSecurity() + if err != nil || ps == nil || ps.SecurityLevel != security.SecurityLevelProduction || len(ps.UserRoles) == 0 { + return nil + } + + assocs := map[string]assocEnds{} + if rows, err := ctx.Query(`SELECT QualifiedName, COALESCE(FromEntity, ''), COALESCE(ToEntity, ''), COALESCE(Owner, '') FROM associations`); err == nil { + for rows.Next() { + var qn string + var a assocEnds + if rows.Scan(&qn, &a.from, &a.to, &a.owner) == nil { + assocs[qn] = a + } + } + rows.Close() + } + if len(assocs) == 0 { + return nil + } + + pageRoles := map[string][]string{} + readsEntity := map[string]map[string]bool{} // entity -> module role -> any READ rule + readsMember := map[string]map[string]bool{} // entity + "\x00" + member -> module role + for p := range ctx.Permissions() { + switch { + case p.ElementType == "PAGE" && p.AccessType == "VIEW": + pageRoles[p.ElementName] = append(pageRoles[p.ElementName], p.ModuleRoleName) + case p.ElementType == "ENTITY" && p.AccessType == "READ" && p.MemberName == "": + addRole(readsEntity, p.ElementName, p.ModuleRoleName) + case p.ElementType == "ENTITY" && p.AccessType == "MEMBER_READ" && p.MemberName != "": + addRole(readsMember, p.ElementName+"\x00"+p.MemberName, p.ModuleRoleName) + } + } + + widgetsOn := map[string]map[string][]string{} // page -> entity -> data widget names + for w := range ctx.Widgets() { + if w.EntityRef == "" || !strings.EqualFold(w.ContainerType, "page") { + continue + } + if _, ok := dataWidgetLabel(w.WidgetType); !ok { + continue + } + if widgetsOn[w.ContainerQualifiedName] == nil { + widgetsOn[w.ContainerQualifiedName] = map[string][]string{} + } + widgetsOn[w.ContainerQualifiedName][w.EntityRef] = append(widgetsOn[w.ContainerQualifiedName][w.EntityRef], w.Name) + } + + var violations []linter.Violation + for x := range ctx.XPathExpressions() { + if x.DocumentType != "PAGE" || x.ComponentType != "WIDGET" || x.TargetEntity == "" { + continue + } + roles := pageRoles[x.DocumentQualifiedName] + if len(roles) == 0 { + continue // no role may open it: MPR007 territory + } + for _, h := range associationHops(x.XPathExpression, x.TargetEntity, assocs) { + if !assocs[h.association].memberOf(h.entity) { + continue // not a member there: measured, the runtime does not refuse it + } + var blind []string + for _, ur := range ps.UserRoles { + if !sharesRole(ur.ModuleRoles, roles) || !anyRole(readsEntity[h.entity], ur.ModuleRoles) { + continue + } + if anyRole(readsMember[h.entity+"\x00"+h.association], ur.ModuleRoles) { + continue + } + blind = append(blind, ur.Name) + } + if len(blind) > 0 { + violations = append(violations, r.violation(x, h, blind, widgetsOn[x.DocumentQualifiedName][x.TargetEntity])) + } + } + } + return violations +} + +func (r *XPathAssociationAccessRule) violation(x linter.XPathExpressionEntry, h hop, blind, widgets []string) linter.Violation { + sort.Strings(blind) + source := "A data source" + if len(widgets) == 1 { + // Named only when the page has exactly one data widget over the entity; + // the catalog does not tie an XPath to its widget. + source = fmt.Sprintf("The data source of '%s'", widgets[0]) + } + entityName := h.entity[strings.LastIndex(h.entity, ".")+1:] + module := moduleFromQualified(h.entity) + return linter.Violation{ + RuleID: r.ID(), + Severity: r.DefaultSeverity(), + Message: fmt.Sprintf("%s on page %s filters through %s from %s, which user role%s %s can open the page but may not read on %s — "+ + "at Production the page fails to open: \"No access rights for %s$%s/%s\"", + source, x.DocumentQualifiedName, h.association, h.entity, plural(blind), quoteList(blind), h.entity, + module, entityName, h.association), + Location: linter.Location{ + Module: moduleFromQualified(x.DocumentQualifiedName), + DocumentType: "page", + DocumentName: docNameFromQualified(x.DocumentQualifiedName), + DocumentID: x.DocumentID, + }, + Suggestion: fmt.Sprintf("Grant read on the %s member in %s's access rule for a module role those user roles hold "+ + "(e.g. GRANT READ (…, %s) ON ENTITY %s TO , repeating that rule's XPath so the grant merges into it), or filter without crossing it", + h.association, h.entity, h.association[strings.LastIndex(h.association, ".")+1:], h.entity), + } +} + +func addRole(m map[string]map[string]bool, key, role string) { + if m[key] == nil { + m[key] = map[string]bool{} + } + m[key][role] = true +} + +func anyRole(granted map[string]bool, roles []string) bool { + for _, r := range roles { + if granted[r] { + return true + } + } + return false +} + +// associationHops returns every association step in constraint, each with the +// entity it starts from, walking from target. A group mxcli cannot parse, and +// any step whose entity the walk loses (a variable root, an unknown name), is +// skipped: the rule reports only steps it can place. +func associationHops(constraint, target string, assocs map[string]assocEnds) []hop { + var hops []hop + for _, group := range visitor.SplitXPathPredicateGroups(constraint) { + expr, ok := visitor.ParseXPathConstraint(group) + if !ok || expr == nil { + continue + } + walkHops(expr, target, assocs, &hops) + } + return hops +} + +func walkHops(expr ast.Expression, cur string, assocs map[string]assocEnds, hops *[]hop) { + switch e := expr.(type) { + case *ast.XPathPathExpr: + for _, st := range e.Steps { + next := "" + if q, ok := st.Expr.(*ast.QualifiedNameExpr); ok && cur != "" { + qn := q.QualifiedName.String() + if a, ok := assocs[qn]; ok { + if next = a.other(cur); next != "" { + *hops = append(*hops, hop{entity: cur, association: qn}) + } + } else { + next = qn // an entity step names where the path is + } + } + for _, pred := range st.Predicates { + walkHops(pred, next, assocs, hops) + } + cur = next + } + case *ast.BinaryExpr: + walkHops(e.Left, cur, assocs, hops) + walkHops(e.Right, cur, assocs, hops) + case *ast.UnaryExpr: + walkHops(e.Operand, cur, assocs, hops) + case *ast.ParenExpr: + walkHops(e.Inner, cur, assocs, hops) + case *ast.FunctionCallExpr: + for _, a := range e.Arguments { + walkHops(a, cur, assocs, hops) + } + } +} diff --git a/mdl/linter/rules/xpath_association_access_test.go b/mdl/linter/rules/xpath_association_access_test.go new file mode 100644 index 000000000..08e1fbe42 --- /dev/null +++ b/mdl/linter/rules/xpath_association_access_test.go @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: Apache-2.0 + +package rules + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// ako/mxcli#945, #1337. Measured on 11.15.0 at Production security, signed in +// as a user role holding only Shop.RoleA, each case below a Data grid 2 over a +// database source whose XPath crosses one association: +// +// - owner Both, traversed from its TO entity (Account), whose rule leaves the +// association at None: the page fails, "No access rights for +// Shop$Account/Shop.Order_Account". +// - owner Default, traversed from its FROM entity, member at None: fails, +// "No access rights for Shop$Order3/Shop.Order3_Account". +// - owner Default, traversed from its TO entity: the association is not a +// member there, and the page opens. +// - the same XPath inside an entity ACCESS RULE whose own member is at None: +// the page opens — the runtime does not check member access there. +// +// Control: with both members granted ReadOnly, all pages open. + +func (f *readAccessFixture) association(qn, from, to, owner string) { + f.n++ + mod, name, _ := strings.Cut(qn, ".") + f.exec(`INSERT INTO associations_data (Id, Name, QualifiedName, ModuleName, FromEntity, ToEntity, Owner, ProjectId, SnapshotId) + VALUES (?,?,?,?,?,?,?,?,?)`, "a"+string(rune('a'+f.n)), name, qn, mod, from, to, owner, "default", "s1") +} + +func (f *readAccessFixture) xpath(documentType, document, componentType, xpath, target string) { + f.n++ + f.exec(`INSERT INTO xpath_expressions_data (Id, DocumentType, DocumentId, DocumentQualifiedName, ComponentType, ComponentId, + XPathExpression, TargetEntity, UsageType, ModuleName, ProjectId, SnapshotId) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)`, + "x"+string(rune('a'+f.n)), documentType, "d-"+document, document, componentType, "c"+string(rune('a'+f.n)), + xpath, target, "DATASOURCE", moduleFromQualified(document), "default", "s1") +} + +func (f *readAccessFixture) checkXPathAccess() []linter.Violation { + f.t.Helper() + ctx := linter.NewLintContext(f.cat, navSecurityReader{sec: f.sec}) + return NewXPathAssociationAccessRule().Check(ctx) +} + +func shopUser() *security.UserRole { + return &security.UserRole{Name: "RoleAUser", ModuleRoles: []string{"System.User", "Shop.RoleA"}} +} + +// shopFixture is the measured project: Account and Order, Order_Account owner +// Both, RoleA reading Account.Code only and all of Order. +func shopFixture(t *testing.T, level string) *readAccessFixture { + f := newReadAccessFixture(t, level, shopUser()) + f.entity("Shop.Account", "") + f.entity("Shop.Order", "") + f.association("Shop.Order_Account", "Shop.Order", "Shop.Account", "Both") + f.read("Shop.Account", "Shop.RoleA", "") + f.memberRead("Shop.Account", "Shop.RoleA", "", "Shop.Account.Code") + f.read("Shop.Order", "Shop.RoleA", "") + f.memberRead("Shop.Order", "Shop.RoleA", "", "Shop.Order.IsActive") + f.memberRead("Shop.Order", "Shop.RoleA", "", "Shop.Order_Account") + f.page("Shop.Portal", "Shop.RoleA") + f.widget("Shop.Portal", "dgOrders", dataGrid, "Shop.Account") + f.xpath("PAGE", "Shop.Portal", "WIDGET", "[Shop.Order_Account/Shop.Order/IsActive = true()]", "Shop.Account") + return f +} + +func TestXPathAssociationAccess_OwnerBothFromTheToSideFails(t *testing.T) { + v := oneViolation(t, shopFixture(t, security.SecurityLevelProduction).checkXPathAccess()) + if v.RuleID != "SEC011" || v.Severity != linter.SeverityError { + t.Errorf("rule/severity = %s %v, want SEC011 error — the page fails to open", v.RuleID, v.Severity) + } + for _, want := range []string{"'dgOrders'", "Shop.Portal", "Shop.Order_Account", "Shop.Account", "'RoleAUser'", "No access rights for Shop$Account/Shop.Order_Account"} { + if !strings.Contains(v.Message, want) { + t.Errorf("message %q does not mention %s", v.Message, want) + } + } + if v.Location.DocumentType != "page" || v.Location.DocumentName != "Portal" || v.Location.Module != "Shop" { + t.Errorf("location = %+v", v.Location) + } + if !strings.Contains(v.Suggestion, "Order_Account") || !strings.Contains(v.Suggestion, "Shop.Account") { + t.Errorf("suggestion %q does not say which member to grant on which entity", v.Suggestion) + } +} + +// The control: the member granted on the entity the step starts from. +func TestXPathAssociationAccess_GrantedMemberIsSilent(t *testing.T) { + f := shopFixture(t, security.SecurityLevelProduction) + f.memberRead("Shop.Account", "Shop.RoleA", "", "Shop.Order_Account") + if vs := f.checkXPathAccess(); len(vs) != 0 { + t.Errorf("want nothing once RoleA reads Order_Account on Account, got %+v", vs) + } +} + +// Granted by another of the user role's module roles counts too: what a person +// logs in as is the user role. +func TestXPathAssociationAccess_AnyModuleRoleOfTheUserRoleCounts(t *testing.T) { + f := shopFixture(t, security.SecurityLevelProduction) + f.sec.UserRoles[0].ModuleRoles = append(f.sec.UserRoles[0].ModuleRoles, "Shop.Auditor") + f.memberRead("Shop.Account", "Shop.Auditor", "[Code != empty]", "Shop.Order_Account") + if vs := f.checkXPathAccess(); len(vs) != 0 { + t.Errorf("want nothing, Shop.Auditor reads the member: %+v", vs) + } +} + +func TestXPathAssociationAccess_FromTheFromSideFails(t *testing.T) { + f := newReadAccessFixture(t, security.SecurityLevelProduction, shopUser()) + f.entity("Shop.Account", "") + f.entity("Shop.Order3", "") + f.association("Shop.Order3_Account", "Shop.Order3", "Shop.Account", "Default") + f.read("Shop.Order3", "Shop.RoleA", "") + f.memberRead("Shop.Order3", "Shop.RoleA", "", "Shop.Order3.IsActive") + f.read("Shop.Account", "Shop.RoleA", "") + f.page("Shop.P_Forward", "Shop.RoleA") + f.widget("Shop.P_Forward", "dgForward", dataGrid, "Shop.Order3") + f.xpath("PAGE", "Shop.P_Forward", "WIDGET", "[Shop.Order3_Account/Shop.Account/Code = 'x']", "Shop.Order3") + + v := oneViolation(t, f.checkXPathAccess()) + if !strings.Contains(v.Message, "No access rights for Shop$Order3/Shop.Order3_Account") { + t.Errorf("message = %q", v.Message) + } +} + +// Measured: a Default-owner association is not a member of its TO entity, and +// traversing it from there opened the page. Must stay silent. +func TestXPathAssociationAccess_DefaultOwnerFromTheToSideIsSilent(t *testing.T) { + f := newReadAccessFixture(t, security.SecurityLevelProduction, shopUser()) + f.entity("Shop.Account", "") + f.entity("Shop.Order2", "") + f.association("Shop.Order2_Account", "Shop.Order2", "Shop.Account", "Default") + f.read("Shop.Account", "Shop.RoleA", "") + f.memberRead("Shop.Account", "Shop.RoleA", "", "Shop.Account.Code") + f.read("Shop.Order2", "Shop.RoleA", "") + f.page("Shop.P_Default", "Shop.RoleA") + f.widget("Shop.P_Default", "dgDefault", dataGrid, "Shop.Account") + f.xpath("PAGE", "Shop.P_Default", "WIDGET", "[Shop.Order2_Account/Shop.Order2/IsActive = true()]", "Shop.Account") + + if vs := f.checkXPathAccess(); len(vs) != 0 { + t.Errorf("want nothing, got %+v", vs) + } +} + +// Measured: the runtime does not check member access inside an access rule's +// own XPath. This pins the measured outcome; an access rule's XPath has no page +// roles, so it is also silent before the rule's ComponentType filter is reached. +func TestXPathAssociationAccess_AccessRuleXPathIsSilent(t *testing.T) { + f := newReadAccessFixture(t, security.SecurityLevelProduction, shopUser()) + f.entity("Shop.Account4", "") + f.entity("Shop.Order4", "") + f.association("Shop.Order4_Account4", "Shop.Order4", "Shop.Account4", "Both") + f.read("Shop.Account4", "Shop.RoleA", "[Shop.Order4_Account4/Shop.Order4/IsActive = true()]") + f.xpath("DOMAIN_MODEL", "Shop.DomainModel", "ACCESS_RULE", "[Shop.Order4_Account4/Shop.Order4/IsActive = true()]", "Shop.Account4") + + if vs := f.checkXPathAccess(); len(vs) != 0 { + t.Errorf("want nothing, got %+v", vs) + } +} + +// Below Production the runtime does not apply entity access, so nothing is +// refused. +func TestXPathAssociationAccess_PrototypeIsSilent(t *testing.T) { + if vs := shopFixture(t, security.SecurityLevelPrototype).checkXPathAccess(); len(vs) != 0 { + t.Errorf("want nothing below Production, got %+v", vs) + } +} + +// A role that cannot open the page is not asked; one that cannot read the +// starting entity at all is SEC010's finding, not this rule's. +func TestXPathAssociationAccess_OnlyRolesThatOpenThePageAndReadTheEntity(t *testing.T) { + f := shopFixture(t, security.SecurityLevelProduction) + f.sec.UserRoles = append(f.sec.UserRoles, &security.UserRole{Name: "Outsider", ModuleRoles: []string{"System.User", "Shop.Other"}}) + if vs := f.checkXPathAccess(); len(vs) != 1 || strings.Contains(vs[0].Message, "Outsider") { + t.Errorf("want only RoleAUser reported, got %+v", vs) + } + + g := shopFixture(t, security.SecurityLevelProduction) + g.exec(`DELETE FROM permissions WHERE ElementName = 'Shop.Account'`) + if vs := g.checkXPathAccess(); len(vs) != 0 { + t.Errorf("RoleA cannot read Shop.Account at all (SEC010's finding); want nothing here, got %+v", vs) + } +} + +// Every hop is checked from the entity it starts on: the second step here +// starts on Order, whose rule leaves Order_Customer at None. +func TestXPathAssociationAccess_SecondHopStartsOnTheIntermediateEntity(t *testing.T) { + f := shopFixture(t, security.SecurityLevelProduction) + f.memberRead("Shop.Account", "Shop.RoleA", "", "Shop.Order_Account") + f.entity("Shop.Customer", "") + f.association("Shop.Order_Customer", "Shop.Order", "Shop.Customer", "Default") + f.read("Shop.Customer", "Shop.RoleA", "") + f.xpath("PAGE", "Shop.Portal", "WIDGET", "[Shop.Order_Account/Shop.Order[Shop.Order_Customer/Shop.Customer/Name = 'x']]", "Shop.Account") + + v := oneViolation(t, f.checkXPathAccess()) + if !strings.Contains(v.Message, "Shop$Order/Shop.Order_Customer") { + t.Errorf("message = %q, want the hop from Shop.Order", v.Message) + } +} From 79beb6c2ed5ec160eea4068a05071d7af923663e Mon Sep 17 00:00:00 2001 From: Ako Date: Sun, 1 Nov 2026 19:36:12 +0000 Subject: [PATCH 56/61] Format cmd_lint.go Co-Authored-By: Claude Opus 5.5 --- cmd/mxcli/cmd_lint.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cmd/mxcli/cmd_lint.go b/cmd/mxcli/cmd_lint.go index 835455eed..02d609cd6 100644 --- a/cmd/mxcli/cmd_lint.go +++ b/cmd/mxcli/cmd_lint.go @@ -486,11 +486,11 @@ func builtinLintRules() []linter.Rule { rules.NewNoEntityAccessRulesRule(), rules.NewWeakPasswordPolicyRule(), rules.NewDemoUsersActiveRule(), - rules.NewPageEntityReadAccessRule(), // SEC010 - full catalog (widgets, permissions) + rules.NewPageEntityReadAccessRule(), // SEC010 - full catalog (widgets, permissions) rules.NewXPathAssociationAccessRule(), // SEC011 - full catalog (xpath_expressions, permissions) - rules.NewOverlappingActivitiesRule(), // MPR008 - requires BSON inspection - rules.NewLoopChildContainmentRule(), // MPR011 - requires BSON inspection - rules.NewNoCommitInLoopRule(), // CONV011-CONV014 - require BSON inspection + rules.NewOverlappingActivitiesRule(), // MPR008 - requires BSON inspection + rules.NewLoopChildContainmentRule(), // MPR011 - requires BSON inspection + rules.NewNoCommitInLoopRule(), // CONV011-CONV014 - require BSON inspection rules.NewExclusiveSplitCaptionRule(), rules.NewErrorHandlingOnCallsRule(), rules.NewNoContinueErrorHandlingRule(), From a2b8331269aa4fcc0f2dd60f60effc32d52f8442 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 20:01:08 +0000 Subject: [PATCH 57/61] test(git): git-merge fixtures commit with a fixed identity gitInE and the real-mx integration fixture ran git without the identity gitIn sets. git merge needs one even when it stops on a conflict, so on a host whose name gives git nothing to guess from (a bare "vm", no domain) every git-merge fixture failed before the command under test ran. The integration fixture drops the global config, so the command's own merge and commit there need the identity in the process environment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...-01-git-merge-fixtures-depend-on-the-hosts-identity.json | 1 + cmd/mxcli/gitdoctor_test.go | 6 ++++++ cmd/mxcli/gitmerge_integration_test.go | 6 ++++++ 3 files changed, 13 insertions(+) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-fixtures-depend-on-the-hosts-identity.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-fixtures-depend-on-the-hosts-identity.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-fixtures-depend-on-the-hosts-identity.json new file mode 100644 index 000000000..e6870bf17 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-fixtures-depend-on-the-hosts-identity.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-01", "symptom": "Every `TestGitMerge_*` fixture and both real-mx `TestGitMergeIntegration_*` tests fail on a host named `vm` with `Committer identity unknown … unable to auto-detect email address (got 'root@vm.(none)')`, before the command under test runs; they pass on CI runners and developer machines", "cause": "The fixtures drop the global git config (GIT_CONFIG_GLOBAL=/dev/null) for hermeticity, but `gitInE` and the command's own in-process `git merge`/`git commit` in the integration fixture ran without the fixed identity `gitIn` sets, so git fell back to guessing one from the hostname — which works wherever the hostname has a domain", "file": "`cmd/mxcli/gitdoctor_test.go` (`gitInE` sets GIT_AUTHOR_*/GIT_COMMITTER_*), `cmd/mxcli/gitmerge_integration_test.go` (`mergeIntegrationRepo` sets them in the process environment)", "insight": "**Removing the global config is half of hermetic**: git still guesses an identity from the hostname, so a fixture that never sets one passes exactly where the hostname has a domain — every runner image and laptop — and fails in a bare container. `git merge` needs the identity even when it stops on a conflict. Pattern: docs-wiki/bug-patterns/the-platform-ci-does-not-run.md", "refs": ["mendixlabs/mxcli#1674"]} diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index bfc2eef32..43ca6a663 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -605,6 +605,12 @@ func TestGitCommit_NothingToCommit(t *testing.T) { func gitInE(t *testing.T, dir string, args ...string) (string, error) { t.Helper() cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + // The same fixed identity gitIn commits with: `git merge` needs one even + // when it stops on a conflict, and a host whose name gives git nothing to + // guess from (a bare "vm", no domain) refused the fixture's merge. + cmd.Env = append(os.Environ(), + "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.com", + "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.com") b, err := cmd.CombinedOutput() return string(b), err } diff --git a/cmd/mxcli/gitmerge_integration_test.go b/cmd/mxcli/gitmerge_integration_test.go index 82e5244c9..f268b42dd 100644 --- a/cmd/mxcli/gitmerge_integration_test.go +++ b/cmd/mxcli/gitmerge_integration_test.go @@ -112,6 +112,12 @@ func mergeIntegrationRepo(t *testing.T) (string, string, string) { t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) t.Setenv("GIT_CONFIG_NOSYSTEM", "1") t.Setenv("GIT_CEILING_DIRECTORIES", filepath.Dir(dir)) + // The command's own git merge and commit run in this process: with the + // global config gone, they need an identity a hostname cannot supply. + t.Setenv("GIT_AUTHOR_NAME", "t") + t.Setenv("GIT_AUTHOR_EMAIL", "t@example.com") + t.Setenv("GIT_COMMITTER_NAME", "t") + t.Setenv("GIT_COMMITTER_EMAIL", "t@example.com") bin := os.Getenv("MXCLI_BIN") if bin == "" { From c900834e111acfb9e211a6b9400ead1e75f76199 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 20:05:24 +0000 Subject: [PATCH 58/61] fix(git-merge): a merge git refuses to start says why When git merge stopped before any conflict existed, the error was built from its stdout, which holds at most an "Updating a..b" line; git's reason is on stderr. A dirty working tree came out as "git merge side: Updating 5d6b73c..2c3fd47" and a missing identity as "git merge side: " with nothing after it. Both the merge and the --plan path now report git's own message. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...merge-refusal-reason-read-from-stdout.json | 1 + cmd/mxcli/cmd_git_merge.go | 11 ++++--- cmd/mxcli/gitmerge_test.go | 31 +++++++++++++++++++ 3 files changed, 39 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-refusal-reason-read-from-stdout.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-refusal-reason-read-from-stdout.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-refusal-reason-read-from-stdout.json new file mode 100644 index 000000000..e8af79698 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-refusal-reason-read-from-stdout.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-01", "symptom": "`mxcli git merge side -p App.mpr` on a working tree with an uncommitted change the merge would overwrite fails with `git merge side: Updating 5d6b73c..2c3fd47`, and without a git identity with `git merge side: ` and nothing after it; the same with `--plan`", "cause": "When git merge stopped before any conflict existed (no MERGE_HEAD), the error was built from `firstLineOf(mergeOut)` — git's stdout, which holds at most the fast-forward 'Updating a..b' line — while git's reason is on stderr, which gitOutIn already returns as the error", "file": "`cmd/mxcli/cmd_git_merge.go` (`mergeRun` and `applyPlan` report `mergeErr`); test `TestGitMerge_RefusedMergeSaysWhy` in `cmd/mxcli/gitmerge_test.go`", "insight": "**A wrapper that splits stdout from stderr has to pick the stream per outcome**: git writes progress to stdout and refusals to stderr, so the line that is right on success is the wrong one on failure. Found only because a fixture ran on a host where git could not guess an identity — the empty message was the first clue. Pattern: docs-wiki/bug-patterns/misleading-diagnostics.md", "refs": ["mendixlabs/mxcli#1674"]} diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index b0fe8c84f..1afca2ff2 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -208,9 +208,12 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) return finishMerge(out, repo, head, branch, push) } - // A ref that does not resolve fails before any conflict exists. + // A ref that does not resolve fails before any conflict exists, and + // so does a merge git refuses to start (local changes it would + // overwrite, no identity): git's reason is on stderr, which is the + // error — stdout holds at most an "Updating a..b" line. if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + return fmt.Errorf("git merge %s: %v", ref, mergeErr) } } @@ -763,12 +766,12 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m // An unfinished merge (text conflicts resolved and staged, say) is // resumed; otherwise the merge runs and must conflict the same way. if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + _, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) if mergeErr == nil { return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") } if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + return fmt.Errorf("git merge %s: %v", ref, mergeErr) } } baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index e67ba1a13..8a1a306c4 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -740,6 +740,37 @@ func TestGitMerge_ResumesAMergeAlreadyInProgress(t *testing.T) { } } +// A git merge that refuses to start (here: local changes it would overwrite) +// writes its reason to stderr, not stdout; the error must carry it, or the +// user is left with "git merge side: " and nothing after it. +func TestGitMerge_RefusedMergeSaysWhy(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + gitIn(t, dir, "switch", "-q", "-c", "side") + if err := os.WriteFile(filepath.Join(dir, "notes.txt"), []byte("side\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "notes.txt") + gitIn(t, dir, "commit", "-q", "-m", "side notes") + gitIn(t, dir, "switch", "-q", "feature") + if err := os.WriteFile(filepath.Join(dir, "notes.txt"), []byte("uncommitted\n"), 0o644); err != nil { + t.Fatal(err) + } + + for _, plan := range []string{"", filepath.Join(t.TempDir(), "plan.json")} { + if plan != "" { + if err := os.WriteFile(plan, []byte(`{"units":[]}`), 0o644); err != nil { + t.Fatal(err) + } + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", plan, false, mxRunners{}, &out) + if err == nil || !strings.Contains(err.Error(), "would be overwritten") { + t.Errorf("plan=%q: want git's reason in the error, got %v", plan, err) + } + } +} + // The plan template is working material: `mxcli git commit` never stages it. func TestJunkPath_CoversMergePlanTemplate(t *testing.T) { if !isJunkPath("App.mpr.merge-plan.json", ".") { From 6cc4ef43556969798b63ee6594017817a4468ce1 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 20:24:36 +0000 Subject: [PATCH 59/61] feat(mpr): read and clear merge conflict markers in the unit index When Mendix's merge engine stops on a decision it still writes a result and marks each undecided unit in its Unit row: ContentsConflicts holds a JSON list instead of '' (every other row holds '', never NULL; measured on mx 11.13), TreeConflict is non-zero for a conflict about whether the document exists. mx check reports CE0137 while a marker is present. Reader.UnitConflicts lists the marked units. Writer.ClearContentsConflicts records contents conflicts as decided, leaving the row as the engine writes every unit it merged, under the Studio Pro guard and with _Transaction bumped like every other write; tree conflicts are left alone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- modelsdk/mpr/merge_conflicts.go | 89 ++++++++++++++++++++++ modelsdk/mpr/merge_conflicts_test.go | 110 +++++++++++++++++++++++++++ 2 files changed, 199 insertions(+) create mode 100644 modelsdk/mpr/merge_conflicts.go create mode 100644 modelsdk/mpr/merge_conflicts_test.go diff --git a/modelsdk/mpr/merge_conflicts.go b/modelsdk/mpr/merge_conflicts.go new file mode 100644 index 000000000..afc16112b --- /dev/null +++ b/modelsdk/mpr/merge_conflicts.go @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "fmt" + "strings" +) + +// Merge conflict markers in the unit index. +// +// When Mendix's merge engine (`mx merge`, Studio Pro's Merge Changes Here) +// meets a change it cannot decide, it still writes a result, and marks the +// undecided units in their Unit row: ContentsConflicts holds a JSON list of +// the conflicting changes instead of '' (measured on mx 11.13: every other row +// holds '', never NULL), and TreeConflict is non-zero when the conflict is +// about whether the document exists at all. `mx check` reports CE0137 ("There +// are merge conflicts in this project") for as long as a marker is present. +// The index's other tables carry no conflict state. + +// UnitConflict is a unit whose index row carries a merge conflict marker. +type UnitConflict struct { + UnitID string // Mendix UUID form + Contents string // ContentsConflicts as stored ("" when only Tree is set) + Tree int64 // TreeConflict as stored (0 when only Contents is set) +} + +// UnitConflicts lists the units whose index row carries a merge conflict +// marker. Read-only. +func (r *Reader) UnitConflicts() ([]UnitConflict, error) { + rows, err := r.db.Query(`SELECT UnitID, COALESCE(ContentsConflicts, ''), COALESCE(TreeConflict, 0) FROM Unit + WHERE COALESCE(ContentsConflicts, '') != '' OR COALESCE(TreeConflict, 0) != 0`) + if err != nil { + return nil, fmt.Errorf("query merge conflict markers: %w", err) + } + defer rows.Close() + var out []UnitConflict + for rows.Next() { + var blob []byte + var c UnitConflict + if err := rows.Scan(&blob, &c.Contents, &c.Tree); err != nil { + return nil, fmt.Errorf("scan merge conflict marker: %w", err) + } + c.UnitID = blobToUUID(blob) + out = append(out, c) + } + return out, rows.Err() +} + +// ClearContentsConflicts empties the ContentsConflicts marker of the given +// units — the step that records a contents conflict as decided, once the +// unit's file holds the chosen content — leaving the row as the engine writes +// every unit it merged without a decision. A TreeConflict is left alone: which +// documents exist is not a contents decision. One SQLite transaction, with +// _Transaction.LastTransactionID bumped as every other write does, under the +// writer's Studio Pro open-project guard. Returns the number of rows cleared. +func (w *Writer) ClearContentsConflicts(unitIDs []string) (int, error) { + if len(unitIDs) == 0 { + return 0, nil + } + if err := w.guardWrite(); err != nil { + return 0, err + } + tx, err := w.reader.db.Begin() + if err != nil { + return 0, err + } + n := 0 + for _, id := range unitIDs { + res, err := tx.Exec(`UPDATE Unit SET ContentsConflicts = '' WHERE UnitID = ? AND COALESCE(ContentsConflicts, '') != ''`, uuidToBlob(id)) + if err != nil { + _ = tx.Rollback() + return 0, fmt.Errorf("clear the merge conflict marker of %s: %w", id, err) + } + if c, _ := res.RowsAffected(); c > 0 { + n += int(c) + } + } + if _, err := tx.Exec(`UPDATE _Transaction SET LastTransactionID = ?`, generateUUID()); err != nil && + !strings.Contains(err.Error(), "no such table") { + _ = tx.Rollback() + return 0, fmt.Errorf("update _Transaction: %w", err) + } + if err := tx.Commit(); err != nil { + return 0, err + } + w.reader.InvalidateCache() + return n, nil +} diff --git a/modelsdk/mpr/merge_conflicts_test.go b/modelsdk/mpr/merge_conflicts_test.go new file mode 100644 index 000000000..8b95a55d2 --- /dev/null +++ b/modelsdk/mpr/merge_conflicts_test.go @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "errors" + "os" + "testing" +) + +// A merge engine that stops on a decision (`mx merge` exit 2) leaves its +// marker in the unit's index row: ContentsConflicts holds a JSON list instead +// of an empty string, and `mx check` reports CE0137 until it is gone. +const engineMarker = `{"Conflicts":[{"ID":"00000000-0000-0000-0000-000000000000","HeadModification":"Change","MineModification":"Change"}]}` + +func setMarker(t *testing.T, w *Writer, unitID, contents string, tree int) { + t.Helper() + if _, err := w.reader.db.Exec(`UPDATE Unit SET ContentsConflicts = ?, TreeConflict = ? WHERE UnitID = ?`, + contents, tree, uuidToBlob(unitID)); err != nil { + t.Fatal(err) + } +} + +func TestUnitConflictsCleanControl(t *testing.T) { + w, _ := newV2WriterForCommitTest(t, hashTestUnit, unitDoc(t, "Stored")) + setMarker(t, w, hashTestUnit, "", 0) // the value every unmarked row holds + got, err := w.ConcreteReader().UnitConflicts() + if err != nil { + t.Fatal(err) + } + if len(got) != 0 { + t.Fatalf("an unmarked project lists conflicts: %+v", got) + } +} + +func TestUnitConflictsListsContentsAndTreeMarkers(t *testing.T) { + w, _ := newV2WriterForCommitTest(t, hashTestUnit, unitDoc(t, "Stored")) + setMarker(t, w, hashTestUnit, engineMarker, 0) + got, err := w.ConcreteReader().UnitConflicts() + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].UnitID != hashTestUnit || got[0].Contents != engineMarker { + t.Fatalf("contents marker not listed: %+v", got) + } + setMarker(t, w, hashTestUnit, "", 1) + got, err = w.ConcreteReader().UnitConflicts() + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Tree != 1 { + t.Fatalf("tree marker not listed: %+v", got) + } +} + +func TestClearContentsConflictsLeavesTheRowLikeAnUnmarkedOne(t *testing.T) { + w, _ := newV2WriterForCommitTest(t, hashTestUnit, unitDoc(t, "Stored")) + hash := storedHash(t, w, hashTestUnit) + setMarker(t, w, hashTestUnit, engineMarker, 0) + n, err := w.ClearContentsConflicts([]string{hashTestUnit}) + if err != nil { + t.Fatal(err) + } + if n != 1 { + t.Fatalf("cleared %d row(s), want 1", n) + } + var contents string + if err := w.reader.db.QueryRow(`SELECT ContentsConflicts FROM Unit WHERE UnitID = ?`, uuidToBlob(hashTestUnit)).Scan(&contents); err != nil { + t.Fatal(err) + } + if contents != "" { + t.Fatalf("ContentsConflicts = %q, want '' (what the engine writes on every unmarked row)", contents) + } + if got := storedHash(t, w, hashTestUnit); got != hash { + t.Fatal("clearing the marker touched ContentsHash") + } + if got, _ := w.ConcreteReader().UnitConflicts(); len(got) != 0 { + t.Fatalf("still listed after clearing: %+v", got) + } +} + +// A tree conflict (deleted on one side, changed on the other) is a different +// decision — which documents exist — and is never cleared by this. +func TestClearContentsConflictsLeavesTreeConflicts(t *testing.T) { + w, _ := newV2WriterForCommitTest(t, hashTestUnit, unitDoc(t, "Stored")) + setMarker(t, w, hashTestUnit, engineMarker, 1) + if _, err := w.ClearContentsConflicts([]string{hashTestUnit}); err != nil { + t.Fatal(err) + } + got, _ := w.ConcreteReader().UnitConflicts() + if len(got) != 1 || got[0].Tree != 1 { + t.Fatalf("tree conflict cleared or lost: %+v", got) + } +} + +func TestClearContentsConflictsRefusedWhileStudioProOpen(t *testing.T) { + w, _ := newV2WriterForCommitTest(t, hashTestUnit, unitDoc(t, "Stored")) + setMarker(t, w, hashTestUnit, engineMarker, 0) + if err := os.WriteFile(w.ConcreteReader().Path()+".lock", nil, 0644); err != nil { + t.Fatal(err) + } + _, err := w.ClearContentsConflicts([]string{hashTestUnit}) + var open *StudioProOpenError + if !errors.As(err, &open) { + t.Fatalf("want StudioProOpenError, got %v", err) + } + if got, _ := w.ConcreteReader().UnitConflicts(); len(got) != 1 { + t.Fatal("marker cleared despite the guard") + } +} From dd08c8dbd30fcb035e6682f1543c506ddb7a83db Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 20:24:37 +0000 Subject: [PATCH 60/61] fix(git-merge): --plan resolves over the engine's result applyPlan resolved from git's two sides: ours for every model file git conflicted, theirs for the planned units. With the template listing only the engine's ManualResolutionRequired units, a unit the engine merges by itself was taken from ours wholesale. Measured with mx 11.13: each side adds an entity to one domain model (AutoResolved) and both edit one entity's documentation (ManualResolutionRequired); the plan names only the second unit, and its commit lost theirs' entity, hash gate passing. The plan step now runs the engine again over clean worktrees and resolves over its result: everything it merged stays merged, each planned unit is taken whole from its side's commit, and its conflict marker is cleared. The plan must name exactly the units the engine left open (a stale or hand-widened plan is refused), and nothing is committed while the index still marks a unit as conflicted. The template and the check read the same set (engineDecisions), with the result's markers as the fallback when the engine prints no -v list. The real-mx integration test applies a generated plan to that scenario on a scaffolded project and asserts both entities, theirs' page, no marker, and no CE0137 from mx check. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...rge-plan-drops-what-the-engine-merged.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_merge.go | 281 +++++++++++++----- cmd/mxcli/gitmerge_integration_test.go | 101 ++++++- cmd/mxcli/gitmerge_test.go | 207 ++++++++++--- docs-site/src/tools/outside-studio-pro.md | 11 +- 6 files changed, 487 insertions(+), 116 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-plan-drops-what-the-engine-merged.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-plan-drops-what-the-engine-merged.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-plan-drops-what-the-engine-merged.json new file mode 100644 index 000000000..c0199fd2c --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-11-01-git-merge-plan-drops-what-the-engine-merged.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-11-01", "symptom": "`mxcli git merge --plan ` commits a valid two-parent merge that has silently lost one side's change to a unit the plan never names: ours adds `MyFirstModule.OursOnly`, theirs adds `MyFirstModule.TheirsOnly`, both edit `FeedbackModule.Feedback`'s documentation; the engine exits 2, the template lists only FeedbackModule's domain model, and after `--plan` with take theirs `TheirsOnly` is gone — the hash gate passes and nothing in the output mentions the unit", "cause": "applyPlan resolved from git's two sides — ours for every model file git conflicted, theirs for the planned units — and never ran the engine. Once the template listed only the engine's ManualResolutionRequired units, a unit the engine merges by itself (MergeContents/AutoResolved: an entity added on each side of one domain model) was taken from ours wholesale", "file": "`cmd/mxcli/cmd_git_merge.go` (`applyPlan` re-runs the engine over `checkoutRevisions` worktrees, checks the plan against `engineDecisions`, copies the result back with `copyEngineResult`, takes each planned unit with `git checkout -- `, `clearDecidedMarkers`, then refuses anything `conflictMarkers` still lists); `modelsdk/mpr/merge_conflicts.go` (`Reader.UnitConflicts`, `Writer.ClearContentsConflicts`); tests `TestGitMerge_PlanKeepsWhatTheEngineMerged`, `TestGitMerge_PlanMustMatchTheEnginesDecisions` in `cmd/mxcli/gitmerge_test.go`, `TestGitMergeIntegration_PlanKeepsWhatTheEngineMerged` (real mx) in `cmd/mxcli/gitmerge_integration_test.go`", "insight": "**\"AutoResolved\" means merged in the engine's result, not merged in yours**: a step that rebuilds from the sides instead of the result has to re-decide every unit, and listing fewer units to it hides the ones it gets wrong. Change the review list and the resolution source together. **The engine's exit-2 result is usable**: MINE holds every auto-resolved unit merged and marks only the undecided ones, in `Unit.ContentsConflicts` (a JSON list where every other row holds '' — not NULL); `mx check` reports CE0137 while a marker is left, which makes it the headless oracle for 'did the clear read as resolved'. Take a planned unit from its side's commit, not from MINE: MINE's copy of an undecided unit is not byte-identical to ours'. Measured with mx 11.13 on the PedApp fixture and an `mx create-project` scaffold. Pattern: docs-wiki/bug-patterns/rewrite-drops-unauthored-state.md", "refs": ["mendixlabs/mxcli#1674"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index aa8643c74..5a051edae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,7 +42,7 @@ Headline: **Write what the project's own Mendix version stores, and commit it th - **Lint rule CONV021 (CreateChangeThenCommit)** — warns when a microflow's Create or Change object activity without commit is directly followed by a Commit of the same object, and suggests the one activity that does both: Commit set on the create/change (Yes, or Yes without events for a commit without events; Refresh in client kept). It stays quiet when merging could change behaviour: another flow into the commit, an error-handler flow between them, or custom error handling on either. - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx merge BASE MINE THEIRS` over the three revisions' project files, which runs on Linux devcontainers too. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`; a merge stopped on text conflicts resumes once those are resolved and staged), and on a model-file conflict checks the base, ours (a clean checkout of HEAD — never the working tree mid-merge, where the engine chokes) and theirs revisions out into temporary worktrees and hands the three projects' `.mpr` files to the engine, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything the engine auto-resolves is copied back from MINE, hash-checked, staged and committed as the normal two-parent merge commit with the mx_metadata note — and the engine owns the `.mpr`, so a unit one side added while the other changed existing ones is kept, not dropped, and a deletion it resolved removes the unit file with it; the engine's own `-v` conflict list (one JSON array per run, `ManualResolutionRequired` vs `AutoResolved`) decides which units the review material and plan template cover, with git's conflicted-file list as the fallback; a conflict the engine reports as `ManualResolutionRequired` (exit 2) is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit at property level (from `mx diff`), with a plan template (`.merge-plan.json`, never staged by `mxcli git commit`) beside the project — the human re-runs with `--plan ` to resolve **per unit** (each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files; modified-on-both units only, added/deleted/moved goes to Studio Pro), or with `--ours`/`--theirs` to take one side of every conflicted unit — hash-gated: when the sides' document sets differ (one side added or deleted a unit) the commit is refused and the merge aborted, because a whole side would orphan the other side's documents — or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; any other engine exit (its 129, an mx that will not start) is reported as the engine failing and the merge is aborted cleanly, never steered into a decision. Non-model conflicts (text sources) are left in place like plain git: resolve them, `git add` them, re-run the command. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx merge BASE MINE THEIRS` over the three revisions' project files, which runs on Linux devcontainers too. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`; a merge stopped on text conflicts resumes once those are resolved and staged), and on a model-file conflict checks the base, ours (a clean checkout of HEAD — never the working tree mid-merge, where the engine chokes) and theirs revisions out into temporary worktrees and hands the three projects' `.mpr` files to the engine, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything the engine auto-resolves is copied back from MINE, hash-checked, staged and committed as the normal two-parent merge commit with the mx_metadata note — and the engine owns the `.mpr`, so a unit one side added while the other changed existing ones is kept, not dropped, and a deletion it resolved removes the unit file with it; the engine's own `-v` conflict list (one JSON array per run, `ManualResolutionRequired` vs `AutoResolved`) decides which units the review material and plan template cover, with git's conflicted-file list as the fallback; a conflict the engine reports as `ManualResolutionRequired` (exit 2) is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit at property level (from `mx diff`), with a plan template (`.merge-plan.json`, never staged by `mxcli git commit`) beside the project — the human re-runs with `--plan ` to resolve **per unit** over the engine's own result (the engine runs again and the plan decides only the units it left open — everything it merged stays merged; each planned unit takes `ours` or `theirs` whole, its conflict marker is cleared and the `.mpr` index repaired from the files; a plan naming more or fewer units than the engine left open is refused; modified-on-both units only, added/deleted/moved goes to Studio Pro), or with `--ours`/`--theirs` to take one side of every conflicted unit — hash-gated: when the sides' document sets differ (one side added or deleted a unit) the commit is refused and the merge aborted, because a whole side would orphan the other side's documents — or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; any other engine exit (its 129, an mx that will not start) is reported as the engine failing and the merge is aborted cleanly, never steered into a decision. Non-model conflicts (text sources) are left in place like plain git: resolve them, `git add` them, re-run the command. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. - **Workflows say what Studio Pro lets them say** — a user task takes a `task name '…'` clause (what the task inbox shows, until now always the caption), and the workflow's display and description and a task's name and description take template parameters, `display 'Approval for {1}' with ({1} = $WorkflowContext/Title)`, in CREATE and ALTER. A timer boundary event takes `caption '…'`, and a non-interrupting timer can recur: `repeat every 1 day at most 3 times`. `alter workflow … set (Annotation: '…')` sets the workflow's note, and `… on ` an activity's. A `describe` → `create or modify` that would still reset a setting MDL cannot state is refused, naming the setting, instead of reporting "Modified workflow". - **MDL-WF20: a user task whose page cannot complete one of its outcomes** — Mendix adds no buttons to a task page, so a page that only shows the task passes `mx check` and the build, and every instance then waits at the task forever. `check` now warns per task, listing the outcomes no `complete task` button, or button whose microflow reaches `set task outcome`, can complete. diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index 1afca2ff2..671858355 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -61,9 +61,14 @@ the engine chokes on the conflicted state. Its result is copied back: "New type > Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: --plan resolve per unit from the plan template written beside the .mpr (.merge-plan.json): set "take" to - "ours"/"theirs" per unit; the .mpr index is repaired - from the mixed files. Modified-on-both units only — - added/deleted/moved conflicts go to Studio Pro. + "ours"/"theirs" per unit. The engine runs again and the + plan decides only the units it left open — everything + it merged stays merged, so the plan must name exactly + those units (a stale or hand-widened plan is refused). + Each planned unit is taken whole from its side, its + conflict marker cleared, and the .mpr index repaired. + Modified-on-both units only — added/deleted/moved + conflicts go to Studio Pro. --ours / --theirs take one side of every conflicted unit (deterministic; mx is not consulted again). The .mpr is taken from that side too and the result is hash-checked: @@ -159,7 +164,8 @@ type mxRunners struct { // always the best outcome — and is committed with the note. // 3. What the engine reports as ManualResolutionRequired (exit 2) is never // merged silently: with no strategy the merge is aborted and both sides' -// changes are printed for review; `--plan` then resolves per unit, and +// changes are printed for review; `--plan` then resolves the units the +// engine left open, over the engine's own result, and // `--ours`/`--theirs` take one side of every conflicted unit, hash-gated // so a side that would drop or orphan documents is refused. Any other // engine exit is a failure, not a decision: reported, aborted cleanly. @@ -191,7 +197,7 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m } if plan != "" { - return applyPlan(root, repo, abs, mprPath, ref, plan, message, branch, push, out) + return applyPlan(root, repo, abs, mprPath, ref, plan, message, branch, push, runners, out) } // A merge already in progress is resumed, not re-run: a previous run may @@ -265,49 +271,17 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m // never the working tree mid-merge — the engine chokes on the conflicted // state (measured: it exits 129 when theirs deleted a document). Its // result lands in MINE and is copied back. - tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") + revs, err := checkoutRevisions(root, ref, baseSha, oursSha, theirsSha) if err != nil { return err } - baseDir := filepath.Join(tmp, "base") - oursDir := filepath.Join(tmp, "ours") - theirsDir := filepath.Join(tmp, "theirs") - if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", baseDir, baseSha); err != nil { - os.RemoveAll(tmp) - return fmt.Errorf("check out the merge base: %w", err) - } - if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", oursDir, oursSha); err != nil { - _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) - os.RemoveAll(tmp) - return fmt.Errorf("check out HEAD: %w", err) - } - if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", theirsDir, theirsSha); err != nil { - _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) - _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", oursDir) - os.RemoveAll(tmp) - return fmt.Errorf("check out %s: %w", ref, err) - } - cleanWorktrees := func() { - _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) - _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", oursDir) - _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", theirsDir) - _ = os.RemoveAll(tmp) - } + tmp, baseDir, oursDir, theirsDir := revs.tmp, revs.baseDir, revs.oursDir, revs.theirsDir + cleanWorktrees := revs.cleanup version := projectMendixVersion(mprPath) fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", strings.Join(shortNames(model), ", "), version) - // File form: the engine takes the three revisions' project files; MINE is - // the clean worktree at HEAD. - code, mxOut, runErr := runners.merge( - filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), - filepath.Join(oursDir, filepath.FromSlash(repo.MPRPath)), - filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), - [3]string{baseSha, oursSha, theirsSha}, - ) - if s := strings.TrimSpace(mxOut); s != "" { - fmt.Fprintln(out, s) - } + code, mxOut, runErr := revs.runEngine(runners, repo, out) if runErr != nil { cleanWorktrees() return abortMerge(root, fmt.Errorf("mx merge failed to run: %w", runErr)) @@ -325,13 +299,15 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m // the human decides. The unit list follows the engine's own conflict // list — git's can differ (a stale one-side index makes the engine // content-merge a unit git sees as changed). Without a parsable - // list, fall back to git's conflicted units: err toward showing work - // rather than hiding it. - manual := engineManualIDs(parseEngineConflicts(mxOut)) + // list, the units the engine's result marks; without those, git's + // conflicted units: err toward showing work rather than hiding it. + // --plan reads the same set back, so the template it is given + // matches what it checks the plan against. + oursMpr := filepath.Join(oursDir, filepath.FromSlash(repo.MPRPath)) + manual := engineDecisions(mxOut, oursMpr) if len(manual) == 0 { manual = conflictedUnitIDs(model) } - oursMpr := filepath.Join(oursDir, filepath.FromSlash(repo.MPRPath)) if runners.diff != nil { oursJSON := filepath.Join(tmp, "ours-diff.json") theirsJSON := filepath.Join(tmp, "theirs-diff.json") @@ -383,6 +359,95 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return nil } +// mergeRevisions are clean worktrees of a merge's three revisions — the +// engine's BASE, MINE (ours, at HEAD) and THEIRS — under one temp folder. +type mergeRevisions struct { + root, tmp string + baseDir, oursDir, theirsDir string + shas [3]string +} + +// checkoutRevisions checks the merge's three revisions out as detached +// worktrees; cleanup removes them again. +func checkoutRevisions(root, ref, baseSha, oursSha, theirsSha string) (*mergeRevisions, error) { + tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") + if err != nil { + return nil, err + } + m := &mergeRevisions{ + root: root, tmp: tmp, + baseDir: filepath.Join(tmp, "base"), + oursDir: filepath.Join(tmp, "ours"), + theirsDir: filepath.Join(tmp, "theirs"), + shas: [3]string{baseSha, oursSha, theirsSha}, + } + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", m.baseDir, baseSha); err != nil { + os.RemoveAll(tmp) + return nil, fmt.Errorf("check out the merge base: %w", err) + } + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", m.oursDir, oursSha); err != nil { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", m.baseDir) + os.RemoveAll(tmp) + return nil, fmt.Errorf("check out HEAD: %w", err) + } + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", m.theirsDir, theirsSha); err != nil { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", m.baseDir) + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", m.oursDir) + os.RemoveAll(tmp) + return nil, fmt.Errorf("check out %s: %w", ref, err) + } + return m, nil +} + +func (m *mergeRevisions) cleanup() { + _, _ = gitOutIn(m.root, nil, "worktree", "remove", "--force", m.baseDir) + _, _ = gitOutIn(m.root, nil, "worktree", "remove", "--force", m.oursDir) + _, _ = gitOutIn(m.root, nil, "worktree", "remove", "--force", m.theirsDir) + _ = os.RemoveAll(m.tmp) +} + +// runEngine runs the engine in file form — the three revisions' project +// files, MINE the clean worktree at HEAD — and echoes what it printed. Its +// result lands in MINE. +func (m *mergeRevisions) runEngine(runners mxRunners, repo *gitNoteRepo, out io.Writer) (int, string, error) { + mprIn := func(dir string) string { return filepath.Join(dir, filepath.FromSlash(repo.MPRPath)) } + code, mxOut, err := runners.merge(mprIn(m.baseDir), mprIn(m.oursDir), mprIn(m.theirsDir), m.shas) + if s := strings.TrimSpace(mxOut); s != "" { + fmt.Fprintln(out, s) + } + return code, mxOut, err +} + +// engineDecisions is the set of units the engine left for a human after an +// exit 2: the ManualResolutionRequired entries of its -v list or, when it +// printed no list, the units its result (mineMpr) marks as conflicted. +func engineDecisions(mxOut, mineMpr string) []string { + if ids := engineManualIDs(parseEngineConflicts(mxOut)); len(ids) > 0 { + return ids + } + marks, err := conflictMarkers(mineMpr) + if err != nil { + return nil + } + var ids []string + for _, c := range marks { + ids = append(ids, c.UnitID) + } + return ids +} + +// conflictMarkers lists the units a project's index marks as conflicted — +// what the engine writes for a decision it did not make, and what Studio Pro +// reports as CE0137 until it is gone. +func conflictMarkers(mprPath string) ([]mpr.UnitConflict, error) { + r, err := mpr.Open(mprPath) + if err != nil { + return nil, err + } + defer r.Close() + return r.UnitConflicts() +} + // copyEngineResult copies the engine's merged model files — the .mpr and // every .mxunit under mprcontents — from the MINE worktree into the working // tree, removing working-tree units the result no longer has (a deletion the @@ -744,11 +809,20 @@ func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, unitIDs []st } // applyPlan re-runs the merge unless one is already in progress (it must -// conflict the same way), checks the plan against those revisions, resolves -// per unit, repairs the .mpr index from the mixed files and commits — with -// the note. Every refusal after the merge started aborts it: a half-done -// merge makes Studio Pro re-run it (and choke) on every open. -func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, message, branch string, push bool, out io.Writer) error { +// conflict the same way), checks the plan against those revisions, runs the +// engine again and resolves over its result: everything the engine merged +// stays merged, and each unit it left open takes the plan's side — whole, +// from that side's commit — with its conflict marker cleared. The .mpr index +// is repaired from the files and the merge committed with the note. Every +// refusal after the merge started aborts it: a half-done merge makes Studio +// Pro re-run it (and choke) on every open. +// +// The plan must name exactly the units the engine left open. Resolving from +// git's two sides instead — ours for every conflicted file, theirs for the +// planned units — took a unit the engine had merged (an entity each side +// added to one domain model) from ours, dropping theirs' change to it +// without the plan ever naming it. +func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, message, branch string, push bool, runners mxRunners, out io.Writer) error { raw, err := os.ReadFile(planPath) if err != nil { return fmt.Errorf("read plan: %w", err) @@ -781,14 +855,6 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m return abortMerge(root, errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate")) } - conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) - var model []string - for _, p := range conflicted { - if isModelPath(repo, p) { - model = append(model, p) - } - } - // Every planned unit must be modified on both sides; anything else // (added on one side, deleted, moved) only Studio Pro can decide. baseSnap, err := repo.snapshot(baseSha) @@ -816,38 +882,101 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m take[u.ID] = u.Take } - // Ours everywhere, then the planned units from theirs. - for _, p := range model { - if _, err := gitOutIn(root, nil, "checkout", "--ours", "--", p); err != nil { - return abortMerge(root, fmt.Errorf("take the ours version of %s: %w", p, err)) + // The engine again, over clean worktrees as in the merge itself: the plan + // decides only what it leaves open. + revs, err := checkoutRevisions(root, ref, baseSha, oursSha, theirsSha) + if err != nil { + return abortMerge(root, err) + } + defer revs.cleanup() + fmt.Fprintf(out, "Merging the model with mx %s; the plan decides what the engine leaves open…\n", projectMendixVersion(mprPath)) + code, mxOut, runErr := revs.runEngine(runners, repo, out) + switch { + case runErr != nil: + return abortMerge(root, fmt.Errorf("mx merge failed to run: %w", runErr)) + case code == 0: + return abortMerge(root, errors.New("the engine now merges the model without a decision — the plan is unnecessary; re-run without --plan")) + case code != 2: + return abortMerge(root, fmt.Errorf("mx merge failed (exit %d); the merge is aborted — if it persists, resolve in Studio Pro (Version Control › Merge Changes Here)", code)) + } + open := engineDecisions(mxOut, filepath.Join(revs.oursDir, filepath.FromSlash(repo.MPRPath))) + if len(open) == 0 { + return abortMerge(root, errors.New("the engine stopped on a decision but named no unit; resolve in Studio Pro (Version Control › Merge Changes Here)")) + } + isOpen := map[string]bool{} + for _, id := range open { + isOpen[id] = true + if _, planned := take[id]; !planned { + return abortMerge(root, fmt.Errorf("plan: unit %s is left for a decision by the engine and the plan does not decide it — delete the plan and re-run without --plan to regenerate it", shortCommit(id))) } } - for _, id := range conflictedUnitIDs(model) { - if take[id] != "theirs" { - continue + for _, u := range plan.Units { + if !isOpen[u.ID] { + return abortMerge(root, fmt.Errorf("plan: unit %s is merged without a decision by the engine — remove it from the plan: taking a side there drops the other side's change to it", shortCommit(u.ID))) } - p := path.Join(repo.Contents, id[:2], id[2:4], id+".mxunit") - if _, err := gitOutIn(root, nil, "checkout", "--theirs", "--", p); err != nil { - return abortMerge(root, fmt.Errorf("take the theirs version of %s: %w", p, err)) + } + + // The engine's result first, then each planned unit whole from its + // side's commit, its marker cleared. The hashes are rewritten from the + // files — files are the truth — and the gate refuses (and aborts) a + // result whose index and files disagree on which units exist. + if err := copyEngineResult(root, repo, revs.oursDir); err != nil { + return abortMerge(root, fmt.Errorf("copy the engine's merge result: %w", err)) + } + var decided []string + for _, u := range plan.Units { + sha := oursSha + if u.Take == "theirs" { + sha = theirsSha + } + p := path.Join(repo.Contents, u.ID[:2], u.ID[2:4], u.ID+".mxunit") + if _, err := gitOutIn(root, nil, "checkout", sha, "--", p); err != nil { + return abortMerge(root, fmt.Errorf("take the %s version of %s: %w", u.Take, p, err)) } - fmt.Fprintf(out, " %s: theirs\n", shortCommit(id)) + decided = append(decided, u.ID) + fmt.Fprintf(out, " %s: %s\n", shortCommit(u.ID), u.Take) + } + if err := clearDecidedMarkers(mprAbs, decided); err != nil { + return abortMerge(root, err) } - // The .mpr index is ours; the mixed unit files have different bytes, so - // the hashes are rewritten from the files — files are the truth. The - // gate refuses (and aborts) when the sides' document sets differ: a unit - // theirs added is not in ours' .mpr, and committing would orphan it. if err := repairOrRefuse(root, mprPath, out); err != nil { return err } + // Nothing may be committed still marked: Studio Pro would report the + // project as mid-conflict (CE0137) on every open. + marks, err := conflictMarkers(mprAbs) + if err != nil { + return abortMerge(root, fmt.Errorf("read the merge conflict markers: %w", err)) + } + if len(marks) > 0 { + var ids []string + for _, c := range marks { + ids = append(ids, shortCommit(c.UnitID)) + } + return abortMerge(root, fmt.Errorf("refusing to commit: the index still marks unit(s) %s as conflicted; resolve in Studio Pro (Version Control › Merge Changes Here)", strings.Join(ids, ", "))) + } - args := []string{"add", "--"} - args = append(args, model...) - if _, err := gitOutIn(root, nil, args...); err != nil { + if _, err := gitOutIn(root, nil, "add", "-A", "--", + filepath.FromSlash(repo.MPRPath), filepath.FromSlash(repo.Contents)); err != nil { return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) } return commitTheMerge(root, repo, nil, message, branch, push, out) } +// clearDecidedMarkers records the planned units as decided in the index: their +// files now hold the chosen side, so the engine's marker on them goes. +func clearDecidedMarkers(mprPath string, ids []string) error { + w, err := mpr.NewWriter(mprPath) + if err != nil { + return fmt.Errorf("open project: %w", err) + } + defer w.Close() + if _, err := w.ClearContentsConflicts(ids); err != nil { + return fmt.Errorf("clear the decided units' conflict markers: %w", err) + } + return nil +} + func changeStatuses(changes []mxModelChange) map[string]string { out := map[string]string{} for _, c := range changes { diff --git a/cmd/mxcli/gitmerge_integration_test.go b/cmd/mxcli/gitmerge_integration_test.go index f268b42dd..fb63e840c 100644 --- a/cmd/mxcli/gitmerge_integration_test.go +++ b/cmd/mxcli/gitmerge_integration_test.go @@ -6,6 +6,7 @@ package main import ( "bytes" + "encoding/json" "fmt" "io/fs" "os" @@ -97,6 +98,96 @@ func TestGitMergeIntegration_PropertyConflictNeedsDecisionAborts(t *testing.T) { } } +// --plan over the engine's own result: each side adds an entity to +// MyFirstModule's domain model — a unit the engine merges by itself — and +// both retitle the same page, which it leaves for a decision. The plan the +// first run writes names only the page; applying it must keep both entities +// and commit an index with no conflict marker left (`mx check` reports +// CE0137 while one is). Resolving from git's two sides instead took the +// domain model from ours and lost theirs' entity, with the plan never +// naming that unit. +func TestGitMergeIntegration_PlanKeepsWhatTheEngineMerged(t *testing.T) { + dir, mpr, bin := mergeIntegrationRepo(t) + + applyPage(t, bin, dir, mpr, "Shared", "Base title") + commitAll(t, dir, "base adds Shared") + gitIn(t, dir, "switch", "-q", "-c", "side") + applyMDL(t, bin, dir, mpr, "TheirsOnly", "mdl 1;\ncreate non-persistent entity MyFirstModule.TheirsOnly (Code: String(50));\n") + applyPage(t, bin, dir, mpr, "Shared", "Theirs retitle") + commitAll(t, dir, "side adds TheirsOnly, retitles Shared") + gitIn(t, dir, "switch", "-q", "main") + applyMDL(t, bin, dir, mpr, "OursOnly", "mdl 1;\ncreate non-persistent entity MyFirstModule.OursOnly (Name: String(50));\n") + applyPage(t, bin, dir, mpr, "Shared", "Ours retitle") + commitAll(t, dir, "main adds OursOnly, retitles Shared") + // Control: the title is distinctive — only the page carries it (the + // scaffold's own units say "Main title", for one). + if n := unitFilesContaining(t, dir, "Ours retitle"); n != 1 { + t.Fatalf("%d unit files carry ours' title before the merge, want 1 (the page)", n) + } + + runners := mxRunners{merge: realMxMerge, diff: realMxDiff} + var first bytes.Buffer + if err := mergeRun(mpr, "side", "", "", "", false, runners, &first); err == nil { + t.Fatalf("the page conflict must stop for a decision\n%s", first.String()) + } + raw, err := os.ReadFile(mpr + ".merge-plan.json") + if err != nil { + t.Fatalf("no plan template: %v\n%s", err, first.String()) + } + _ = os.Remove(mpr + ".merge-plan.json") // working material, never committed + var plan mergePlanTemplate + if err := json.Unmarshal(raw, &plan); err != nil { + t.Fatal(err) + } + if len(plan.Units) != 1 { + t.Fatalf("the template must name only the unit the engine left open (the page), got %+v\n%s", plan.Units, first.String()) + } + plan.Units[0].Take = "theirs" + edited, _ := json.Marshal(plan) + planPath := filepath.Join(t.TempDir(), "plan.json") + if err := os.WriteFile(planPath, edited, 0o644); err != nil { + t.Fatal(err) + } + + var out bytes.Buffer + if err := mergeRun(mpr, "side", "", "", planPath, false, runners, &out); err != nil { + t.Fatalf("mergeRun --plan: %v\n%s", err, out.String()) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + if st := gitOutT(t, dir, "status", "--porcelain"); strings.TrimSpace(st) != "" { + t.Errorf("working tree not clean after the merge:\n%s", st) + } + for _, name := range []string{"OursOnly", "TheirsOnly", "Theirs retitle"} { + if unitFilesContaining(t, dir, name) == 0 { + t.Errorf("no unit file carries %q: the plan's merge lost it", name) + } + } + if unitFilesContaining(t, dir, "Ours retitle") != 0 { + t.Error("the page still carries ours' title: the plan said theirs") + } + if marks, err := conflictMarkers(mpr); err != nil || len(marks) != 0 { + t.Errorf("merge conflict markers committed: %+v (%v)", marks, err) + } + mxPath, err := docker.ResolveMxForVersion("", projectMendixVersion(mpr)) + if err != nil { + t.Fatal(err) + } + check := exec.Command(mxPath, "check", mpr) + docker.PrepareMxCommand(check) + // The scaffold's own errors make the exit code meaningless here; what + // matters is that the check loaded the project and found no conflict. + checkOut, _ := check.CombinedOutput() + if !bytes.Contains(checkOut, []byte("Loading the mpr file")) { + t.Fatalf("mx check never loaded the project, so its silence on CE0137 proves nothing:\n%s", checkOut) + } + if bytes.Contains(checkOut, []byte("CE0137")) { + t.Errorf("mx check still reports the merge conflict (CE0137):\n%s", checkOut) + } +} + // mergeIntegrationRepo scaffolds a real Mendix project (`mx create-project`) // in a hermetic git repository and returns its dir, .mpr path and a built // mxcli binary for applying MDL. The project and the binary live in SHORT @@ -176,6 +267,13 @@ func scaffoldProject(t *testing.T, dir string) { // The script lives outside the repository: it is test scaffolding, not // project content, and committing it would conflict on both sides. func applyPage(t *testing.T, bin, dir, mpr, name, title string) { + t.Helper() + applyMDL(t, bin, dir, mpr, name, fmt.Sprintf("mdl 1;\ncreate or modify page MyFirstModule.%s (title: '%s', layout: Atlas_Core.Atlas_Default) {\n dynamictext t1 (content: '%s')\n};\n", name, title, title)) +} + +// applyMDL applies one MDL script the same way; what names the change in a +// failure message. +func applyMDL(t *testing.T, bin, dir, mpr, what, mdl string) { t.Helper() f, err := os.CreateTemp("", "mxpage") if err != nil { @@ -183,7 +281,6 @@ func applyPage(t *testing.T, bin, dir, mpr, name, title string) { } script := f.Name() t.Cleanup(func() { os.Remove(script) }) - mdl := fmt.Sprintf("mdl 1;\ncreate or modify page MyFirstModule.%s (title: '%s', layout: Atlas_Core.Atlas_Default) {\n dynamictext t1 (content: '%s')\n};\n", name, title, title) if _, err := f.WriteString(mdl); err != nil { t.Fatal(err) } @@ -195,7 +292,7 @@ func applyPage(t *testing.T, bin, dir, mpr, name, title string) { var stderr bytes.Buffer cmd.Stderr = &stderr if err := cmd.Run(); err != nil { - t.Fatalf("apply %s: %v\n%s", name, err, stderr.String()) + t.Fatalf("apply %s: %v\n%s", what, err, stderr.String()) } } diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 8a1a306c4..ba2a71261 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -4,6 +4,7 @@ package main import ( "bytes" + "database/sql" "encoding/json" "errors" "fmt" @@ -13,6 +14,7 @@ import ( "strings" "testing" + "github.com/mendixlabs/mxcli/modelsdk/mpr" "go.mongodb.org/mongo-driver/v2/bson" ) @@ -494,50 +496,187 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { } } -// A plan resolves per unit: tDM from ours, tMf from theirs — one merge commit, -// hashes repaired from the mixed files, note attached. -func TestGitMerge_PlanMixesPerUnit(t *testing.T) { - dir, mprPath := twoUnitConflictRepo(t) - doctorEnv(t, dir) - base := strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")) - ours := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) - theirs := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")) - plan := fmt.Sprintf(`{"base":%q,"ours":%q,"theirs":%q,"units":[{"id":%q,"take":"ours"},{"id":%q,"take":"theirs"}]}`, - base, ours, theirs, tDM, tMf) - planPath := filepath.Join(dir, "merge-plan.json") - if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { - t.Fatal(err) - } +// engineMarker is what `mx merge` writes into ContentsConflicts for a unit it +// left for a decision (measured, mx 11.13); every other row holds an empty +// string. +const engineMarker = `{"Conflicts":[{"ID":"00000000-0000-0000-0000-000000000000","HeadModification":"Change","MineModification":"Change"}]}` - noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { - t.Fatal("--plan must not consult mx") - return 0, "", nil - }} - var out bytes.Buffer - if err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out); err != nil { - t.Fatalf("mergeRun --plan: %v\n%s", err, out.String()) - } - assertUnitHas := func(id, key string, want bool) { - t.Helper() - raw, err := os.ReadFile(filepath.Join(dir, unitRel(id))) +// engineDecides simulates `mx merge` stopping on a decision (exit 2): it +// writes result into MINE as the merged project, marks the manual units in +// MINE's index the way the engine does, and prints the -v list naming manual +// units ManualResolutionRequired and auto units AutoResolved. +func engineDecides(t *testing.T, result map[string]tUnit, manual, auto []string) mxMergeRunner { + return func(base, mine, theirs string, shas [3]string) (int, string, error) { + writeProject(t, filepath.Dir(mine), "11.13.0", result) + db, err := sql.Open("sqlite", mine) if err != nil { t.Fatal(err) } - if got := bytes.Contains(raw, []byte(key)); got != want { - t.Errorf("%s contains %q = %v, want %v", id, key, got, want) + defer db.Close() + var list []string + for _, id := range manual { + if _, err := db.Exec(`UPDATE Unit SET ContentsConflicts = ? WHERE UnitID = ?`, engineMarker, mpr.IDToBsonBinary(id).Data); err != nil { + t.Fatal(err) + } + list = append(list, `{"UnitId":"`+id+`","Type":"MergeContents","TreeConflict":"None","ConflictResolutionDetails":"ManualResolutionRequired"}`) + } + for _, id := range auto { + list = append(list, `{"UnitId":"`+id+`","Type":"MergeContents","TreeConflict":"None","ConflictResolutionDetails":"AutoResolved"}`) } + return 2, "Merging MPRs.\n[" + strings.Join(list, ",") + "]\nConflicts found during merging.", nil + } +} + +// writePlanFor writes a plan for the merge in progress in dir — outside the +// repository, so it never shows in the working tree — taking each unit from +// the given side. +func writePlanFor(t *testing.T, dir string, take map[string]string) string { + t.Helper() + plan := mergePlanTemplate{ + Base: strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")), + Ours: strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")), + Theirs: strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")), + } + for id, side := range take { + plan.Units = append(plan.Units, planUnit{ID: id, Take: side}) + } + raw, err := json.Marshal(plan) + if err != nil { + t.Fatal(err) + } + p := filepath.Join(t.TempDir(), "merge-plan.json") + if err := os.WriteFile(p, raw, 0o644); err != nil { + t.Fatal(err) + } + return p +} + +func assertUnitHas(t *testing.T, dir, id, key string, want bool) { + t.Helper() + raw, err := os.ReadFile(filepath.Join(dir, unitRel(id))) + if err != nil { + t.Fatal(err) } - assertUnitHas(tDM, "DocA", true) - assertUnitHas(tDM, "DocB", false) - assertUnitHas(tMf, "DocB", true) - assertUnitHas(tMf, "DocA", false) + if got := bytes.Contains(raw, []byte(key)); got != want { + t.Errorf("%s contains %q = %v, want %v", id, key, got, want) + } +} + +// assertPlanMergeCommitted checks what every successful plan must leave: one +// two-parent merge commit, a clean working tree, an index whose hashes match +// the files, and no merge conflict marker left in it. +func assertPlanMergeCommitted(t *testing.T, dir, mprPath string) { + t.Helper() parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) if len(parents) != 3 { t.Fatalf("want a two-parent merge commit, got %v", parents) } - var out2 bytes.Buffer - if err := fixHashes(mprPath, false, &out2); err != nil { - t.Errorf("hashes drift after the plan merge: %v\n%s", err, out2.String()) + if st := gitOutT(t, dir, "status", "--porcelain"); strings.TrimSpace(st) != "" { + t.Errorf("working tree not clean after the plan merge:\n%s", st) + } + var out bytes.Buffer + if err := fixHashes(mprPath, false, &out); err != nil { + t.Errorf("hashes drift after the plan merge: %v\n%s", err, out.String()) + } + r, err := mpr.Open(mprPath) + if err != nil { + t.Fatal(err) + } + defer r.Close() + if marks, err := r.UnitConflicts(); err != nil || len(marks) != 0 { + t.Errorf("merge conflict markers left in the committed index: %+v (%v)", marks, err) + } +} + +// A plan resolves per unit: the engine left both units for a decision, and +// the plan takes tDM from ours and tMf from theirs — one merge commit, hashes +// repaired from the files, both markers cleared. +func TestGitMerge_PlanMixesPerUnit(t *testing.T) { + dir, mprPath := twoUnitConflictRepo(t) + doctorEnv(t, dir) + // The engine's result for undecided units: ours' content, marked. + result := baseUnits(t) + result[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} + result[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", mainSideKey)} + planPath := writePlanFor(t, dir, map[string]string{tDM: "ours", tMf: "theirs"}) + + mx := mxRunners{merge: engineDecides(t, result, []string{tDM, tMf}, nil)} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", planPath, false, mx, &out); err != nil { + t.Fatalf("mergeRun --plan: %v\n%s", err, out.String()) + } + assertUnitHas(t, dir, tDM, "DocA", true) + assertUnitHas(t, dir, tDM, "DocB", false) + assertUnitHas(t, dir, tMf, "DocB", true) + assertUnitHas(t, dir, tMf, "DocA", false) + assertPlanMergeCommitted(t, dir, mprPath) +} + +// The plan decides only what the engine could not. Here the engine merged +// tDM on its own (both sides' keys in its result) and left tMf for a +// decision; a plan deciding tMf must keep the engine's tDM. Taking tDM from +// ours instead drops theirs' change to it without the plan ever naming it — +// measured with real mx: an entity each side added to the same domain model +// is AutoResolved, while both sides editing one entity's documentation is not, +// and the plan's commit lost theirs' entity. +func TestGitMerge_PlanKeepsWhatTheEngineMerged(t *testing.T) { + dir, mprPath := twoUnitConflictRepo(t) + doctorEnv(t, dir) + result := baseUnits(t) + result[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey, sideKey)} + result[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", mainSideKey)} + planPath := writePlanFor(t, dir, map[string]string{tMf: "theirs"}) + + mx := mxRunners{merge: engineDecides(t, result, []string{tMf}, []string{tDM})} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", planPath, false, mx, &out); err != nil { + t.Fatalf("mergeRun --plan: %v\n%s", err, out.String()) + } + assertUnitHas(t, dir, tDM, "DocA", true) + assertUnitHas(t, dir, tDM, "DocB", true) // theirs' change, merged by the engine + assertUnitHas(t, dir, tMf, "DocB", true) + assertUnitHas(t, dir, tMf, "DocA", false) + assertPlanMergeCommitted(t, dir, mprPath) +} + +// A plan must decide exactly the units the engine left for a decision: one it +// leaves out would be committed with the engine's marker still in the index +// (CE0137 in Studio Pro), and one the engine merged on its own would undo +// that merge. Both are refused, and the refusal aborts the merge. +func TestGitMerge_PlanMustMatchTheEnginesDecisions(t *testing.T) { + for _, tc := range []struct { + name string + manual, auto []string + take map[string]string + want string + }{ + {"leaves out an undecided unit", []string{tDM, tMf}, nil, + map[string]string{tMf: "theirs"}, "does not decide"}, + {"decides a unit the engine merged", []string{tMf}, []string{tDM}, + map[string]string{tDM: "ours", tMf: "theirs"}, "merged without a decision"}, + } { + t.Run(tc.name, func(t *testing.T) { + dir, mprPath := twoUnitConflictRepo(t) + doctorEnv(t, dir) + result := baseUnits(t) + result[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} + result[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", mainSideKey)} + planPath := writePlanFor(t, dir, tc.take) + before := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) + + mx := mxRunners{merge: engineDecides(t, result, tc.manual, tc.auto)} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", planPath, false, mx, &out) + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("want a refusal mentioning %q, got %v\n%s", tc.want, err, out.String()) + } + if got := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")); got != before { + t.Errorf("HEAD moved despite the refusal: %s", got) + } + if _, mhErr := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr == nil { + t.Error("MERGE_HEAD still present: the refusal must abort the merge") + } + }) } } diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 77317d526..b9772f570 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -268,9 +268,14 @@ it is copied back, so the engine owns the `.mpr`: Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: `--plan ` resolves **per unit** from the plan template written beside the .mpr (`.merge-plan.json`, which `mxcli git commit` never stages): - set `"take"` to `ours`/`theirs` per unit, and the `.mpr` index is repaired - from the mixed files (modified-on-both units only; added/deleted/moved - conflicts go to Studio Pro). `--ours` or `--theirs` takes one side of every + set `"take"` to `ours`/`theirs` per unit. The engine runs again and the plan + decides **only the units it left open** — everything the engine merged + stays merged, so the template names just those units, and a plan that names + more or fewer (stale, or edited by hand) is refused. Each planned unit is + taken whole from its side, its conflict marker cleared (Studio Pro reports + a marker left in the index as CE0137), and the `.mpr` index repaired from + the files (modified-on-both units only; added/deleted/moved conflicts go to + Studio Pro). `--ours` or `--theirs` takes one side of every conflicted unit, hash-gated: when the sides' document sets differ (one side added or deleted a unit), committing a whole side would orphan the other side's documents, so the commit is refused and the merge aborted. Any other From e08a671db73a0fa7b3a23448f8cd5c5439d100b9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 20:25:48 +0000 Subject: [PATCH 61/61] docs(plans): mark the git-merge folder-mode plan superseded in part Its engine section (folder arguments, MINE as the working tree) is what failed on Linux; the shipped design takes three .mpr paths over clean worktrees, and --plan now resolves over the engine's result. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- docs/plans/2026-10-30-git-merge-mx-folder-mode.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/plans/2026-10-30-git-merge-mx-folder-mode.md b/docs/plans/2026-10-30-git-merge-mx-folder-mode.md index 85de356f7..992a22764 100644 --- a/docs/plans/2026-10-30-git-merge-mx-folder-mode.md +++ b/docs/plans/2026-10-30-git-merge-mx-folder-mode.md @@ -1,5 +1,7 @@ # Rework `mxcli git merge` per ako's review (PR #1674) +**Status:** Superseded in part. Section A did not hold on Linux: `mx merge` with folder arguments exits 129, and MINE as the working tree mid-merge crashes it, so the engine takes three `.mpr` paths over clean worktrees (the command's help and `docs-site/src/tools/outside-studio-pro.md` describe the shipped design). `--plan` no longer resolves from git's two sides either: it re-runs the engine and decides only the units the engine left open. + Decisions (user, 2026-10-30): switch the engine call to `mx merge` (folder mode); keep `--ours`/`--theirs` but limited to conflicted units with a hash gate; one push after the rework passes `make ci-local`.