vms-c55: the executive holds cross-node lock traffic for the duration of a cluster state transition

[baron-3dl]

What this is

A real OpenVMS VAX V7.3 bugchecks LOCKMGRERR in the cluster demo at about 1 in 9 fires. The previous lead — "OVMX is a member the VAX admitted that silently refuses what it doesn't recognise" — is falsified here, and a different signal, on the cause side, discriminates just as sharply.

The refusals, measured everywhere they exist

Two committed crash captures plus the twelve case-runs of openvmx-site 38811815171 / 38811829040 / 38811843112 / 38814006004 (frames + all three consoles each):

• %DLM, refusing a lock message appears once in the whole corpus — cat 0x02 op 0x19, in the one crashing case-run.
• %CNXMAN, an unroutable VMS$VAXcluster frame was received — the other place an inbound frame can be dropped — appears zero times in all fourteen runs.
• Every other inbound frame is answered, or is one that is never answered (the 01/0a GO, the 01/0c release, the 01/06 burst — all consumed by the barrier/join FSMs).

So the refusal set is {02/19}, and nothing OVMX refused can have provoked it.

What does discriminate

OVMX originates a cat-0x02 lock request while a cluster state transition is running.

	OVMX cat-0x02 originated IN a transition	outside	bugcheck
3 crash runs	3 (02/04@step1 ×3, 02/01@step2 ×2)	1	YES ×3
11 clean runs	0	26	no

And over all 122 committed pcaps, real DEC-OUI senders only: inside a transition the DLM traffic is the op-0x0d rebuild record and its echo (2797 + 1240) and essentially nothing else; every non-0x0d cat-0x02 frame a real node put inside one flew at barrier step 10, 11 or 12 of 12 — zero at steps 1–5. In the one capture holding a whole 12-step barrier the in-window breakdown is exactly 307 × 02/0d, 307 × 82/0d and the barrier frames.

That is the published behaviour, measured: a transition quiesces the lock manager — Phase 1 discards all directory information cluster-wide, Phase 2 rebuilds it (Davis, VAXcluster Principles, pp. 7-40/7-41, p. 6-33).

Why OVMX did it

There was no stall. dlm_arm_transition_begin() only counted the boundary, and dq_may_transmit() — the single gate every outbound lock frame passes — had no transition condition. The quorum hang deliberately does not cover it (vms_dlm_quorum.h limit 1: "this node does not stall the POST of such a request … deferring the post until resume is a queue this rung does not build"). This is that queue.

The change

Executive-resident, reading only state the executive already holds:

• struct dlm_req_ops gains transition_active — a live read of the CLUB's own transition_active, the flag the barrier and coordinator FSMs already maintain and cnxman_dlm_seq_learn already gates on. An injected op, not a copy, for the same INV-6 reason dir_generation is one.
• dq_may_transmit() holds an ENQ or a CONVERT while it reads true. The block is retained exactly as vms-97c2 retains a frame the connection manager could not take, and the hold does not spend a rung of the retransmit ladder: a transition makes lock activity wait, it does not fail it.
• a directory-entry removal raised during one is dropped, counted — it has no sleeper behind it and nothing to re-read, and Phase 1 discards the entry cluster-wide anyway.
• a $DEQ still goes — giving a lock back is how the cluster is left in a state the rebuild can use, the same carve-out vms_dlm_quorum.h makes for a release during a quorum hang.
• dlm_arm_transition_end() releases the hold last, after the new directory vector has been applied, through the ordinary ladder beat — so a released request is built from a fresh refill_post read under the new vector.

No new wire shape is emitted. The change moves when two existing shapes go out.

Proof

R1 test_dlm_requester.c	410 checks, 0 failures (41 new)
test_dlm_scs_arm.c	131 checks — the two bindings and the ORDER of the release
negative controls	4 new, each reddening exactly its require_fail set; full host negctl 134/134 PASS
census selftest	tools/cluster/transition_dlm_census.py --selftest, wired into ctest as cluster_host_transition_dlm_census — re-derives every figure above from the committed corpus
ctest	355/355, cluster 110/110
Linux	vms.ko builds clean

Evidence: tests/lab/captures/vms-c55-transition-hold-20261016 — the THIRD frame-level LOCKMGRERR specimen (openvmx-site 38811843112, page-order-throttled4, the recurrence that carried a nonzero PID cell), the census, and PREDICTION.md pre-registering the demo-gate fire.

Not claimed

That this is the cause. What cat-0x02 op-0x19 IS remains unknown and is still not guessed — no answer frame invented, RULE C not relaxed, the removal not suppressed outside a transition (vms-e09's 8,106 real removals stand). The demo-gate fire scored against PREDICTION.md is what says whether the crash stops.

🤖 Generated with Claude Code

[baron-3dl]

FIRE DISPATCHED 2026-10-16 ~01:38Z, scoring tests/lab/captures/vms-c55-transition-hold-20261016/PREDICTION.md, which was pushed with the commit and BEFORE these were fired.

Four preflights, each three cases (b-first, all-at-once, page-order-throttled4 — the three orders a LOCKMGRERR has been seen in), capture=true, ref=work/vms-c55-transition-hold:

• openvmx-site 38836182130
• openvmx-site 38836196104
• openvmx-site 38836210245
• openvmx-site 38836224210

Node B was rebuilt from THIS head first (vms run 38833987086, success) and rd vms-4f4 has since put the cases input on site main alongside Node B staging from the build under test — so for the first time a case-filtered run can be the build under test end to end (P6).

CI on this PR: 115 success / 51 skipped / 0 failures, including NetBSD/vax vms module cross-compiles — both substrates clean.

How to score, in this order (P1 carries the argument, not P5):

• P1 transition_dlm_census.py emitter <frames.jsonl> — the in-txn column must be 0 for every case-run. Baseline 3 of 3 crash runs had one.
• P2 the same tool's outside column must be nonzero somewhere, or P1 is vacuous: a fire in which OVMX emits no lock requests at all proves nothing.
• P3 both OVMX nodes still reach a logged-in DCL $; no new DLM timeout failures on their consoles.
• P4 no new (category, opcode) pair — this change emits no new shape.
• P5 zero LOCKMGRERR across ≥12 case-runs, against a 1-in-9 baseline, with the refusing a lock message (cat 0x02 op 0x19) string as the companion signal (5 of 5 crashes, 0 of 30 clean to date).

A recurrence is not a dead end: capture=true, so the window around any bugcheck says whether an OVMX cat-0x02 was inside a transition when the VAX died — and if it was not, that falsifies this diagnosis the same way the nonzero PID cell falsified the last one.

[baron-3dl]

FIRE SCORED — 24 case-runs across two arms. Committed as tests/lab/captures/vms-c55-transition-hold-20261016/FIRE.md.

	hold (18 case-runs)	control / main (6 case-runs)
OVMX cat-0x02 originated inside a transition (P1)	0	0
the same, outside one (P2)	232+	28
a new emitted (cat, op) pair (P4)	none	none
LOCKMGRERR (P5)	0	0
refusing a lock message (cat 0x02 op 0x19)	0	0
CN=3	18 of 18	6 of 6
OVMXB reaches Username: (P3)	no, 17 of 18	no, 6 of 6

What it settles

1. The hold is live. In 38846164733 / b-first the OVMX/VAX node's console carries, once each, %DLM, holding cross-node lock requests until the VAXcluster state transition completes and %DLM, the VAXcluster state transition completed; held cross-node lock requests resume. The mechanism is exercised by real traffic, not only by its tests, and the node went on serving afterwards.
2. No regression. OVMXB never reaching Username: is present identically on main — 6 of 6 — so it is rd vms-97c2 (P0, open; vms-97c2 (PARTIAL): a cross-node lock request whose frame could not go out is retried, never dropped on a sleeping $ENQW — measured NOT to be the demo stall #1729 was partial), made visible for the first time by rd vms-4f4 putting a real Node B in the gate. Without the control arm this fire would have read as a regression caused by the hold.
3. No peer harmed in either arm.

What it does NOT settle, said plainly

That the LOCKMGRERR is fixed. The control did not reproduce it either (0 in 6, where the baseline is 1 in 9–12), so 0 of 18 is consistent with the fix working and with the crash simply not firing. And P1 is not a discrimination in these runs — the control also shows 0 in-transition originations, so the condition the hold guards against did not arise there. P1's categorical content is a property of the code, proven at R1 and R2 where the condition is forced.

So the change stands on what it was defended on and nothing more: real VMS nodes do not put lock traffic into a rebuild; OVMX did, in 3 of 3 crash captures and 0 of 11 clean ones; this removes it, harms nothing and emits no new shape. vms-c55 stays open, and vms-8cb5's question — what cat-0x02 op-0x19 IS — is still unanswered and still not guessed.

Also added since the first review: the R2 leg (the REAL vms_lock.c engine's $ENQ waits across a transition and is granted when it lifts), the console diagnostic and its fifth negative control. Full host negctl 135/135, cluster ctest 110/110, CI green including NetBSD/vax vms module cross-compiles.

Queuing for merge.