Skip to content

HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920) - #3032

Queued
wshallwshall wants to merge 7 commits into
mainfrom
mgr7-w3-R-hl7core
Queued

wshallwshall wants to merge 7 commits into
mainfrom
mgr7-w3-R-hl7core

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Merge danger

Reversible: yes -- a revert of the squash commit restores the HL7 core.

Blast radius: the built-in HL7 parser and message model (parsing/_builtin_hl7.py, message.py, groups.py, validate.py), docs/HL7-VALIDATION.md, changelog fragments, tests. Touches every inbound and outbound HL7 path: message copy-on-write, delimiter rewrite of BHS/FHS headers on an outbound encoding override, group-scoped writes, and strict validation of dropped segments.

Evidence

  • Before: on old copy.deepcopy the copy-on-write test fails (RuntimeError: dictionary changed size during iteration); on main BHS|^~\& came back BHS|||^~\& with an unchanged set and BHS|^~\&EXTRA|B silently lost EXTRA; a same-set override turned BHS| into BHS|^~\&; dropped MSA/ERR/DSC passed strict validation in "any segment" structures.
  • After: changed-area files 591 passed, 1 skipped (at the head); a wider -k subset 4827 passed at 9dcb806.

BACKLOG #5801, #5802, #5917, #5920 (batch 264), Builder report

Source branch mgr7-w3-R-hl7core, head 1196670bb8191505c0568d36f125bcef067ed488, base origin/main f0e49f45c5c.

  • #5801: copy-on-write uses _builtin_hl7.clone() (takes _lazy in one dict.copy); copy.copy/copy.deepcopy route through it; a deep copy shares nothing and keeps the subclass.
  • #5802: with a new delimiter set, BHS/FHS field 2 is rewritten as MSH-2 is; a header field 2 holding more than the encoding characters plus one truncation character, or fields 1/2 not the message's own delimiters, raises DelimiterRewriteRefused (before: silently corrupted); with the same set, header fields are written as they came.
  • #5917: SegmentGroup.set and set_data write with a group-scoped occurrence (ordinal stays positional).
  • #5920 step 2: a dropped DSC, MSA, ERR, QAK or MFA is refused in "any HL7 segment" structures. BHS/BTS/FHS/FTS stay exempt because split_batch leaves them on members.
  • Manager decisions: #5920 step 1 keeps the pinned same-code borrow; the batch/file envelope half waits on split_batch (bundle S); #5917 step 1 accepted as met.

Checks run: ruff, git diff --check, mypy linux and win32 packages legs and the tests leg on changed files, pytest as above. Hosted legs: none specific.

Open findings (round-7f88d6e76b on 9dcb806..a448e89): 1 a truncation character that is a target delimiter is dropped (deliberate, documented); 2 __deepcopy__ records no memo for the old _m; 3 subclass __slots__ values are lost on copy; 4 behaviour change on an outbound override: MSH-2 is no longer normalised to four characters (a same-set override keeps ^~\&* and a short ^~); 5 Message.copy() builds a base Message while __copy__ uses type(self); 6 the 5920 fragment's list leaves out MSH; 7 ADD is still exempted; 8-9 efficiency and reuse; 10 no test deep-copies a message with a live copy() clone. Earlier rounds: any-position BHS/BTS/FHS/FTS exemption and ADT batch last members (await split_batch); add_repetition/repetitions have no group-scoped form; add_segment('BHS|...') corruption and the BHS/FHS-1/2 set_data gap (pre-existing); docs/CONNECTIONS.md does not name the new refusal (open PR 3019 owns it).

Proposed banners:

  • #5801 CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and copy.copy/copy.deepcopy go through it; tests fail on a copy.deepcopy of the tree.
  • #5802 CLOSED. Verified real. With a new set, BHS/FHS field 2 is rewritten as MSH-2 is, and a header holding other text is refused; with the same set, the header is written as it came.
  • #5917 CLOSED. Step 1 met by SegmentGroup.set and set_data (Manager accepted).
  • #5920 PARTIAL. Step 2 fixed for DSC, MSA, ERR, QAK and MFA; the envelope waits on split_batch; step 1 kept as pinned (Manager ruling); step 3 open.

…set, envelope exemption

BACKLOG #5801 added item 1: Message._ensure_owned copies the shared tree with
_builtin_hl7.clone, which takes _lazy in one dict.copy call before deep-copying
segments, so a concurrent lazy split cannot raise RuntimeError mid-copy.

BACKLOG #5802: encode_with_separators writes a BHS or FHS header like MSH, with
field 2 set to the target encoding characters, instead of writing its field
separator as an extra field.

BACKLOG #5917 step 1: SegmentGroup.set writes with a group-scoped occurrence,
so a write after a fieldless bare boundary lands on this group's order.

BACKLOG #5920 step 2: strict validation refuses a dropped MSH, BHS, BTS, FHS,
FTS, MSA or ERR line even in a structure with the any-HL7-segment slot.
#5920 steps 1 and 3 are not built (event-to-structure table and nested-slot
grouping need a design decision).

Proposed PR title: HL7 core: race-safe copy-on-write clone, BHS/FHS delimiter
rewrite, SegmentGroup.set, envelope segments refused in any-segment structures
(BACKLOG #5801, #5802, #5917, #5920)

Proposed banners:
#5801: CLOSED. Added item 1 fixed: clone() snapshots _lazy in one dict.copy;
tests/test_message.py test fails on copy.deepcopy (RuntimeError) and passes now.
#5802: CLOSED. Verified real (BHS|...|^~\& came back BHS|||^~\& even with an
unchanged set); fixed with tests in test_mllp_encoding_override.py and
test_builtin_hl7_header_no_trailing_separator.py.
#5917: CLOSED if the Lander accepts SegmentGroup.set as step 1's fix.
#5920: PARTIAL. Step 2 fixed with tests; steps 1 and 3 open, need a decision.
…p set_data

Fixes from the round-1 code review of this branch:
- encode_with_separators refuses a header line whose field 1 or 2 is not the
  message's own delimiters, instead of dropping its text; a character past
  the four encoding characters is kept unless it is a target delimiter.
- BTS and FTS leave the never-exempt set, because split_batch leaves them on
  a batch's last member; DSC joins it.
- SegmentGroup.set_data, sharing set's group-scoped occurrence.
- A test that the copy-on-write copy never walks a shared _lazy map, with
  negative controls (deepcopy and a comprehension both fail it).
- ADR 0206's accepted text is restored; the module docstring names the set
  once.

Cites BACKLOG #5801, #5802, #5917, #5920.

Proposed PR title: HL7 core: race-safe copy-on-write clone, BHS/FHS delimiter
rewrite, SegmentGroup.set, control segments refused in any-segment structures
(BACKLOG #5801, #5802, #5917, #5920)

Proposed banners:
#5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy;
tests in tests/test_message.py fail on copy.deepcopy and pass now.
#5802: CLOSED. Verified real; BHS/FHS field 2 rewritten as MSH-2, a header
it cannot rewrite refused; tests in test_mllp_encoding_override.py and
test_builtin_hl7_header_no_trailing_separator.py.
#5917: CLOSED if step 1 is met by SegmentGroup.set and set_data.
#5920: PARTIAL. Step 2 fixed for BHS, FHS, MSA, ERR (plus DSC); BTS/FTS kept
exempt because split_batch leaves them on the last member. Steps 1 and 3
open: they need a decision.
…odule

Fixes from the round-2 code review of this branch:
- A header whose field 2 starts with the source encoding characters takes
  the four target characters and drops the rest, as MSH did on main, so a
  v2.7+ MSH-2 truncation character no longer refuses the rewrite.
- BHS and FHS leave the never-exempt set too: split_batch cuts only at MSH,
  so a member can carry the next batch's header. The set is MSH, DSC, MSA,
  ERR.
- Message.__copy__ and __deepcopy__ return the copy-on-write copy, so the
  copy module neither walks a shared _lazy nor shares a tree unflagged.
- Changelog and HL7-VALIDATION.md text match the code.

Cites BACKLOG #5801, #5802, #5917, #5920.

Proposed PR title: HL7 core: race-safe copy-on-write copy, BHS/FHS delimiter
rewrite, SegmentGroup.set, DSC/MSA/ERR refused in any-segment structures
(BACKLOG #5801, #5802, #5917, #5920)

Proposed banners:
#5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and
the copy module goes through Message.copy; tests in tests/test_message.py
fail on copy.deepcopy and pass now.
#5802: CLOSED. Verified real; BHS/FHS field 2 rewritten as MSH-2, a header it
cannot rewrite refused; tests in test_mllp_encoding_override.py and
test_builtin_hl7_header_no_trailing_separator.py.
#5917: CLOSED if step 1 is met by SegmentGroup.set and set_data.
#5920: PARTIAL. Step 2 fixed for MSA and ERR (plus DSC); the batch and file
envelope stays exempt until split_batch stops leaving it on members. Steps 1
and 3 open: they need a decision.
…, QAK and MFA

Manager-directed fixes after the unrepaired round on 9dcb806:
- With a new delimiter set, a header field 2 holding text past the four
  encoding characters is refused; one v2.7 truncation character is kept, or
  left out where it is a target delimiter.
- With the message's own delimiter set, header lines are written as they
  came: no refusal, and BHS| is not rewritten.
- copy.deepcopy(msg) builds an independent tree through clone() and leaves
  the source unflagged; copy.copy and copy.deepcopy keep the subclass.
- QAK and MFA join MSH, DSC, MSA and ERR in the never-exempt set.
- The 5920 fragment is renamed and both fragments say what the code does.

Cites BACKLOG #5801, #5802, #5920.

Proposed PR title: HL7 core: race-safe copy-on-write copy, BHS/FHS delimiter
rewrite, SegmentGroup.set, continuation and acknowledgment segments refused in
any-segment structures (BACKLOG #5801, #5802, #5917, #5920)

Proposed banners:
#5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and
copy.copy/copy.deepcopy go through it; tests in tests/test_message.py fail on
copy.deepcopy of the tree and pass now.
#5802: CLOSED. Verified real; with a new set, BHS/FHS field 2 rewritten as
MSH-2 and a header holding other text refused; with the same set, written as
it came; tests in test_mllp_encoding_override.py and
test_builtin_hl7_header_no_trailing_separator.py.
#5917: CLOSED. Step 1 met by SegmentGroup.set and set_data (Manager accepted).
#5920: PARTIAL. Step 2 fixed for DSC, MSA, ERR, QAK, MFA; the batch and file
envelope waits on split_batch (after bundle S). Step 1: the same-code borrow
stays as pinned (Manager ruling). Step 3 open.
The __copy__ comment, the encode_with_separators refusal sentence and the
5802 changelog fragment now say what the code does. No behaviour change.

Cites BACKLOG #5802.

Proposed PR title: HL7 core: race-safe copy-on-write copy, BHS/FHS delimiter
rewrite, SegmentGroup.set, continuation and acknowledgment segments refused in
any-segment structures (BACKLOG #5801, #5802, #5917, #5920)

Proposed banners:
#5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and
copy.copy/copy.deepcopy go through it; tests in tests/test_message.py fail on
copy.deepcopy of the tree and pass now.
#5802: CLOSED. Verified real; with a new set, BHS/FHS field 2 rewritten as
MSH-2 and a header holding other text refused; with the same set, written as
it came; tests in test_mllp_encoding_override.py and
test_builtin_hl7_header_no_trailing_separator.py.
#5917: CLOSED. Step 1 met by SegmentGroup.set and set_data (Manager accepted).
#5920: PARTIAL. Step 2 fixed for DSC, MSA, ERR, QAK, MFA; the batch and file
envelope waits on split_batch (after bundle S). Step 1: the same-code borrow
stays as pinned (Manager ruling). Step 3 open.
@wshallwshall wshallwshall added the qa Builder QA record posted; not a merge gate label Nov 17, 2026
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Covering review (Manager be8c6eff): whole diff origin/main...1196670, xhigh, nonce round-e565d932, reviewer fork ab901132160e541c3, review_tag.py exit 0. Tag: xhigh effort -> 10 inline angles -> dedup (no verify) -> sweep -> <=15 findings.

13 findings. A repair Builder is now fixing 1, 2, 4, 5 and 12 (restore main's MSH-2 handling where this branch changed it; refuse a control/space/delimiter truncation character; the _NEVER_EXEMPT comment and the 5920 security fragment made true). The new head and its own covering round will be posted here before handover. Not ready to merge yet.

  1. _builtin_hl7.py:1256 same-set override no longer normalises MSH-2 (behaviour change). 2. validate.py:373 _NEVER_EXEMPT comment states a rationale the six-ID list only partly covers (ADD, SFT, UAC, QPD, QRD, QRF, RCP). 3. validate.py:413 BHS/BTS/FHS/FTS exemption applies on MLLP too. 4. _builtin_hl7.py:1297 any fifth char accepted as truncation, control chars included. 5. _builtin_hl7.py:1261 MSH can now raise DelimiterRewriteRefused where main delivered. 6. message.py:565 subclass slots lost on copy. 7. message.py:580 deepcopy memo. 8. _builtin_hl7.py:267 race fix rests on a docstring rule. 9. groups.py:150 repetitions/add_repetition not group-scoped. 10-11 efficiency. 12. 5920 security fragment omits MSH. 13. ('MSH','BHS','FHS') literal duplicated.

…haracter

The delimiter rewrite changes behaviour only on BHS and FHS lines now. An MSH line takes the
origin/main handling again: MSH-2 always becomes the four target characters, under the same set
too, and an MSH line is never refused. A BHS or FHS field 2 keeps a fifth character only when it
is printable, not a space and not a source or target delimiter; anything else is refused, the
fail-closed choice. The _NEVER_EXEMPT comment now names the six IDs it holds and says at least
ADD, SFT, UAC, QPD, QRD, QRF and RCP keep the exemption. The 5920 fragment names MSH, and the
5802 fragment says MSH is rewritten as before.

Review findings 1, 2, 4, 5 and 12 on PR 3032. BACKLOG #5802, #5920.

Proposed PR title: HL7 core: race-safe copy-on-write clone, batch header rewrite, group set,
continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802,
#5917, #5920)

Proposed banners:
#5801 SHIPPED: a copy-on-write copy takes the lazy map in one dict.copy call.
#5802 SHIPPED: the delimiter override rewrites BHS and FHS field 2; MSH unchanged.
#5917 SHIPPED: SegmentGroup.set and set_data write within one order group.
#5920 SHIPPED: a dropped MSH, DSC, MSA, ERR, QAK or MFA line is refused in an any-segment
structure.
…gment

The lead said strict validation refuses a dropped header, continuation or acknowledgment segment,
a completeness claim: BHS, FHS, ADD, SFT and the query segments keep the exemption. It now says
six more segments, which the body names. Review round-81738618 finding 4 on PR 3032.
BACKLOG #5920.

Proposed PR title: HL7 core: race-safe copy-on-write clone, batch header rewrite, group set,
continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802,
#5917, #5920)

Proposed banners:
#5801 SHIPPED: a copy-on-write copy takes the lazy map in one dict.copy call.
#5802 SHIPPED: the delimiter override rewrites BHS and FHS field 2; MSH unchanged.
#5917 SHIPPED: SegmentGroup.set and set_data write within one order group.
#5920 SHIPPED: a dropped MSH, DSC, MSA, ERR, QAK or MFA line is refused in an any-segment
structure.
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Manager be8c6eff: repair round 2 pushed, head cbf480a. MSH lines now encode exactly as origin/main (a comparison script found MSH byte-identical across 24 input/target pairs; new test test_an_msh_line_is_rewritten_as_main_rewrote_it passes with main's module and fails 9 cases on 1196670). BHS/FHS keep the rewrite; a BHS/FHS fifth character that is non-printable, a space or a source/target delimiter is REFUSED (fail closed). _NEVER_EXEMPT comment narrowed with "at least"; the 5920 fragment names MSH. Checks: ruff, mypy three legs, 734 + 353 tests passed.

Review on the final head (no commit after it): xhigh round-e59bc17a, fork a0148476e2096ce2b, review_tag.py exit 0, tag xhigh effort -> 10 inline angles -> dedup (no verify) -> sweep -> <=15 findings; earlier round-81738618 (fork a24b98d49929b5f8f) led to cbf480a.

Behaviour changes for the Lander to weigh (Manager decision: keep refusal; a silent drop or the main-era doubled separator loses or corrupts text): (a) a BHS/FHS truncation character equal to a target delimiter is refused, so with CONNECTIONS.md's example override #@*!% a v2.7 FHS|^~\&# batch dead-letters instead of dropping the character as MSH does; (b) one stray refused BHS line in an MLLP batch envelope dead-letters every member, where main delivered the envelope with a doubled separator.
Other open: BTS/FTS left on the last split_batch member make strict validation refuse it in ADT/ORU (pre-dates the PR; candidate row); MSH keeps main's silent drop and main's rewrite of a non-separator field 1 (by decision); __deepcopy__ memo; subclass __slots__; copy guard tests lack a firing control; clone() deepcopy cost; copy()/__copy__ disagree on subclasses; SegmentGroup.set scans; t_enc built thrice; add_repetition/repetitions not group-scoped; identity translate; _SplitOnWalkLazy intercepts only items(); never-exempt IDs restated four times (SDS-3.5); the 5802 fragment points at a private docstring.

Five fields: PR 3032; head cbf480a; review proof round-e59bc17a; open findings as above, (a) and (b) change behaviour; squash with an updated title: "HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920)".

@wshallwshall wshallwshall changed the title HL7 core: copy shares no lazy map, BHS/FHS header rewrite refuses foreign text, group-scoped writes, acknowledgment segments never exempt (BACKLOG #5801, #5802, #5917, #5920) HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920) Nov 17, 2026
@github-actions github-actions Bot added the ci-red A required check went red. Attribute it before retrying. label Nov 17, 2026
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Lander: merge decision d20261117T174711884-6d6abe, decided by the Lander under the owner's delegation of 2026-11-17 (owner table question 'accept the two deliberate changes?', answered /driver). A sonnet adversarial review AFFIRMED it: both changes refuse and dead-letter, which is a logged ERROR, rather than drop. Covering review: round-e59bc17a on cbf480a (acct7 fork started 15:10:38Z, after the 15:01:21Z head commit).
Open findings, one line each, all queued for filing: (a) and (b), the BHS/FHS refusals, recorded as accepted behaviour changes; BTS/FTS on the last split_batch member fail strict validation (pre-existing); the deepcopy memo; subclass slots; the copy guard tests lack a firing control; clone() deepcopy cost; copy()/copy disagree on subclasses; SegmentGroup.set scans; t_enc is built three times; add_repetition/repetitions are not group-scoped; the identity translate; _SplitOnWalkLazy intercepts only items(); the never-exempt IDs are restated four times; the 5802 fragment points at a private docstring.

@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Lander pin: enqueuing at head cbf480a, behind PR 3035. All 9 required contexts are green. Review proof: round-e59bc17a. Its open findings were named in my earlier comment and are queued for filing.

@wshallwshall
wshallwshall added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026
@github-actions

Copy link
Copy Markdown

CI failed in test shard 2 (windows-2025, py3.14) / Tests (pytest) while this pull request was in the merge queue, so the queue ejected it.

Its own head can still be green: the queue revalidates the merge, and the path gates that skip on a pull request run there. Read the run before retrying.

https://github.com/MEFORORG/MessageFoundry/actions/runs/44179423570

@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Lander: evicted from the queue at 22:03Z by merge-group run 44185 (CI run 44179423570), test shard 2 (windows-2025): tests/test_uploads_cross_process_quota.py::test_the_shutdown_drain_does_not_wait_for_a_heartbeat_touch[ended-slow-unwind], assert 0 == 1. Blast radius: this PR changes only parsing/, its tests, docs/HL7-VALIDATION.md and changelog fragments; none intersects the uploads quota or the web console shutdown drain, and the test is on the known flake list. Pre-registered reading for the one re-run (a re-enqueue at the same pin cbf480a): green, or a different unrelated test, means flake and it lands; the SAME test failing again means a finding, and I stop and file it rather than enqueue a third time.

@wshallwshall
wshallwshall added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-red A required check went red. Attribute it before retrying. qa Builder QA record posted; not a merge gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant