Repository navigation
HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920) - #3032
Conversation
…set, envelope exemption BACKLOG #5801 added item 1: Message._ensure_owned copies the shared tree with _builtin_hl7.clone, which takes _lazy in one dict.copy call before deep-copying segments, so a concurrent lazy split cannot raise RuntimeError mid-copy. BACKLOG #5802: encode_with_separators writes a BHS or FHS header like MSH, with field 2 set to the target encoding characters, instead of writing its field separator as an extra field. BACKLOG #5917 step 1: SegmentGroup.set writes with a group-scoped occurrence, so a write after a fieldless bare boundary lands on this group's order. BACKLOG #5920 step 2: strict validation refuses a dropped MSH, BHS, BTS, FHS, FTS, MSA or ERR line even in a structure with the any-HL7-segment slot. #5920 steps 1 and 3 are not built (event-to-structure table and nested-slot grouping need a design decision). Proposed PR title: HL7 core: race-safe copy-on-write clone, BHS/FHS delimiter rewrite, SegmentGroup.set, envelope segments refused in any-segment structures (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801: CLOSED. Added item 1 fixed: clone() snapshots _lazy in one dict.copy; tests/test_message.py test fails on copy.deepcopy (RuntimeError) and passes now. #5802: CLOSED. Verified real (BHS|...|^~\& came back BHS|||^~\& even with an unchanged set); fixed with tests in test_mllp_encoding_override.py and test_builtin_hl7_header_no_trailing_separator.py. #5917: CLOSED if the Lander accepts SegmentGroup.set as step 1's fix. #5920: PARTIAL. Step 2 fixed with tests; steps 1 and 3 open, need a decision.
…p set_data Fixes from the round-1 code review of this branch: - encode_with_separators refuses a header line whose field 1 or 2 is not the message's own delimiters, instead of dropping its text; a character past the four encoding characters is kept unless it is a target delimiter. - BTS and FTS leave the never-exempt set, because split_batch leaves them on a batch's last member; DSC joins it. - SegmentGroup.set_data, sharing set's group-scoped occurrence. - A test that the copy-on-write copy never walks a shared _lazy map, with negative controls (deepcopy and a comprehension both fail it). - ADR 0206's accepted text is restored; the module docstring names the set once. Cites BACKLOG #5801, #5802, #5917, #5920. Proposed PR title: HL7 core: race-safe copy-on-write clone, BHS/FHS delimiter rewrite, SegmentGroup.set, control segments refused in any-segment structures (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy; tests in tests/test_message.py fail on copy.deepcopy and pass now. #5802: CLOSED. Verified real; BHS/FHS field 2 rewritten as MSH-2, a header it cannot rewrite refused; tests in test_mllp_encoding_override.py and test_builtin_hl7_header_no_trailing_separator.py. #5917: CLOSED if step 1 is met by SegmentGroup.set and set_data. #5920: PARTIAL. Step 2 fixed for BHS, FHS, MSA, ERR (plus DSC); BTS/FTS kept exempt because split_batch leaves them on the last member. Steps 1 and 3 open: they need a decision.
…odule Fixes from the round-2 code review of this branch: - A header whose field 2 starts with the source encoding characters takes the four target characters and drops the rest, as MSH did on main, so a v2.7+ MSH-2 truncation character no longer refuses the rewrite. - BHS and FHS leave the never-exempt set too: split_batch cuts only at MSH, so a member can carry the next batch's header. The set is MSH, DSC, MSA, ERR. - Message.__copy__ and __deepcopy__ return the copy-on-write copy, so the copy module neither walks a shared _lazy nor shares a tree unflagged. - Changelog and HL7-VALIDATION.md text match the code. Cites BACKLOG #5801, #5802, #5917, #5920. Proposed PR title: HL7 core: race-safe copy-on-write copy, BHS/FHS delimiter rewrite, SegmentGroup.set, DSC/MSA/ERR refused in any-segment structures (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and the copy module goes through Message.copy; tests in tests/test_message.py fail on copy.deepcopy and pass now. #5802: CLOSED. Verified real; BHS/FHS field 2 rewritten as MSH-2, a header it cannot rewrite refused; tests in test_mllp_encoding_override.py and test_builtin_hl7_header_no_trailing_separator.py. #5917: CLOSED if step 1 is met by SegmentGroup.set and set_data. #5920: PARTIAL. Step 2 fixed for MSA and ERR (plus DSC); the batch and file envelope stays exempt until split_batch stops leaving it on members. Steps 1 and 3 open: they need a decision.
…, QAK and MFA Manager-directed fixes after the unrepaired round on 9dcb806: - With a new delimiter set, a header field 2 holding text past the four encoding characters is refused; one v2.7 truncation character is kept, or left out where it is a target delimiter. - With the message's own delimiter set, header lines are written as they came: no refusal, and BHS| is not rewritten. - copy.deepcopy(msg) builds an independent tree through clone() and leaves the source unflagged; copy.copy and copy.deepcopy keep the subclass. - QAK and MFA join MSH, DSC, MSA and ERR in the never-exempt set. - The 5920 fragment is renamed and both fragments say what the code does. Cites BACKLOG #5801, #5802, #5920. Proposed PR title: HL7 core: race-safe copy-on-write copy, BHS/FHS delimiter rewrite, SegmentGroup.set, continuation and acknowledgment segments refused in any-segment structures (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and copy.copy/copy.deepcopy go through it; tests in tests/test_message.py fail on copy.deepcopy of the tree and pass now. #5802: CLOSED. Verified real; with a new set, BHS/FHS field 2 rewritten as MSH-2 and a header holding other text refused; with the same set, written as it came; tests in test_mllp_encoding_override.py and test_builtin_hl7_header_no_trailing_separator.py. #5917: CLOSED. Step 1 met by SegmentGroup.set and set_data (Manager accepted). #5920: PARTIAL. Step 2 fixed for DSC, MSA, ERR, QAK, MFA; the batch and file envelope waits on split_batch (after bundle S). Step 1: the same-code borrow stays as pinned (Manager ruling). Step 3 open.
The __copy__ comment, the encode_with_separators refusal sentence and the 5802 changelog fragment now say what the code does. No behaviour change. Cites BACKLOG #5802. Proposed PR title: HL7 core: race-safe copy-on-write copy, BHS/FHS delimiter rewrite, SegmentGroup.set, continuation and acknowledgment segments refused in any-segment structures (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801: CLOSED. Added item 1 fixed: clone() takes _lazy in one dict.copy, and copy.copy/copy.deepcopy go through it; tests in tests/test_message.py fail on copy.deepcopy of the tree and pass now. #5802: CLOSED. Verified real; with a new set, BHS/FHS field 2 rewritten as MSH-2 and a header holding other text refused; with the same set, written as it came; tests in test_mllp_encoding_override.py and test_builtin_hl7_header_no_trailing_separator.py. #5917: CLOSED. Step 1 met by SegmentGroup.set and set_data (Manager accepted). #5920: PARTIAL. Step 2 fixed for DSC, MSA, ERR, QAK, MFA; the batch and file envelope waits on split_batch (after bundle S). Step 1: the same-code borrow stays as pinned (Manager ruling). Step 3 open.
|
Covering review (Manager be8c6eff): whole diff origin/main...1196670, xhigh, nonce round-e565d932, reviewer fork ab901132160e541c3, review_tag.py exit 0. Tag: 13 findings. A repair Builder is now fixing 1, 2, 4, 5 and 12 (restore main's MSH-2 handling where this branch changed it; refuse a control/space/delimiter truncation character; the _NEVER_EXEMPT comment and the 5920 security fragment made true). The new head and its own covering round will be posted here before handover. Not ready to merge yet.
|
…haracter The delimiter rewrite changes behaviour only on BHS and FHS lines now. An MSH line takes the origin/main handling again: MSH-2 always becomes the four target characters, under the same set too, and an MSH line is never refused. A BHS or FHS field 2 keeps a fifth character only when it is printable, not a space and not a source or target delimiter; anything else is refused, the fail-closed choice. The _NEVER_EXEMPT comment now names the six IDs it holds and says at least ADD, SFT, UAC, QPD, QRD, QRF and RCP keep the exemption. The 5920 fragment names MSH, and the 5802 fragment says MSH is rewritten as before. Review findings 1, 2, 4, 5 and 12 on PR 3032. BACKLOG #5802, #5920. Proposed PR title: HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801 SHIPPED: a copy-on-write copy takes the lazy map in one dict.copy call. #5802 SHIPPED: the delimiter override rewrites BHS and FHS field 2; MSH unchanged. #5917 SHIPPED: SegmentGroup.set and set_data write within one order group. #5920 SHIPPED: a dropped MSH, DSC, MSA, ERR, QAK or MFA line is refused in an any-segment structure.
…gment The lead said strict validation refuses a dropped header, continuation or acknowledgment segment, a completeness claim: BHS, FHS, ADD, SFT and the query segments keep the exemption. It now says six more segments, which the body names. Review round-81738618 finding 4 on PR 3032. BACKLOG #5920. Proposed PR title: HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920) Proposed banners: #5801 SHIPPED: a copy-on-write copy takes the lazy map in one dict.copy call. #5802 SHIPPED: the delimiter override rewrites BHS and FHS field 2; MSH unchanged. #5917 SHIPPED: SegmentGroup.set and set_data write within one order group. #5920 SHIPPED: a dropped MSH, DSC, MSA, ERR, QAK or MFA line is refused in an any-segment structure.
|
Manager be8c6eff: repair round 2 pushed, head cbf480a. MSH lines now encode exactly as origin/main (a comparison script found MSH byte-identical across 24 input/target pairs; new test Review on the final head (no commit after it): xhigh round-e59bc17a, fork a0148476e2096ce2b, review_tag.py exit 0, tag Behaviour changes for the Lander to weigh (Manager decision: keep refusal; a silent drop or the main-era doubled separator loses or corrupts text): (a) a BHS/FHS truncation character equal to a target delimiter is refused, so with CONNECTIONS.md's example override Five fields: PR 3032; head cbf480a; review proof round-e59bc17a; open findings as above, (a) and (b) change behaviour; squash with an updated title: "HL7 core: race-safe copy-on-write clone, batch header rewrite, group set, continuation and acknowledgment refused in an any-segment structure (BACKLOG #5801, #5802, #5917, #5920)". |
|
Lander: merge decision d20261117T174711884-6d6abe, decided by the Lander under the owner's delegation of 2026-11-17 (owner table question 'accept the two deliberate changes?', answered /driver). A sonnet adversarial review AFFIRMED it: both changes refuse and dead-letter, which is a logged ERROR, rather than drop. Covering review: round-e59bc17a on cbf480a (acct7 fork started 15:10:38Z, after the 15:01:21Z head commit). |
|
Lander pin: enqueuing at head cbf480a, behind PR 3035. All 9 required contexts are green. Review proof: round-e59bc17a. Its open findings were named in my earlier comment and are queued for filing. |
|
CI failed in Its own head can still be green: the queue revalidates the merge, and the path gates that skip on a pull request run there. Read the run before retrying. https://github.com/MEFORORG/MessageFoundry/actions/runs/44179423570 |
|
Lander: evicted from the queue at 22:03Z by merge-group run 44185 (CI run 44179423570), test shard 2 (windows-2025): tests/test_uploads_cross_process_quota.py::test_the_shutdown_drain_does_not_wait_for_a_heartbeat_touch[ended-slow-unwind], assert 0 == 1. Blast radius: this PR changes only parsing/, its tests, docs/HL7-VALIDATION.md and changelog fragments; none intersects the uploads quota or the web console shutdown drain, and the test is on the known flake list. Pre-registered reading for the one re-run (a re-enqueue at the same pin cbf480a): green, or a different unrelated test, means flake and it lands; the SAME test failing again means a finding, and I stop and file it rather than enqueue a third time. |
Merge danger
Reversible: yes -- a revert of the squash commit restores the HL7 core.
Blast radius: the built-in HL7 parser and message model (
parsing/_builtin_hl7.py,message.py,groups.py,validate.py), docs/HL7-VALIDATION.md, changelog fragments, tests. Touches every inbound and outbound HL7 path: message copy-on-write, delimiter rewrite of BHS/FHS headers on an outbound encoding override, group-scoped writes, and strict validation of dropped segments.Evidence
copy.deepcopythe copy-on-write test fails (RuntimeError: dictionary changed size during iteration); on mainBHS|^~\&came backBHS|||^~\&with an unchanged set andBHS|^~\&EXTRA|Bsilently lost EXTRA; a same-set override turnedBHS|intoBHS|^~\&; dropped MSA/ERR/DSC passed strict validation in "any segment" structures.BACKLOG #5801, #5802, #5917, #5920 (batch 264), Builder report
Source branch
mgr7-w3-R-hl7core, head1196670bb8191505c0568d36f125bcef067ed488, base origin/mainf0e49f45c5c._builtin_hl7.clone()(takes_lazyin onedict.copy);copy.copy/copy.deepcopyroute through it; a deep copy shares nothing and keeps the subclass.DelimiterRewriteRefused(before: silently corrupted); with the same set, header fields are written as they came.SegmentGroup.setandset_datawrite with a group-scoped occurrence (ordinalstays positional).split_batchleaves them on members.split_batch(bundle S); #5917 step 1 accepted as met.Checks run: ruff,
git diff --check, mypy linux and win32 packages legs and the tests leg on changed files, pytest as above. Hosted legs: none specific.Open findings (round-7f88d6e76b on 9dcb806..a448e89): 1 a truncation character that is a target delimiter is dropped (deliberate, documented); 2
__deepcopy__records no memo for the old_m; 3 subclass__slots__values are lost on copy; 4 behaviour change on an outbound override: MSH-2 is no longer normalised to four characters (a same-set override keeps^~\&*and a short^~); 5Message.copy()builds a base Message while__copy__usestype(self); 6 the 5920 fragment's list leaves out MSH; 7ADDis still exempted; 8-9 efficiency and reuse; 10 no test deep-copies a message with a livecopy()clone. Earlier rounds: any-position BHS/BTS/FHS/FTS exemption and ADT batch last members (await split_batch);add_repetition/repetitionshave no group-scoped form;add_segment('BHS|...')corruption and the BHS/FHS-1/2set_datagap (pre-existing); docs/CONNECTIONS.md does not name the new refusal (open PR 3019 owns it).Proposed banners: