Skip to content
Merged
5 changes: 5 additions & 0 deletions changelog.d/5801-clone-lazy-snapshot.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **A copy-on-write copy no longer raises while another thread reads the shared message.** When a
`Message` and its `copy()` were read on two threads, the first write copied the shared tree with
`copy.deepcopy`. A lazy segment split on the other thread could change the map that copy was
walking, and the write raised `RuntimeError`. The copy now takes that map in one call, so the
split lands before or after it. (`BACKLOG #5801`)
8 changes: 8 additions & 0 deletions changelog.d/5802-batch-header-delimiters.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- **The outbound delimiter override now writes a `BHS` or `FHS` header correctly.** It wrote the
header's field separator as an extra field and changed only the escape character of its old
encoding characters, so every batch or file header field moved one place. With a new delimiter
set, field 2 of a `BHS` or `FHS` header now advertises the target encoding characters, as
`MSH-2` does. A `BHS` or `FHS` line whose fields 1 and 2 the rewrite cannot carry is refused,
which fails the delivery; the docstring of `_header_encoding` says which. With the message's own
delimiter set, `BHS` and `FHS` lines are written as they came. `MSH` lines are rewritten as
before. (`BACKLOG #5802`)
4 changes: 4 additions & 0 deletions changelog.d/5917-group-set.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **`SegmentGroup.set` and `SegmentGroup.set_data` write a field within one order group.** Their
`occurrence` counts within the group, as `SegmentGroup.field` does. A group's `ordinal` counts a fieldless bare boundary line
and `Message.set` does not, so `Message.set(..., occurrence=group.ordinal)` after a bare `OBR`
could write to another order. (`BACKLOG #5917`)
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **Strict validation refuses six more dropped segments in an "any HL7 segment" structure.** In a
structure such as `MFN_M01`, an `MSH`, `DSC`, `MSA`, `ERR`, `QAK` or `MFA` line that `hl7apy`
could not place was passed and listed in `unvalidated_segments`. It is now refused. Other defined
segments keep the exemption. (`BACKLOG #5920`)
4 changes: 3 additions & 1 deletion docs/HL7-VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ with its position, as it lists a Z-segment, though only when the comparison runs
`hl7apy` parsed but an earlier check refused lists its Z-segments alone, and a size, segment-cap
or parse refusal lists nothing. A dropped segment ID the version does not define is still
refused, and so is a dropped bare defined ID with no field separator (a lone `ORC` line in
`MFN_M01`); a bare line `hl7apy` placed is not. Every dropped segment is refused where MSH-9.3
`MFN_M01`); a bare line `hl7apy` placed is not. A dropped `MSH`, `DSC`, `MSA`, `ERR`, `QAK` or
`MFA` line is refused too. A batch or file envelope line (`BHS`, `BTS`, `FHS`, `FTS`) is exempted like any other
defined segment, because batch splitting can leave one on a message. Every dropped segment is refused where MSH-9.3
borrows a slot-bearing structure under another message code (`ADT^A01^MFN_M01`); a structure
borrowed under the same message code (`MFN^M02^MFN_M01`) still gets the exemption, since MSH-9.2
is not compared. Where the slot is required and its group is present, `hl7apy`'s own check
Expand Down
115 changes: 92 additions & 23 deletions messagefoundry/parsing/_builtin_hl7.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
Data model (a plain built-in structure)::

ParsedMessage = {
"_lazy": {index: raw line}, # first, so a deepcopy reads it before segments (_ensure_split)
"_lazy": {index: raw line}, # first, read before segments (_ensure_split, clone)
"segments": [ {"id": str, "fields": list[str | dict]}, ... ],
"raw": str | None, # cached encode; None once dirtied
"seps": (field, component, repetition, subcomponent, escape),
Expand Down Expand Up @@ -45,6 +45,7 @@

from __future__ import annotations

import copy
import operator
import re
from collections.abc import Callable
Expand All @@ -59,6 +60,7 @@
"HL7ParseError",
"DelimiterRewriteRefused",
"parse",
"clone",
"extract_field",
"extract_part",
"raw_field",
Expand Down Expand Up @@ -124,7 +126,8 @@ class Segment(TypedDict):

class ParsedMessage(TypedDict):
"""The whole parsed message — see the module docstring for the shape. Build it with ``_lazy``
as the first key, as :func:`parse` does: :func:`_ensure_split` relies on that order."""
as the first key, as :func:`parse` does, and copy it with :func:`clone`, never a walk of the
dict (:func:`_ensure_split` says why)."""

# Per-segment raw text awaiting a lazy split (index in ``segments`` → raw line). MSH is never
# here (parsed eagerly). A segment leaves this map the first time a field path touches it.
Expand Down Expand Up @@ -243,8 +246,8 @@ def parse(norm: str) -> ParsedMessage:
segments.append({"id": line.split(field_sep, 1)[0] if line else "", "fields": []})
lazy[index] = line

# ``_lazy`` goes in first on purpose: ``copy.deepcopy`` (``Message._ensure_owned``) copies a dict
# in insertion order, so it reads ``_lazy`` before ``segments``, the order _ensure_split needs.
# ``_lazy`` goes in first so a reader walking the dict in insertion order meets it before
# ``segments``, the order _ensure_split needs; clone() takes that order explicitly.
return {"_lazy": lazy, "segments": segments, "raw": strmsg + "\r", "seps": seps}


Expand All @@ -258,14 +261,12 @@ def _ensure_split(msg: ParsedMessage, seg_index: int) -> None:
The fields are stored **before** the raw line leaves ``_lazy``. A :meth:`Message.copy` clone
shares this tree until one side writes, so another thread can read the segment mid-split. That
can happen on a GIL build too, not only a free-threaded one: the GIL can switch threads between
storing the fields and the ``pop``, or inside a deepcopy's loop. A reader that looks at
``_lazy`` before ``fields`` then sees either the raw line or the full field list, never an
empty segment (BACKLOG #5801). That holds for at least the readers in this module and the
copy-on-write ``copy.deepcopy``, because :func:`parse` puts ``_lazy`` first in the dict; a new
reader must keep the same order. It does not make a deepcopy safe against the ``pop`` itself:
one that is iterating ``_lazy`` at that moment can still raise ``RuntimeError`` for a dict that
changed size. The ``pop`` keeps its default because a racing split of the same segment may have
removed it.
storing the fields and the ``pop``. A reader that looks at ``_lazy`` before ``fields`` then sees
either the raw line or the full field list, never an empty segment (BACKLOG #5801). That holds
for at least the readers in this module and :func:`clone`; a new reader must keep the same
order. The ``pop`` changes the size of ``_lazy``, so nothing may iterate a shared ``_lazy`` in
Python code: :func:`clone` copies it in one ``dict.copy`` call instead. The ``pop`` keeps its
default because a racing split of the same segment may have removed it.
"""
lazy = msg["_lazy"]
raw = lazy.get(seg_index)
Expand All @@ -279,6 +280,25 @@ def _ensure_split(msg: ParsedMessage, seg_index: int) -> None:
lazy.pop(seg_index, None)


def clone(msg: ParsedMessage) -> ParsedMessage:
"""A private deep copy of ``msg``, safe while another thread lazily splits the same tree.

The copy-on-write copy behind :meth:`Message._ensure_owned`. ``_lazy`` is copied first and in
one ``dict.copy`` call, which runs no Python code, so a concurrent :func:`_ensure_split` lands
wholly before or after it and its ``pop`` cannot change the dict's size mid-iteration, the
``RuntimeError`` a ``copy.deepcopy`` of the tree could raise (BACKLOG #5801). ``segments`` is
deep-copied after it, so a split landing in between leaves the copy holding the raw line, the
fields, or both, never neither. The raw lines are ``str`` and shared, as ``copy.deepcopy``
shares them.
"""
return {
"_lazy": msg["_lazy"].copy(), # first: a dict display evaluates in order
"segments": copy.deepcopy(msg["segments"]),
"raw": msg["raw"],
"seps": msg["seps"],
}


def segment_index(msg: ParsedMessage, seg_id: str, occurrence: int) -> int | None:
"""List index of the ``occurrence``-th (1-based) segment with ``seg_id``, or None if absent.

Expand Down Expand Up @@ -1185,17 +1205,22 @@ def encode_with_separators(msg: ParsedMessage, target: tuple[str, str, str, str,
* any other escape sequence (``H``, ``N``, ``Xhh``, ``.br`` and the rest) keeps its text under the
target escape character.

Two values the target set cannot represent raise :class:`DelimiterRewriteRefused`: an escape
sequence other than the five separator escapes whose text holds a target delimiter, and a segment
id holding the target field separator. A field holding no source escape character takes a single
``str.translate`` that also escapes its data characters, and every field takes the plain one when
the target set equals the source set.
At least these values the target set cannot represent raise :class:`DelimiterRewriteRefused`:
an escape sequence other than the five separator escapes whose text holds a target delimiter, a
segment id holding the target field separator, and, with a target set unlike the source set, a
``BHS`` or ``FHS`` line whose field 1 or 2 holds text :func:`_header_encoding` cannot carry. A
field holding no source escape character takes a single ``str.translate`` that also escapes its
data characters, and every field takes the plain one when the target set equals the source set.

Where two source separators are the same character, the one split first wins: repetition, then
component, then subcomponent, the order python-hl7 split them. The segment id is kept verbatim. On
an ``FHS`` or ``BHS`` header the raw MSH-1/MSH-2 leaves change only their escape character. Both
match the python-hl7 tree walk this replaced (``transports/mllp.py`` ``reencode_delimiters``). Only
an ``MSH`` header has its MSH-2 rewritten. Blank segment lines are kept.
component, then subcomponent, the order python-hl7 split them, as the python-hl7 tree walk this
replaced did (``transports/mllp.py`` ``reencode_delimiters``). The segment id is kept verbatim.
Every ``MSH`` line has its MSH-2 written as the four target encoding characters, whatever it
held. When the target set differs, a ``BHS`` or ``FHS`` header has its field 2 rewritten the
same way, so a batch or file header advertises the delimiters its message is written in
(BACKLOG #5802); :func:`_header_encoding` says which field 2 it refuses. When the target set
equals the source set, a ``BHS`` or ``FHS`` header's fields 1 and 2 are written as they came.
Blank segment lines are kept.

Raises ``ValueError`` when the message's own field, component, repetition and subcomponent
characters are not four distinct characters. Such a header cannot say which structure a character
Expand All @@ -1215,9 +1240,10 @@ def encode_with_separators(msg: ParsedMessage, target: tuple[str, str, str, str,
table.setdefault(ord(s_esc), t_esc)
# The same set on both sides changes no leaf, so one translate is exact. Any difference can make
# a literal character a target delimiter, or give a separator escape's letter a new meaning.
same_set = tuple(msg["seps"]) == tuple(target)
rewrite: Callable[[str], str] = (
operator.methodcaller("translate", table)
if tuple(msg["seps"]) == tuple(target)
if same_set
else _LeafRewrite(table, structure, msg["seps"], target)
)
lines: list[str] = []
Expand All @@ -1228,10 +1254,17 @@ def encode_with_separators(msg: ParsedMessage, target: tuple[str, str, str, str,
lines.append(seg["id"])
continue
if seg["id"] == "MSH" and len(texts) >= 3:
# MSH keeps the handling it had before BACKLOG #5802: MSH-2 always becomes the four
# target characters, with no refusal.
parts = ["MSH", t_comp + t_rep + t_esc + t_sub]
tail = texts[3:]
elif seg["id"] in ("BHS", "FHS") and len(texts) >= 3:
parts = [texts[0], texts[1].replace(s_esc, t_esc), texts[2].replace(s_esc, t_esc)]
if same_set:
# Nothing to translate in fields 1 and 2, so the header keeps its own bytes.
lines.append(_join_header([*texts[:3], *map(rewrite, texts[3:])], t_field))
continue
# Field 1 is the field separator the join writes, so only field 2 is a part.
parts = [seg["id"], _header_encoding(texts[1], texts[2], msg["seps"], target)]
tail = texts[3:]
else:
if t_field in texts[0]:
Expand All @@ -1245,6 +1278,42 @@ def encode_with_separators(msg: ParsedMessage, target: tuple[str, str, str, str,
return "\r".join(lines) + "\r"


def _header_encoding(
field_1: str,
field_2: str,
source: tuple[str, str, str, str, str],
target: tuple[str, str, str, str, str],
) -> str:
"""Field 2 of a ``BHS`` or ``FHS`` line rewritten for :func:`encode_with_separators`.

A field 2 that is the source encoding characters, a leading part of them, or empty becomes the
four target characters. One character after the four is kept as the v2.7 truncation character
when it is printable, not a space, and not a source or target delimiter. Any other field 2, or
a field 1 that is not the source field separator, raises :class:`DelimiterRewriteRefused`
(BACKLOG #5802): rewriting it would lose text, or write a character the receiver would read as
structure."""
s_field, s_comp, s_rep, s_sub, s_esc = source
t_field, t_comp, t_rep, t_sub, t_esc = target
s_enc = s_comp + s_rep + s_esc + s_sub
t_enc = t_comp + t_rep + t_esc + t_sub
if field_1 == s_field:
if s_enc.startswith(field_2):
return t_enc
truncation = field_2[len(s_enc) :]
if (
field_2.startswith(s_enc)
and len(truncation) == 1
and truncation.isprintable()
and not truncation.isspace()
and truncation not in s_field + s_enc + t_field + t_enc
):
return t_enc + truncation
raise DelimiterRewriteRefused(
"a batch or file header's field separator or encoding characters are not ones the "
"rewrite can carry, so the rewrite would lose text"
)


#: How many distinct escape sequences one :class:`_LeafRewrite` remembers. A real message repeats a
#: handful; the cap keeps a field of many distinct ones from building a table as large as itself.
_SEQUENCE_CACHE_MAX = 64
Expand Down
49 changes: 44 additions & 5 deletions messagefoundry/parsing/groups.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,8 @@ def ordinal(self) -> int:

Positional, so a fieldless bare ``OBR`` line counts here though :meth:`Message.field` and
:meth:`Message.count_segments` skip it: after one, ``ordinal`` is not the occurrence
``Message.field`` reads. Read the boundary through :meth:`field` instead (BACKLOG #5917)."""
``Message.field`` reads. Read and write the boundary through :meth:`field` and :meth:`set`
instead (BACKLOG #5917)."""
return self._ordinal

def segment_ids(self) -> list[str]:
Expand Down Expand Up @@ -139,16 +140,38 @@ def field(self, path: str, *, occurrence: int = 1, repetition: int | None = None
separator/escaping/repetition behavior is identical. Returns None if absent, which includes
every boundary field of a group opened by a fieldless bare boundary line, and a fieldless
member line is no occurrence (BACKLOG #5917)."""
if occurrence < 1:
raise ValueError("occurrence is 1-based (>= 1)")
seg = self._segment_of_path(path)
global_occurrence = self._to_global_occurrence(seg, occurrence)
_seg, global_occurrence = self._resolve(path, occurrence)
if global_occurrence is None:
return None
return self._m.field(path, occurrence=global_occurrence, repetition=repetition)

# --- mutate --------------------------------------------------------------

def set(
self, path: str, value: str, *, occurrence: int = 1, repetition: int | None = None
) -> None:
"""Write ``value`` at ``path`` within this group, with ``occurrence`` scoped to the group as
:meth:`field` scopes it.

Use this, not ``Message.set(..., occurrence=group.ordinal)``: after a fieldless bare
boundary line the two numberings differ, and that write would land on another order
(BACKLOG #5917). Raises ``KeyError`` first when the group holds fewer than ``occurrence``
addressable segments of that id, before any check on ``value``; a group opened by a
fieldless boundary line has no boundary to write. Otherwise it is :meth:`Message.set` at
the translated occurrence, with its escaping and refusals."""
self._m.set(
path, value, occurrence=self._write_occurrence(path, occurrence), repetition=repetition
)

def set_data(
self, path: str, value: str, *, occurrence: int = 1, repetition: int | None = None
) -> None:
""":meth:`Message.set_data` within this group, with ``occurrence`` scoped to the group as
:meth:`set` scopes it, and the same ``KeyError`` first."""
self._m.set_data(
path, value, occurrence=self._write_occurrence(path, occurrence), repetition=repetition
)

def append_segment(self, line: str) -> None:
"""Add a raw segment ``line`` (``"OBX|3|NM|K^Potassium||4.1"``) at the **end of this group**.

Expand Down Expand Up @@ -201,6 +224,22 @@ def rebuild(self, lines: list[str]) -> None:

# --- internals -----------------------------------------------------------

def _resolve(self, path: str, occurrence: int) -> tuple[str, int | None]:
"""``path``'s segment id and the message-wide occurrence of its ``occurrence``-th segment
in this group (None if the group holds fewer), the translation :meth:`field` and
:meth:`set` share."""
if occurrence < 1:
raise ValueError("occurrence is 1-based (>= 1)")
seg = self._segment_of_path(path)
return seg, self._to_global_occurrence(seg, occurrence)

def _write_occurrence(self, path: str, occurrence: int) -> int:
"""The message-wide occurrence a group-scoped write lands on, or ``KeyError``."""
seg, global_occurrence = self._resolve(path, occurrence)
if global_occurrence is None:
raise KeyError(f"group {self._ordinal} holds no {seg} occurrence {occurrence}")
return global_occurrence

def _segment_of_path(self, path: str) -> str:
"""The segment id of an HL7 field ``path`` (e.g. ``"OBX-5"`` -> ``"OBX"``), validated."""
from messagefoundry.parsing.peek import parse_path
Expand Down
Loading
Loading