Skip to content

refactor(http): move test_start_line and its patterns into the http base module (#1519) #1525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 1 commit into from
Oct 9, 2026
Merged

refactor(http): move test_start_line and its patterns into the http base module (#1519) #1525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions docs/source/pcapkit/protocols/application/http.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ HTTP - Hypertext Transfer Protocol
.. module:: pcapkit.protocols.application.http

:mod:`pcapkit.protocols.application.http` contains
:class:`~pcapkit.protocols.application.http.HTTP`
only, which is a base class for Hypertext Transfer
:class:`~pcapkit.protocols.application.http.HTTP`,
which is a base class for Hypertext Transfer
Protocol (HTTP) [*]_ family, eg.
:class:`HTTP/1.* <pcapkit.protocols.application.httpv1.HTTP>`
and :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`.
and :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`,
and :func:`~pcapkit.protocols.application.http.test_start_line`,
which tells whether a payload opens with an HTTP/1.* start line.

.. autoclass:: pcapkit.protocols.application.http.HTTP
:no-members:
Expand All @@ -30,6 +32,8 @@ and :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`.

.. autoattribute:: _preface_length

.. autofunction:: pcapkit.protocols.application.http.test_start_line

.. autodata:: pcapkit.protocols.application.http._HTTP2_PREFACE

.. rubric:: Footnotes
Expand Down
2 changes: 0 additions & 2 deletions docs/source/pcapkit/protocols/application/httpv1.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,6 @@ as below:
.. automethod:: _read_http_header
.. automethod:: _read_http_body

.. autofunction:: pcapkit.protocols.application.httpv1._test_start_line

Auxiliary Data
--------------

Expand Down
114 changes: 104 additions & 10 deletions pcapkit/protocols/application/http.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,19 @@
.. module:: pcapkit.protocols.application.http

:mod:`pcapkit.protocols.application.http` contains
:class:`~pcapkit.protocols.application.http.HTTP`
only, which is a base class for Hypertext Transfer
:class:`~pcapkit.protocols.application.http.HTTP`,
which is a base class for Hypertext Transfer
Protocol (HTTP) [*]_ family, eg.
:class:`HTTP/1.* <pcapkit.protocols.application.httpv1.HTTP>`
and :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`.
and :class:`HTTP/2 <pcapkit.protocols.application.httpv2.HTTP>`,
and :func:`~pcapkit.protocols.application.http.test_start_line`,
which tells whether a payload opens with an HTTP/1.* start line.

.. [*] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

"""
import contextlib
import re
import struct
from typing import TYPE_CHECKING, Generic, cast

Expand All @@ -30,7 +33,7 @@

from typing_extensions import Literal, Self

__all__ = ['HTTP']
__all__ = ['HTTP', 'test_start_line']

#: The HTTP/2 connection preface (:rfc:`9113#section-3.4`). A client opens every
#: HTTP/2 connection -- with prior knowledge, over TLS, or after an upgrade --
Expand Down Expand Up @@ -102,6 +105,99 @@ def _test_http2_frame(data: 'bytes') -> 'bool':
return any(data[5:9]) is stream


# Regular expression to match HTTP methods. Anchored at both ends: :func:`re.match`
# anchors only at the start, so an unanchored pattern prefix-matches and accepts the
# leading ``G`` of ``Get`` as a whole method token. Method tokens are case-sensitive
# per :rfc:`9110#section-9.1`, so ``Get`` is not ``GET`` and must not parse as one.
_RE_METHOD = re.compile(rb"(?P<method>[A-Z][A-Z-]*)\Z") # RFC 9110, section 16.1.1, 9.1, 5.6.2
# Regular expression to match HTTP version string.
_RE_VERSION = re.compile(rb"HTTP/(?P<version>\d\.\d)")
# Regular expression to match HTTP status code. Anchored for the same reason as
# ``_RE_METHOD``, and it matters more here: this pattern is only a guard, and
# ``httpv1.HTTP`` takes the value from ``int(para2)`` on the raw token, so an
# unanchored prefix match would let ``200x`` and ``2000`` past the guard and then
# out of ``int()`` as a bare ``ValueError`` -- where ``_read_http_header``
# documents ``ProtocolError``.
# :rfc:`9112#section-4` gives ``status-code = 3DIGIT``, exactly three -- the
# grammar is in HTTP/1.1 because ``status-code`` is part of its ``status-line``
# production; :rfc:`9110#section-15` covers the code semantics and registry, not
# the syntax.
_RE_STATUS = re.compile(rb'\d{3}\Z')

# The HTTP/2 connection preface's header block, as ``httpv1.HTTP.read`` splits it
# off at the first CRLFCRLF: ``PRI * HTTP/2.0``. A start line with zero field
# lines, so the patterns above accept it; it is refused by name in
# ``httpv1.HTTP._read_http_header`` and :func:`test_start_line` so that the
# preface never parses as HTTP/1.
_HTTP2_PREFACE_HEADER = _HTTP2_PREFACE.split(b'\r\n\r\n', maxsplit=1)[0]


def test_start_line(data: 'bytes') -> 'bool':
"""Whether ``data`` opens with an HTTP/1.* start line.

This is a *classification* predicate and parses nothing: it answers "is this
HTTP/1?" for :meth:`HTTP._guess_version
<pcapkit.protocols.application.http.HTTP._guess_version>`. Classifying by
trial-parsing every version in the family would decide a payload that is
not HTTP at all by which parser happened to fail less loudly.

Args:
data: Payload to classify.

Returns:
Whether the payload's first line is a ``request-line`` or a
``status-line`` (:rfc:`9112#section-2.1`).

Note:
The acceptance rule is deliberately the *same* one
:meth:`HTTP._read_http_header
<pcapkit.protocols.application.httpv1.HTTP._read_http_header>` applies
-- ``_RE_METHOD`` with ``_RE_VERSION`` for a request, ``_RE_VERSION``
with ``_RE_STATUS`` for a response -- which is why those three patterns
are defined beside this predicate and
:mod:`~pcapkit.protocols.application.httpv1` imports them from here. The
two must accept the same start lines: a predicate looser than the parser
classifies payloads the parser then refuses, and one tighter than the
parser hands real HTTP/1 to a later arm.
``test_start_line_predicate_agrees_with_the_httpv1_parser`` pins that
agreement.

The checks the parser performs *before* those patterns are mirrored too,
and this is not pedantry -- they are the whole reason the HTTP/2
connection preface is not claimed here. ``PRI *
HTTP/2.0\\r\\n\\r\\nSM\\r\\n\\r\\n`` is deliberately a well-formed
HTTP/1.1 *request line* (:rfc:`9113#section-3.4`), so a predicate that
tested only the first line would answer :data:`True` for it. Split at the
header/body separator first, as :meth:`HTTP.read
<pcapkit.protocols.application.httpv1.HTTP.read>` does, and the preface's
header is exactly ``PRI * HTTP/2.0`` (``_HTTP2_PREFACE_HEADER``), which
the parser refuses by name and so does this. Any *other* header with no
CRLF in it is a start line with zero field lines, which
:rfc:`9112#section-2.1` allows and both accept (:issue:`1041`).

An HTTP/0.9 request line carries only two tokens and so is not recognised
here either, matching the parser, which raises on fewer than three.

"""
header = data.split(b'\r\n\r\n', maxsplit=1)[0]
if header == data: # no header/body separator -- ``read`` raises
return False

if header == _HTTP2_PREFACE_HEADER: # the preface -- ``_read_http_header`` raises
return False

startline = header.split(b'\r\n', maxsplit=1)[0]
try:
para1, para2, para3 = re.split(rb'\s+', startline, maxsplit=2)
except ValueError:
return False

return bool(
(re.match(_RE_METHOD, para1) and re.match(_RE_VERSION, para3)) # request-line
or (re.match(_RE_VERSION, para1) and re.match(_RE_STATUS, para2)) # status-line
)


class HTTP(Application[_PT, _ST], Generic[_PT, _ST]):
"""This class implements all protocols in HTTP family.

Expand Down Expand Up @@ -473,14 +569,12 @@ def _guess_version(self, length: 'int', **kwargs: 'Any') -> 'HTTP':
return http

from pcapkit.protocols.application.httpv1 import HTTP as HTTPv1 # isort: skip # pylint: disable=line-too-long,import-outside-toplevel
from pcapkit.protocols.application.httpv1 import \
_test_start_line # isort: skip # pylint: disable=import-outside-toplevel

# NOTE: The second identification: an HTTP/1 ``request-line`` or
# ``status-line``, tested with the very patterns ``httpv1.HTTP`` parses
# with (``httpv1.py``'s ``_RE_METHOD``, ``_RE_VERSION`` and
# ``_RE_STATUS``, all anchored), so the predicate and the parser cannot
# disagree about what HTTP/1 looks like.
# with (``_RE_METHOD``, ``_RE_VERSION`` and ``_RE_STATUS`` above, all
# anchored, which ``httpv1.py`` imports from this module), so the
# predicate and the parser cannot disagree about what HTTP/1 looks like.
#
# Identified means *committed*: a malformed HTTP/1 message is reported as
# such instead of being handed to the HTTP/2 arm, which accepts any
Expand All @@ -494,7 +588,7 @@ def _guess_version(self, length: 'int', **kwargs: 'Any') -> 'HTTP':
# pins that a :exc:`struct.error` -- or pcapkit's own ``StructError``,
# whose ``eof`` flag ``NoPayload`` handling reads -- reaches the caller
# from this route rather than being converted or swallowed.
if _test_start_line(self._data[:length]):
if test_start_line(self._data[:length]):
return HTTPv1(self._data, length, **kwargs)

# NOTE: Neither identification matched, so this is the fall-through: a
Expand Down
99 changes: 8 additions & 91 deletions pcapkit/protocols/application/httpv1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@
from pcapkit.const.http.status_code import StatusCode as Enum_StatusCode
from pcapkit.corekit.enum import EnumLookup
from pcapkit.corekit.multidict import OrderedMultiDict
from pcapkit.protocols.application.http import _HTTP2_PREFACE
from pcapkit.protocols.application.http import (_HTTP2_PREFACE_HEADER, _RE_METHOD, _RE_STATUS,
_RE_VERSION)
from pcapkit.protocols.application.http import HTTP as HTTPBase
from pcapkit.protocols.data.application.httpv1 import HTTP as Data_HTTP
from pcapkit.protocols.data.application.httpv1 import RequestHeader as Data_RequestHeader
Expand All @@ -56,95 +57,10 @@

__all__ = ['HTTP']

# Regular expression to match HTTP methods. Anchored at both ends: :func:`re.match`
# anchors only at the start, so an unanchored pattern prefix-matches and accepts the
# leading ``G`` of ``Get`` as a whole method token. Method tokens are case-sensitive
# per :rfc:`9110#section-9.1`, so ``Get`` is not ``GET`` and must not parse as one.
_RE_METHOD = re.compile(rb"(?P<method>[A-Z][A-Z-]*)\Z") # RFC 9110, section 16.1.1, 9.1, 5.6.2
# Regular expression to match HTTP version string.
_RE_VERSION = re.compile(rb"HTTP/(?P<version>\d\.\d)")
# Regular expression to match HTTP status code. Anchored for the same reason as
# ``_RE_METHOD``, and it matters more here: this pattern is only a guard, and the
# value is taken from ``int(para2)`` on the raw token, so an unanchored prefix
# match would let ``200x`` and ``2000`` past the guard and then out of ``int()``
# as a bare ``ValueError`` -- where ``_read_http_header`` documents
# ``ProtocolError``.
# :rfc:`9112#section-4` gives ``status-code = 3DIGIT``, exactly three -- the
# grammar is in HTTP/1.1 because ``status-code`` is part of its ``status-line``
# production; :rfc:`9110#section-15` covers the code semantics and registry, not
# the syntax.
_RE_STATUS = re.compile(rb'\d{3}\Z')

# The HTTP/2 connection preface's header block, as ``HTTP.read`` splits it off at
# the first CRLFCRLF: ``PRI * HTTP/2.0``. A start line with zero field lines, so
# the patterns above accept it; it is refused by name in ``_read_http_header``
# and ``_test_start_line`` so that the preface never parses as HTTP/1.
_HTTP2_PREFACE_HEADER = _HTTP2_PREFACE.split(b'\r\n\r\n', maxsplit=1)[0]


def _test_start_line(data: 'bytes') -> 'bool':
"""Whether ``data`` opens with an HTTP/1.* start line.

This is a *classification* predicate and parses nothing: it answers "is this
HTTP/1?" for :meth:`HTTP._guess_version
<pcapkit.protocols.application.http.HTTP._guess_version>`. Classifying by
trial-parsing every version in the family would decide a payload that is
not HTTP at all by which parser happened to fail less loudly.

Args:
data: Payload to classify.

Returns:
Whether the payload's first line is a ``request-line`` or a
``status-line`` (:rfc:`9112#section-2.1`).

Note:
The acceptance rule is deliberately the *same* one
:meth:`HTTP._read_http_header
<pcapkit.protocols.application.httpv1.HTTP._read_http_header>` applies
further down this module -- ``_RE_METHOD`` with ``_RE_VERSION`` for a
request, ``_RE_VERSION`` with ``_RE_STATUS`` for a response -- which is
why this lives beside those three patterns rather than in the dispatcher
that calls it. The two must accept the same start lines: a predicate
looser than the parser classifies payloads the parser then refuses, and
one tighter than the parser hands real HTTP/1 to a later arm.
``test_start_line_predicate_agrees_with_the_httpv1_parser`` pins that
agreement.

The checks the parser performs *before* those patterns are mirrored too,
and this is not pedantry -- they are the whole reason the HTTP/2
connection preface is not claimed here. ``PRI *
HTTP/2.0\\r\\n\\r\\nSM\\r\\n\\r\\n`` is deliberately a well-formed
HTTP/1.1 *request line* (:rfc:`9113#section-3.4`), so a predicate that
tested only the first line would answer :data:`True` for it. Split at the
header/body separator first, as :meth:`HTTP.read
<pcapkit.protocols.application.httpv1.HTTP.read>` does, and the preface's
header is exactly ``PRI * HTTP/2.0`` (``_HTTP2_PREFACE_HEADER``), which
the parser refuses by name and so does this. Any *other* header with no
CRLF in it is a start line with zero field lines, which
:rfc:`9112#section-2.1` allows and both accept (:issue:`1041`).

An HTTP/0.9 request line carries only two tokens and so is not recognised
here either, matching the parser, which raises on fewer than three.

"""
header = data.split(b'\r\n\r\n', maxsplit=1)[0]
if header == data: # no header/body separator -- ``read`` raises
return False

if header == _HTTP2_PREFACE_HEADER: # the preface -- ``_read_http_header`` raises
return False

startline = header.split(b'\r\n', maxsplit=1)[0]
try:
para1, para2, para3 = re.split(rb'\s+', startline, maxsplit=2)
except ValueError:
return False

return bool(
(re.match(_RE_METHOD, para1) and re.match(_RE_VERSION, para3)) # request-line
or (re.match(_RE_VERSION, para1) and re.match(_RE_STATUS, para2)) # status-line
)
# NOTE: The start-line patterns ``_RE_METHOD``, ``_RE_VERSION`` and ``_RE_STATUS``,
# and ``_HTTP2_PREFACE_HEADER``, are imported from ``http.py``, where
# :func:`~pcapkit.protocols.application.http.test_start_line` classifies with
# them; one copy keeps that predicate and this parser in agreement.


class Type(EnumLookup, StrEnum):
Expand Down Expand Up @@ -536,7 +452,8 @@ def _read_http_header(self, header: 'bytes') -> 'tuple[Data_Header, OrderedMulti
# one such header refused by name is the HTTP/2 connection preface's:
# ``PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n`` splits in ``read`` to exactly
# ``PRI * HTTP/2.0``, which ``_RE_METHOD`` and ``_RE_VERSION`` would
# otherwise accept as an HTTP/1 request (see ``_HTTP2_PREFACE_HEADER``).
# otherwise accept as an HTTP/1 request (see ``_HTTP2_PREFACE_HEADER`` in
# ``http.py``).
if header == _HTTP2_PREFACE_HEADER:
raise ProtocolError('HTTP: invalid format')
startline, _, headerfield = header.partition(b'\r\n')
Expand Down
17 changes: 9 additions & 8 deletions tests/const/test_const_method_value_lookup_908_unit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,11 @@
the constructor (:meth:`~pcapkit.const.http.method.Method._missing_`, which
this override never touched).

**Why a capture reaches it.** ``pcapkit/protocols/application/httpv1.py``'s
``_RE_METHOD = re.compile(rb"(?P<method>[A-Z][A-Z-]*)\\Z")`` admits a hyphen,
so a request line carrying ``BASELINE-CONTROL`` matches, reaches
``Enum_Method.get(...)`` at ``httpv1.py:434``, parses successfully, and is
**Why a capture reaches it.** ``pcapkit/protocols/application/http.py``'s
``_RE_METHOD = re.compile(rb"(?P<method>[A-Z][A-Z-]*)\\Z")``, which
``httpv1.py`` imports, admits a hyphen, so a request line carrying
``BASELINE-CONTROL`` matches, reaches ``Enum_Method.get(...)`` in
``httpv1.HTTP._read_start_line``, parses successfully, and is
reported with the wrong ``idempotent`` -- the harder kind of defect to
notice, since the parse itself never fails.
:class:`HTTPv1EndToEndTests` below pins that whole path, not just the direct
Expand Down Expand Up @@ -191,7 +192,7 @@ class HTTPv1EndToEndTests(unittest.TestCase):
def test_a_baseline_control_request_line_reports_the_registrys_idempotent_flag(
self) -> None:
"""``_RE_METHOD`` admits the hyphen, so this request line reaches
``Enum_Method.get(...)`` at ``httpv1.py:434`` and used to come back
``Enum_Method.get(...)`` in ``httpv1.HTTP._read_start_line`` and used to come back
with ``idempotent=False`` -- the wrong answer for a parse that
otherwise succeeds silently. Pins the whole path, not only the
direct ``Method.get`` call above."""
Expand Down Expand Up @@ -269,9 +270,9 @@ def test_a_bytes_key_raises_instead_of_returning_a_bytes_valued_member(self) ->

Before #908 this handed back a member whose ``name`` and ``value``
were both ``b'GET'``. ``bytes`` is the plausible mistake here, since
:attr:`~pcapkit.protocols.application.httpv1.HTTP._RE_METHOD` is a
bytes pattern and ``httpv1`` is bytes throughout -- the live call at
``httpv1.py:434`` is safe only because it wraps the match in
``_RE_METHOD`` (in ``pcapkit/protocols/application/http.py``) is a
bytes pattern and ``httpv1`` is bytes throughout -- the live call in
``httpv1.HTTP._read_start_line`` is safe only because it wraps the match in
``self.decode(...)``.
"""
from pcapkit.const.http.method import Method
Expand Down
Loading
Loading