diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7572c4a5dc..e40002e98d 100755 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -167,8 +167,8 @@ rewrites the regenerated constants, not as a check, so nothing verifies import o request. One trap: `make vermin` redirects its report into `temp/vermin.txt` rather than to your terminal, -and vermin exits 1 because `vermin.ini` sets `targets = 3.6` against a real floor of 3.11. Make -gives up at the redirect with `make: *** [vermin] Error 1`, the report left in the file and the line +and vermin exits 1 because `vermin.ini` sets `targets = 3.6` against the 3.11 minimum it detects. +Make gives up at the redirect with `make: *** [vermin] Error 1`, the report left in the file and the line that would have opened it never reached. `make vermin-ci` runs the same flags to stdout, which is why CI uses it and why it is easier to read at a desk. Running the lot before you push still saves a review round. @@ -199,9 +199,9 @@ BLANK LINE footer (optional) ``` -`type` is one of `feat`, `fix`, `docs`, `test`, `perf`, `refactor`, `ci` or `chore` — those are the -ones in use. `scope` names the part of the package affected — `tcp`, `corekit`, `schema`, `ipv4`, -`vendor` — and several are separated by commas inside the parentheses, as in `fix(link,internet):` +`type` is one of `feat`, `fix`, `docs`, `test`, `perf`, `refactor`, `ci`, `chore` or `release` — +those are the ones in use. `scope` names the part of the package affected — `tcp`, `corekit`, +`schema`, `ipv4`, `vendor` — and several are separated by commas inside the parentheses, as in `fix(link,internet):` or `test(utilities,foundation):`. The scope may be omitted where nothing narrower than the whole project applies, as in `docs:`. diff --git a/docs/source/contributing/conventions/documentation.rst b/docs/source/contributing/conventions/documentation.rst index 58a5204afd..e78fbbea0a 100644 --- a/docs/source/contributing/conventions/documentation.rst +++ b/docs/source/contributing/conventions/documentation.rst @@ -230,10 +230,8 @@ check the members one at a time.** Several of :issue:`719`'s findings were of ex shape: * A sweep asserted that every ``.. module::`` target in the documentation resolved. - One did not: :file:`docs/source/pcapkit/protocols/link/rarp.rst` (its path at the - time; now under :file:`application/`) declared - ``pcapkit.protocols.data.link.rarp``, which has never existed, because RARP and - DRARP reuse ARP's data class. Fixed in ``68fbccd90``. + One did not: the RARP page declared ``pcapkit.protocols.data.link.rarp``, which has + never existed, because RARP and DRARP reuse ARP's data class. Fixed in ``68fbccd90``. * :issue:`911`'s ruling -- export the sentinel objects and leave their types out -- was read as describing all three modules that then held a sentinel. One ran the other way: :mod:`pcapkit.corekit.fields.field` exported neither, so applying the rule diff --git a/docs/source/contributing/conventions/process.rst b/docs/source/contributing/conventions/process.rst index d5285e66b0..9d98bca366 100644 --- a/docs/source/contributing/conventions/process.rst +++ b/docs/source/contributing/conventions/process.rst @@ -3,7 +3,7 @@ Running the Repository ---------------------- -The four pages before this one are about writing library code. The rulings here are +The five pages before this one are about writing library code. The rulings here are about running the repository -- what an install carries, what a changelog entry is, and what the issue and pull request labels mean. None of them is derivable from a module, and none fits a code-convention page, so the owner ruled on diff --git a/docs/source/contributing/conventions/protocol-layer-placement.rst b/docs/source/contributing/conventions/protocol-layer-placement.rst index 2f27cd7d99..5963fd27af 100644 --- a/docs/source/contributing/conventions/protocol-layer-placement.rst +++ b/docs/source/contributing/conventions/protocol-layer-placement.rst @@ -194,8 +194,8 @@ they share (``__data__``, ``__layer__``, ``__schema__``, ``layer``) plus ``register`` and ``_read_protos`` -- but all three also exist on ``ProtocolBase``, so the strict difference ``Link`` minus ``Application`` minus ``ProtocolBase`` is **empty** and every one of them still resolves on ``OSPF``. What changes is which registry it -resolves *to*: ``OSPF.__proto__`` is now ``ProtocolBase.__proto__`` rather than -``Link.__proto__``, so OSPF no longer sees Link's EtherType entries. That is inert -- +resolves *to*: ``OSPF.__proto__`` is ``ProtocolBase.__proto__`` rather than +``Link.__proto__``, so OSPF does not see Link's EtherType entries. That is inert -- nothing reaches OSPF by EtherType, and a ``-1`` lookup misses in both registries alike and resolves to :class:`~pcapkit.protocols.misc.raw.Raw`, inserting nothing on the miss. ``RARP`` keeps ``Link``'s through ``ARP``. diff --git a/docs/source/contributing/conventions/registry-protocol.rst b/docs/source/contributing/conventions/registry-protocol.rst index ce8a290480..ba56f4735b 100644 --- a/docs/source/contributing/conventions/registry-protocol.rst +++ b/docs/source/contributing/conventions/registry-protocol.rst @@ -570,10 +570,9 @@ rather than changing it. ``Command.get`` upper-cases its key before matching, which makes it look as though the base were case-insensitive -- deliberately, since :rfc:`959#section-5` treats FTP command codes identically - regardless of case. ``Method.get`` used to fold case the same way, but - :issue:`896` made it - case-sensitive instead: :rfc:`9110#section-9.1` says the HTTP method token is - case-sensitive, so ``Method.get('get')`` no longer resolves to + regardless of case. ``Method.get`` does not fold case, as ruled on :issue:`896`: + :rfc:`9110#section-9.1` says the HTTP method token is case-sensitive, so + ``Method.get('get')`` does not resolve to ``Method.GET`` -- it builds its own unregistered member, preserving the caller's exact casing, the same way an unrecognised value always does. diff --git a/docs/source/contributing/conventions/sentinel-convention.rst b/docs/source/contributing/conventions/sentinel-convention.rst index ec6f0a0211..534704543d 100644 --- a/docs/source/contributing/conventions/sentinel-convention.rst +++ b/docs/source/contributing/conventions/sentinel-convention.rst @@ -59,7 +59,7 @@ What reaches users is the **object only**. The owner ruled on GitHub issue :issu that the objects alone -- ``NULL`` and its siblings -- are exported to users, so a public sentinel names its instance in its module's ``__all__`` and leaves the type out of it. The type stays importable by its dotted path, for an annotation or an ``is`` guard; it -is ``import *`` that no longer offers it. A private sentinel such as ``ABSENT`` is in +is only ``import *`` that does not offer it. A private sentinel such as ``ABSENT`` is in neither, which is what private means here -- dropping its leading underscore did not add it to either list, and :class:`~pcapkit.corekit.sentinels.AbsentType` and :data:`~pcapkit.corekit.sentinels.ABSENT` are documented on diff --git a/docs/source/contributing/pep.rst b/docs/source/contributing/pep.rst index 190949a292..af2f9bf81a 100644 --- a/docs/source/contributing/pep.rst +++ b/docs/source/contributing/pep.rst @@ -62,7 +62,8 @@ whose *layout* is identical -- it holds the whole of the tag, and ``NotImplemented`` folder. **Those files are not a roadmap.** They are scratch reminders, so a stub's presence does not mean a protocol is planned or claimed, and its absence does not mean the protocol is unwanted: - ``LINUX_SLL``, ``QUIC`` and **DTLS** have no stub and are on the list above, + ``LINUX_SLL``, ``LINUX_SLL2``, ``QUIC`` and **DTLS** have no stub and are on the + list above, while **NGAP** has none either and is implemented. Take this page as the record and ignore the folder. @@ -210,8 +211,8 @@ look like ordinary fields and are not: DTLS ~~~~ -**Not started**, and unlike everything in the list above it has no stub in the -tree at all -- only TLS/SSL does. It earns its own entry because the registries +**Not started**, and it has no stub in the tree -- of the TLS/SSL and DTLS pair, +only TLS/SSL does. It earns its own entry because the registries have moved ahead of it, and a growing part of SCTP's surface now names a protocol that does not exist: @@ -820,9 +821,9 @@ from *wrong*, or it will cry wolf on the commonest capture there is — model for how to spell "expected, not an error". Sequenced for **wave 2 or 3**. The cryptographic half could be done sooner since -ESP has already laid the groundwork, but the checksum half genuinely wants the -parent-access question answered first, and that is worth doing deliberately -rather than as a side effect of a checksum patch. +ESP has already laid the groundwork, but the checksum half needs the IP +pseudo-header class defined first, and that is worth doing deliberately rather +than as a side effect of a checksum patch. Release Plan — 1.5.0 in Two Steps --------------------------------- @@ -969,7 +970,7 @@ dispatcher registers exactly three link types work above rather than a separate concern — all four are link types the library enumerates but cannot parse. It surfaced while deciding what an unresolvable link-layer name should do: neither value is an honest stand-in for "unknown link -type", which is why the toolkit now raises instead of defaulting to +type", which is why the toolkit raises instead of defaulting to ``LinkType.NULL``. **Wave 3 — the remaining protocols** from the same list, taken three or four at a diff --git a/docs/source/contributing/releasing.rst b/docs/source/contributing/releasing.rst index b06cd24cf0..db783b658b 100644 --- a/docs/source/contributing/releasing.rst +++ b/docs/source/contributing/releasing.rst @@ -186,10 +186,10 @@ pushed, trading a clean skip for a checkout failure; the equality pair skips ``needs.version_check.result == 'success'`` with no ``|| 'skipped'``. It produces the evidence the gates read, so a skipped or cancelled ``version_check`` leaves them nothing to decide on. ``unit-tests`` is held to -the same bar for a different reason: it is the release test gate, and since -`#1052 `__ it skips when -nothing is publishable, so accepting ``skipped`` there would publish -untested. ``github`` names it in ``needs:`` instead and relies on the implicit +the same bar for a different reason: it is the release test gate, and it skips +when nothing is publishable +(`#1052 `__), so accepting +``skipped`` there would publish untested. ``github`` names it in ``needs:`` instead and relies on the implicit ``success()``. The gate's own ``if:`` is the union of the four publishers' conditions, with ``!= 'true'`` so unknown evidence runs it; should the two ever drift apart, the failure is a release that did not happen, which @@ -257,7 +257,7 @@ Precautions above is what prevents it: ``tag``, ``pypi`` and ``conda`` each check whether *their own* artefact is missing rather than whether the ``v*`` tag exists, so an incomplete release runs the jobs that did not finish instead of skipping them. - A half-finished release now self-heals on the next ``workflow_run``-triggered + A half-finished release self-heals on the next ``workflow_run``-triggered attempt, or on re-running the workflow by hand -- see `Recovery`_ below. ``release_status`` is the other half: it runs unconditionally and reports, with a ``::notice``, a ``::warning`` or a failing ``::error``, why a run @@ -275,7 +275,7 @@ here), so the push only fires again after the tag is deleted -- which touches a ref the release automation owns, and lands the retry back on the tag-push path, where every job's guard is bypassed regardless of what has already gone out. Neither is worth the risk of a double upload to an index that cannot take one -back, and neither is needed, since a plain re-run now self-heals; see +back, and neither is needed, since a plain re-run self-heals; see `Recovery`_ below. **``environment: pypi`` stays even though its reviewer is gone.** ``pypi`` @@ -305,8 +305,8 @@ actual fix rather than a best-effort suggestion: `Per-Job Evidence, Not a Shared Proxy`_ above means ``tag``, ``pypi`` and ``conda`` each check whether *their own* artefact is missing, so a re-run finishes whichever jobs did not complete last time instead of skipping them on the ``v*`` tag's mere -existence. ``pypi`` was always safe to re-run (``skip-existing: true``); -``conda`` now is too, because each matrix leg checks Anaconda for its own +existence. ``pypi`` is safe to re-run (``skip-existing: true``), and so is +``conda``, because each matrix leg checks Anaconda for its own platform/Python distribution before uploading and skips only that leg's upload if it is already there. diff --git a/docs/source/contributing/workflows.rst b/docs/source/contributing/workflows.rst index c55d41a3e2..4a9edc219b 100644 --- a/docs/source/contributing/workflows.rst +++ b/docs/source/contributing/workflows.rst @@ -260,8 +260,7 @@ The Skip Cascade (`#888 `__) The two relationships above compose into a failure mode worth seeing on its own graph. Create Release's first job, ``version_check`` -- ahead of the -``uses:`` call above since -`#1052 `__ -- carries: +``uses:`` call above -- carries: .. code-block:: yaml diff --git a/docs/source/ext.rst b/docs/source/ext.rst index 3e8dac47ed..1f8b617c93 100644 --- a/docs/source/ext.rst +++ b/docs/source/ext.rst @@ -648,9 +648,9 @@ The following code snippet shows how to create a new dumper class: # dumper registries, and use it to dump the extracted network packets. class MyDumper(Dumper, fmt='pcap', ext='.pcap'): - # NOTE: This property is to define the file format of the dumper. - # It is expected to be a string, which is used as the file extension - # of the output file. + # NOTE: This property names the file format of the dumper, as a string. + # It is not the file extension -- that is the ``ext`` given at class + # definition above. @property def kind(self) -> 'str': return 'pcap' @@ -856,8 +856,8 @@ Callback Functions ~~~~~~~~~~~~~~~~~~ Callback functions can be registered on the reassembly and flow tracing -classes, and run at the end of the respective process -- to inspect the -reassembled datagrams or flows, or to discard some of them. +classes, to inspect the reassembled datagrams or traced flows as they are +produced. .. seealso:: @@ -869,9 +869,13 @@ reassembled datagrams or flows, or to discard some of them. - :func:`~pcapkit.foundation.registry.foundation.register_reassembly_tcp_callback` - :func:`~pcapkit.foundation.registry.foundation.register_traceflow_tcp_callback` -A callback takes one argument -- the list of reassembled datagrams or flows -- -and returns :obj:`None`; any return value is ignored. +A reassembly callback runs on every submission and takes one argument, the list +of datagrams that submission produced. A flow tracing callback runs as each flow +is finalised and takes that flow's +:class:`~pcapkit.foundation.traceflow.data.tcp.Index`. Either returns +:obj:`None`; any return value is ignored. -That list can be modified in place, but prefer not to: if the reassembly or -tracing result itself is wrong for your purpose, subclass the reassembly or -flow tracing class and change its algorithm instead. +A reassembly callback can modify its list in place, which discards datagrams, +but prefer not to: if the reassembly or tracing result itself is wrong for your +purpose, subclass the reassembly or flow tracing class and change its algorithm +instead. diff --git a/docs/source/index.rst b/docs/source/index.rst index 10ed8ebe32..c7d7fe568b 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -271,7 +271,7 @@ Or install the latest version from the git repository: .. code-block:: shell git clone https://github.com/JarryShaw/PyPCAPKit.git - cd pypcapkit + cd PyPCAPKit pip install -e . # and to update at any time git pull diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst index adf7ebb23f..c0f83e4e81 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst @@ -169,6 +169,8 @@ Terminology | | |--> (int) packet range number | |--> 'header' : (bytes) header buffer | |--> 'datagram' : (bytearray) data buffer, holes set to b'\\x00' + | |--> 'timestamp' : (float) capture timestamp of the + | | first-arriving fragment | |--> 'conflict' : (list) octet ranges on which an arriving | | fragment disagreed with bytes already in | | 'datagram' diff --git a/docs/source/pcapkit/foundation/reassembly/tcp.rst b/docs/source/pcapkit/foundation/reassembly/tcp.rst index 4659a7d5a3..0ed79aab30 100644 --- a/docs/source/pcapkit/foundation/reassembly/tcp.rst +++ b/docs/source/pcapkit/foundation/reassembly/tcp.rst @@ -2,8 +2,10 @@ TCP Datagram Reassembly ======================= +.. module:: pcapkit.foundation.reassembly.tcp + :mod:`pcapkit.foundation.reassembly.tcp` contains -:class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` only, +:class:`~pcapkit.foundation.reassembly.tcp.TCP` only, which reconstructs fragmented TCP packets back to origin. .. autoclass:: pcapkit.foundation.reassembly.tcp.TCP @@ -159,27 +161,31 @@ Terminology .. code-block:: python + ip_info = ip.info + tcp_info = tcp.info + + raw_len = len(tcp.packet.payload) packet_dict = Info( - bufid = tuple( - ip.src, # source IP address - tcp.srcport, # source port - ip.dst, # destination IP address - tcp.dstport, # destination port + bufid = ( + ip_info.src, # source IP address + tcp_info.srcport.port, # source port + ip_info.dst, # destination IP address + tcp_info.dstport.port, # destination port ), - dsn = tcp.seq, # data sequence number - ack = tcp.ack, # acknowledgement number - num = frame.number, # original packet range number - syn = tcp.flags.syn, # synchronise flag - fin = tcp.flags.fin, # finish flag - rst = tcp.flags.rst, # reset connection flag - len = tcp.raw_len, # payload length, header excludes - first = tcp.seq, # first sequence number of payload - last = tcp.seq + tcp.raw_len - 1, - # last sequence number of payload - header = tcp.packet.header, # raw bytes type header - payload = tcp.raw, # raw bytearray type payload + num = frame.info.number, # original packet range number + ack = tcp_info.ack, # acknowledgement + dsn = tcp_info.seq, # data sequence number + syn = tcp_info.flags.syn, # synchronise flag + fin = tcp_info.flags.fin, # finish flag + rst = tcp_info.flags.rst, # reset connection flag + header = tcp.packet.header, # raw bytes type header + payload = bytearray( + tcp.packet.payload), # raw bytearray type payload + first = tcp_info.seq, # first sequence number of payload + last = tcp_info.seq + raw_len - 1, # last sequence number of payload + len = raw_len, # payload length, header excludes timestamp = float( - frame.time_epoch), # capture timestamp + frame.info.time_epoch), # capture timestamp ) Both ``first`` and ``last`` are absolute TCP sequence numbers and diff --git a/docs/source/pcapkit/index.rst b/docs/source/pcapkit/index.rst index 3070ee1f06..9b43bc2647 100644 --- a/docs/source/pcapkit/index.rst +++ b/docs/source/pcapkit/index.rst @@ -59,9 +59,12 @@ Command Line Tool .. module:: pcapkit.__main__ -.. important:: +.. note:: - This module requires ``emoji`` package to be installed. + The ``emoji`` package (the ``cli`` extra) decorates the ``-v`` output. + Without it, importing the module warns with + :class:`~pcapkit.utilities.warnings.EmojiWarning` and the CLI prints plain + text instead. :mod:`pcapkit.__main__` provides the CLI, merged in from the deprecated |jspcapy|_ project. @@ -72,23 +75,24 @@ Command Line Tool .. code-block:: text usage: pcapkit-cli [-h] [-V] [-o file-name] [-f format] [-j] [-p] [-t] [-a] - [-v] [-F] [-E PKG] [-P PROTOCOL] [-L LAYER] + [-v] [-F] [-E PKG] [-P PROTOCOL] [-L LAYER] [-B] + [-O file-name] input-file-name PCAP file extractor and formatted dumper positional arguments: input-file-name The name of input pcap file. If ".pcap" omits, it will - be automatically appended. + be automatically appended. Use "-" to indicate reading + from `stdin'. - optional arguments: + options: -h, --help show this help message and exit -V, --version show program's version number and exit - -o file-name, --output file-name + -o, --output file-name The name of input pcap file. If format extension omits, it will be automatically appended. - -f format, --format format - Print a extraction report in the specified output + -f, --format format Print a extraction report in the specified output format. Available are all formats supported by dictdumper, e.g.: json, plist, and tree. -j, --json Display extraction report as json. This will yield @@ -104,13 +108,17 @@ Command Line Tool -a, --auto-extension If output file extension omits, append automatically. -v, --verbose Show more information. -F, --files Split each frame into different files. - -E PKG, --engine PKG Indicate extraction engine. Note that except default + -E, --engine PKG Indicate extraction engine. Note that except default or pcapkit engine, all other engines need support of corresponding packages. - -P PROTOCOL, --protocol PROTOCOL + -P, --protocol PROTOCOL Indicate extraction stops after which protocol. - -L LAYER, --layer LAYER - Indicate extract frames until which layer. + -L, --layer LAYER Indicate extract frames until which layer. + -B, --buffer-save Indicate if store buffer to file when reading from + stdin. + -O, --buffer-path file-name + The name of buffer storage file. If `--buffer-save` is + set and this omits, it will be automatically assigned. Environment Variables =====================