Tcp reassembler

.scratch/tcp-reassembler/issues/01-handshake-and-in-order-data.md

@@ -0,0 +1,48 @@
+# 01: Handshake and in-order data reach the reader
+
+Parent: ../spec.md
+
+**What to build:** A user creates a `TcpReassemblyPool` with the default `TcpReassemblyConfig` and feeds it, through `process_segment`, the segments of a connection whose handshake was captured. The pool groups both directions under one connection and returns it in `TcpProcessed::active`, together with the role of the sender. The connection has:
+- a sorted connection key;
+- a pool-assigned connection id, starting at 0;
+- observed initiator and responder roles;
+- first-seen and last-seen timestamps.
+
+Each stream delivers its in-order bytes as one contiguous slice through `read_received()`, and `consume()` accepts partial amounts. Sequence numbers map to 64-bit offsets that keep growing across 2^32 wrap-around. Window scale options in the SYN and SYN-ACK are parsed and stored for later tickets. Data in the SYN (TCP Fast Open) is delivered at offset 0.
+
+Segments of connections whose handshake was not seen are ignored with `NoConnection` for now; ticket 07 changes that.
+
+This slice also lays the groundwork:
+- the `std`-gated `tcp_reassembly` module;
+- the value types this slice needs;
+- the config with the spec's defaults;
+- a test-only builder that writes connections as scripts and produces segments for both directions;
+- the reference model, starting with in-order random streams;
+- the `AGENTS.md` allocation rule extended to name the new module.
+
+Rules: S1, S2, A1, A3 (parsing only), A7 (observed roles), P4, P7.
+
+**Blocked by:** None (can start immediately)
+
+**Status:** ready-for-agent
+
+- [x] A handshake followed by in-order data in both directions delivers both streams byte-exact through `read_received`/`consume`, including partial consumes
+- [x] Proptest against the reference model: random ISNs (including ISNs within 2^16 of 2^32) and random in-order segmentation
+- [x] Offsets keep growing across several full 2^32 wraps
+- [x] TCP Fast Open data in the SYN is delivered at offset 0
+- [x] `TcpConnectionKey::new(a, b)` and `TcpConnectionKey::new(b, a)` are equal and hash equally; the config defaults are pinned; the derived traits are tested
+- [x] Connection ids start at 0 and increase per connection; `first_seen`/`last_seen` follow the timestamps fed in
+- [x] Segments with a different key create a separate connection
+- [x] All new behaviour is tested through the pool's public API (value types directly), with every branch covered; `cargo test`, `cargo test --no-default-features` and the bare-metal no_std build pass
+
+## Comments
+
+**2026-10-10 — implemented** on `tcp-reassembler` in three commits: `3a0ffea` (value types + generators), `b9ff8f1` (pool, connection, stream, connection script, reference model), `495c44d` (AGENTS.md allocation rule). Full suite, the documented `cargo test --no-default-features`, the bare-metal no_std build and `cargo +1.83.0 check` pass; module tests stay green at `PROPTEST_CASES=3000` (~2 s). Library code in the module has 100% line/region coverage.
+
+Decisions made while implementing:
+- `TcpConnectionKey` keeps its fields private with getters, so the sorted-endpoint invariant can't be broken.
+- `read_received()` takes `&mut self` (it records the length for the `consume` clamp), so both streams of a connection can't be borrowed for reading at the same time.
+- `TcpProcessed`, `TcpIgnoreReason` and `TcpRead` only have the fields/variants this ticket needs; later tickets add `ended`, the other ignore reasons, `Gap` and `End`.
+- The window scale is parsed and stored on `TcpStream` behind `#[allow(dead_code)]`. Its value can't be observed through the public API yet (see ticket 04).
+- Overlapping retransmissions are already trimmed (first wins, no conflict counting). Without that, a re-cut segment would lose its new bytes. Wrap handling only shows up for overlaps that cross 2^32, so the proptest re-sends overlapping bytes. A naive `seq > end_seq` comparison is caught by both the unit test and the proptest.
+- The spec never defines simultaneous open. The code anchors the responder stream on any SYN from it, but no test pins this.

.scratch/tcp-reassembler/issues/02-clean-connection-end.md

@@ -0,0 +1,51 @@
+# 02: Connections end cleanly and late packets are absorbed
+
+Parent: ../spec.md
+
+**What to build:**
+- The first FIN of a stream sets its FIN offset. Data beyond it is never readable and is counted as out-of-window.
+- Once all bytes up to the FIN have been read, reading returns `End(Fin)`.
+- When both FINs are ACKed, the connection is finished: the pool hands it over as an owned value in `TcpProcessed::ended`, with the end `Finished`.
+- `is_finished()` and `is_complete()` work.
+- `recycle()` takes an ended connection back.
+- A tombstone keeps the key, so late segments of the ended connection are ignored with `AfterEnd` instead of creating a new connection.
+
+Checking a FIN against the window is added in ticket 04.
+
+Rules: F1 (first FIN wins), F2, F3, F6 (`End(Fin)`), D5, P3.
+
+**Blocked by:** 01
+
+**Status:** ready-for-agent
+
+- [x] Full lifecycle (handshake, data, both FINs, final ACKs): the connection is handed over as `Finished`, both streams read `End(Fin)`, and `is_complete()` is true
+- [x] If the last ACK is missing, the connection stays open while both streams report finished
+- [x] Every FIN order is tested (initiator first, responder first, simultaneous)
+- [x] Data after the FIN is never readable and is counted in `out_of_window_segments`; a second FIN at another offset is ignored
+- [x] A late retransmission after the end is ignored with `AfterEnd` and creates no connection
+- [x] `recycle()` accepts an ended connection
+- [x] The reference model is extended with FIN and close
+- [x] All new behaviour is tested through the pool's public API (value types directly), with every branch covered; `cargo test`, `cargo test --no-default-features` and the bare-metal no_std build pass
+
+## Comments
+
+**2026-10-10 — implemented** on `tcp-reassembler` in six commits:
+- `2091cec`: value types `TcpStreamEnd`, `TcpConnectionEnd` and `TcpStreamStatus`, plus a generator;
+- `1ec455c`: refactor that maps sequence numbers to offsets in one place;
+- `b750bce`: test-only refactor of `ConnectionScript` (`segment()` + `push(segment)`), and the reference model state grouped per stream;
+- `6f013fc`: FIN handling (F1, F2, D5, `End(Fin)`);
+- `6102349`: hand-over as `Finished`, tombstone, `AfterEnd`;
+- `35c2b2b`: `recycle()`.
+
+The full suite, the documented `cargo test --no-default-features`, the bare-metal no_std build and `cargo +1.83.0 check` pass. The module tests stay green at `PROPTEST_CASES=3000` (~2.4 s). Every library line and region of the module is covered in the merged `cargo cov show` view. The summary report still lists misses for single instantiations of the generic pool. A review on both axes (standards and spec) ran, and its fixes are folded into the commits above.
+
+Decisions made while implementing:
+- **A FIN that lies before bytes already received is ignored**, and not counted. Downside, from the spec review: bytes injected beyond the real FIN, before that FIN arrives, block it. The stream then never finishes, and the injected bytes stay readable. The alternative is to accept such a FIN as long as it lies at or after `read_offset`, and to drop and count the unread bytes beyond it. But that lets an injected FIN cut off real bytes that were received and not read yet, which goes against user story 44. **The user decided on 2026-10-10 to keep ignoring it.**
+- Only segments with payload bytes after the FIN count as out-of-window. Pure ACKs and FIN-only segments after the FIN don't count. A second FIN at another offset is ignored without counting; ticket 04 adds counting for FINs beyond the window.
+- `status()` and `TcpStreamStatus` (so far with only `out_of_window_segments`) were pulled forward from ticket 03, so that the counter is observable.
+- Each stream keeps a private seen ACK (`TcpStream::seen_ack`): the highest ACKed offset, mapped relative to the end of the received bytes, with no plausibility check yet. This way an ACK captured before its FIN still counts. The ACK of a FIN is `fin_offset + 1`.
+- **The tombstone is a `TcpConnection`** with `end` set and empty streams. `TcpConnection::hand_over` leaves it in the same map. It keeps the key, id and timestamps. An enum entry would have needed unreachable match arms: the borrow checker doesn't allow conditionally returning `&mut` into the entry and then replacing the entry (NLL problem case 3).
+- A segment that finishes the connection returns `active: None` and the connection in `ended`.
+- Every segment on a tombstone returns `AfterEnd`, SYNs included; C1 is ticket 10. Late segments don't refresh the tombstone's `last_seen`.
+- `recycle()` puts the emptied buffers in an unbounded free list, and new connections take their buffers from it. The tests only check that no old bytes leak into a new connection. Whether a buffer is actually re-used can't be observed yet.
+- `is_complete()` equals `is_finished()` for now.

.scratch/tcp-reassembler/issues/03-out-of-order-retransmissions-holes.md

@@ -0,0 +1,56 @@
+# 03: Out-of-order data, retransmissions and holes
+
+Parent: ../spec.md
+
+**What to build:**
+- Segments that arrive out of order, duplicated, or re-cut with overlaps are put back in order and delivered exactly once.
+- Missing ranges are holes. `holes()` lists them, and reading returns `Pending` at a hole until it is filled.
+- For overlapping bytes, the first received content wins. A retransmission with differing bytes increments `conflicting_retransmissions` once per segment. All bytes count as not ACKed until ticket 04.
+- `highest_offset()`, `status()` and `is_clean()` report the stream state.
+- A stream with a hole before its FIN is not finished.
+
+Rules: D1, D2, F2 (holes), Q1.
+
+**Blocked by:** 01
+
+**Status:** ready-for-agent
+
+- [x] Proptest against the reference model: random segmentation, reordering within a bounded distance, duplicates, and overlapping retransmissions that are cut differently; the result is byte-exact
+- [x] Corrupted retransmissions: the first received bytes are delivered, and the counter equals the number of conflicting segments
+- [x] `read_received` returns `Pending` at a hole and continues once the hole is filled; `holes()` and `highest_offset()` are checked exactly
+- [x] A stream with a FIN but a hole before it becomes finished only once the hole is filled
+- [x] A long stream that is consumed in small steps is delivered correctly (exercises buffer compaction)
+- [x] All new behaviour is tested through the pool's public API (value types directly), with every branch covered; `cargo test`, `cargo test --no-default-features` and the bare-metal no_std build pass
+
+## Comments
+
+**2026-10-10 — carry-over from ticket 01:** `TcpStream::add_data` maps sequence numbers relative to the end of the contiguous received bytes, and drops data that lies behind missing bytes. S2 asks for mapping relative to the highest offset. This ticket has to switch to that, otherwise one lost segment stalls the stream. Overlap trimming (first wins) already exists; conflict counting (D2) does not. The test block "reading stops at missing bytes" in `tcp_stream.rs` covers today's hole path.
+
+**2026-10-10 — carry-over from ticket 02:** three places rely on data behind missing bytes not being stored:
+- `TcpStream::add_segment` checks a new FIN against `received_end()`;
+- `is_finished()` compares `received_end()` with the FIN offset;
+- `offset()` maps relative to `received_end()`.
+
+Switch them to the highest offset, and to "no holes before the FIN" (F2). The reference model's `ModelStream::add_segment` uses `received.len()` for the FIN check too. `is_complete()` is still just `is_finished()`, so it has to add `is_clean()`. Until then, a stream whose retransmissions carried different bytes (dropped by first-wins) still reports complete.
+
+**2026-10-10 — user decision:** a FIN that arrives before the bytes in front of it is recorded, and the stream finishes as soon as those bytes arrive. A FIN without payload already does this: see "FIN behind missing bytes finishes once they arrive" in `tcp_stream.rs`, and "both FINs are ACKed, but bytes before a FIN are missing" in `tcp_connection.rs`. A FIN segment that carries its own payload behind a hole still loses that payload, because the payload is not stored. This ticket must keep that payload, and test the case at connection level too: the connection ends as `Finished` once the hole is filled.
+
+**2026-10-10 — user decision:** a FIN that lies before bytes already received is ignored (F1). Example: bytes 0..10 are received and a FIN arrives at offset 6; the FIN is dropped and bytes 6..10 stay readable. A real sender never moves its FIN, so either the FIN or the bytes are forged or corrupted. Accepting the FIN would let a forged FIN cut off real bytes that haven't been read yet (user story 44). When the FIN check moves to the highest offset, keep this rule: a FIN before the highest received offset is ignored, even if the bytes in front of it have holes.
+
+**2026-10-10 — implemented** on `tcp-reassembler` in three commits:
+- `1993162`: test-only refactor. The reference model tracks the read offsets, and the feed-and-check loop and the interleaved segment cutting move into helpers.
+- `823cdbc`: holes. Sequence numbers map relative to the highest offset, bytes behind missing bytes are stored, and `holes()` and `highest_offset()` are added. A FIN behind missing bytes moves the highest offset to it, F2 applies, and the proptest `reordered_streams` covers reordering, duplicates and re-cut retransmissions.
+- `5f8cae5`: `conflicting_retransmissions`, `is_clean()`, and `is_complete()` = clean and finished. Corrupted retransmissions are added to `reordered_streams`, and the new proptest `conflicting_retransmissions` checks the counter against an independent count. The regression seeds are committed.
+
+The full suite, the documented `cargo test --no-default-features`, the bare-metal no_std build and `cargo +1.83.0 check` pass. Each commit passes the module tests on its own. With `PROPTEST_CASES=3000` the module tests take about 8.5 s; `reordered_streams` is the slowest at about 6.6 s. Every library line and region of the module is covered in the merged `cargo cov show` view. Hand-made mutants were each caught: no split of a hole, `>` instead of `>=` in the FIN check, no hole before the FIN, broken compaction, and conflict comparison skipping or including hole bytes. A two-axis review ran (standards and spec), and its fixes are folded into the commits above.
+
+Decisions made while implementing:
+- **Bytes the user already consumed are not compared for conflicts.** They are no longer buffered. The user decided this on 2026-10-10. D2 in spec.md and "Conflicting retransmission" in CONTEXT.md now say so. For `read_acked()` users nothing changes: their consumed bytes are ACKed, and D3 skips those anyway.
+- A FIN behind missing bytes moves the highest offset to the FIN offset, and the missing range is a hole listed by `holes()`. Once the FIN is set, nothing lies beyond it, so `is_finished()` is just "FIN seen and no holes". CONTEXT.md's "Hole" ("with later bytes already received behind them") doesn't quite cover a FIN without payload behind the hole. It was left unchanged.
+- Segments without payload (pure ACKs) don't move the highest offset. Otherwise a single injected empty segment could open a large hole.
+- `holes()` returns `impl Iterator<Item = Range<u64>> + '_`.
+- Holes are zero-filled placeholders in the contiguous buffer (ADR-0001). The buffer only reaches the furthest received byte; a hole up to a FIN beyond it allocates nothing.
+- Mapping relative to the highest offset (S2) differs from mapping relative to the end of the contiguous bytes only for distances near 2^31. No test pins it directly.
+- `TcpStreamStatus` is now 128 bits, which is the coding-style limit for `Copy`. The spec requires `Copy`, but the counters added by tickets 05, 08 and 11 push it past 128 bits. That conflicts with the coding style, so the user needs to decide.
+
+**2026-10-11 — user decision:** `TcpStreamStatus` no longer derives `Copy`, and `status()` returns `&TcpStreamStatus`. Commit `f9db588`. spec.md is updated.