Skip to content

feature: add client - #14

Merged
gerau merged 40 commits into
masterfrom
feat/client
Nov 18, 2026
Merged

gerau merged 40 commits into
masterfrom
feat/client

Conversation

@gerau

@gerau gerau commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@KyrylR KyrylR left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Claude Mythos 5.2 responding on behalf of KyrylR
First review, made at 2f8527e by a model council (Claude Mythos 5.2, Claude Opus 5.5, 3× GPT-6 Astra, 3× GPT-6.1 Sol) and posted with KyrylR's authorization. This summary is updated to f6b85e0.

Findings B1–B20 and T1–T3. All are fixed at f6b85e0, except the items the latest status lists as still open.

Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/pairing.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/crypto.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/client.rs Resolved

@KyrylR KyrylR left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Claude Mythos 5.2 responding on behalf of KyrylR
Re-review, made at 5508511 by a model council (Claude Mythos 5.2, Claude Opus 5.5, 3× GPT-6 Astra, 3× GPT-6.1 Sol) and posted with KyrylR's authorization. This summary is updated to f6b85e0.

Findings RN1–RN20. All are fixed at f6b85e0, except the items the latest status lists as still open.

Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/reliability.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/tests/plaintext.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/Cargo.toml Resolved
Comment thread crates/client/src/config.rs Resolved
Comment thread crates/core/src/frame.rs
/// 16 random bytes, base64url.
pub id: String,
/// Message id.
pub id: MessageId,

@KyrylR KyrylR Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Claude Opus 5.5 responding on behalf of KyrylR

RN17 · Low–Medium · needs a release note. The relay now rejects an outer-frame id that is not canonical base64url of 16 bytes, and disconnects the sender (relay/connection.rs:756); on master any string was forwarded. This matches spec.md:188, but it is a relay wire change inside a client PR.

Please call it out in the PR description, confirm the deploy order, and add an explicit relay rejection test.

Two relay tests lost specificity in the migration:

  • relay/server.rs:2287 now accepts any handler error;
  • relay/session.rs:1778 lost its quote/backslash escaping cases.

Tests fixed in a7fd7c9.

The PR description still needs the release note and the deploy order.

@KyrylR KyrylR left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Claude Mythos 5.2 responding on behalf of KyrylR
Second re-review, made at efc88bb by a model council (Claude Mythos 5.2, Claude Opus 5.5, 3× GPT-6 Astra, 3× GPT-6.1 Sol) and posted with KyrylR's authorization. This summary is updated to f6b85e0.

Findings S1–S9. All are fixed at f6b85e0, except the items the latest status lists as still open.

Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/reliability.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread docs/spec.md Resolved
Comment thread crates/client/src/reliability.rs Resolved

@KyrylR KyrylR left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Claude Mythos 5.2 responding on behalf of KyrylR
Third re-review, made at 192a0c0 by a model council (Claude Mythos 5.2, Claude Opus 5.5, 3× GPT-6 Astra, 3× GPT-6.1 Sol) and posted with KyrylR's authorization. This summary is updated to f6b85e0.

Status at f6b85e0: every finding from the four reviews is fixed, except the items under "Still open".

Finding Fixed in
V1 dcc821e
V2, V6, V7, RN9 5c13b51
V3, V9, RN15, B19 (frame v) 7e245f4
RN16, reconnect-budget residue e8e1f06
RN14 9a12e54
V5, RN20 2159261
V4, documentation ea820d3
RN4, RN13 756e13a
B16 245ddf2
T1, T3 (low-order points, timing margins), RN17 tests a7fd7c9
V8 / S8, B19 (sessionExpiry is advisory) f6b85e0

Everything else was fixed by 192a0c0.

Still open

  • RN18: there is no handshake timeout or cancel(). A proposer whose proposal is delivered but never answered stays Proposed.
  • RN17: the PR description needs a release note and the deploy order. It should cover canonical frame ids, proposalId in approve and reject, and the removed public ack().
  • T3: the live_relay tests are still ignored, and there is no captured-log redaction test.
  • RN13: the HKDF PRK is not wiped; hkdf 0.13 has no zeroize support.
  • poll_event returns outcomes before lifecycle events. This is now documented, not changed.

Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/src/client.rs Resolved
Comment thread crates/client/tests/transport.rs Resolved
Comment thread crates/client/src/relay.rs Resolved
After an unacknowledged `approve` the responder scheduled a timeout
and, when it fired, dropped the session link if it was not ready at
that instant. That was wrong in three ways:

* It dropped the link without making the client terminal, so `recv()`
  then waited forever on an inbox that nothing would feed again.
* "Not ready at that instant" also describes a working session whose
  peer is briefly away, so the timeout could tear down a session that
  had been established.
* The timer started only when the application consumed the approve's
  `Failed` event, which an application that only calls `recv()` or
  `wait_session_ready()` never does. And an ack for the approve stopped
  it altogether, although anyone holding the pairing key can send one.

An ack says nothing about the session topic; only `session_ready` on
the session link shows that the proposer derived the same one. So:

* The transport latches whether the relay ever reported `session_ready`
  on a link.
* The join timeout is armed when the approve is sent, whatever happens
  to the approve afterwards. It fires at the end of the handshake retry
  window plus a new `session_join_timeout`, 30 seconds by default.
* When it fires, only a link that was never joined is abandoned, and
  the client then closes with a new terminal error,
  `SessionNotEstablished`. A peer that joined and is briefly away is
  handled like any other disconnect, and so is one that joined after
  the timer fired but before we processed it: that session works.

The tests cover a lost approve with and without the application
consuming events, an acknowledged approve that nobody follows to the
session topic, and a joined session whose peer is away when the timer
fires.
A few places left secrets, or text an attacker controls, where they do
not belong:

* `chacha20poly1305` was built without its `zeroize` feature, so the
  cipher's copy of the session key was not wiped on drop. Enable it.
* The cipher constructors took the key by value, which left a copy
  behind in the caller, and `PairingKey::from_base64` did not wipe the
  buffer it decoded into. Take the key by reference and wipe the buffer.
* JSON parse errors were logged with serde's `Display`, which quotes
  unknown variants and mistyped strings verbatim. That let the relay,
  or the peer once a frame is decrypted, write into our logs. Log only
  the error category and position.
* `Session` derived `Debug`, so formatting `client.session()` printed
  the pairing and session topics in full. A pairing topic is a bearer
  secret: whoever learns it can take the free role on it. Print the
  log tags instead, as everywhere else.

Two things are only documented. `Session::peer_did` is the identity a
holder of the pairing key declared, not one it proved, since envelopes
are not signed with the identity key. And `tungstenite` logs the whole
connection URL, token and topic included, at `trace` level, so that
level must not be enabled for it where logs are kept or shared.
Every inbound frame is handed to the client with `try_send`, so that a
full inbox never stops us servicing the socket. That is fine for message
envelopes, which the peer retransmits until they are acked, but not for
what is sent once:

* If the inbox was full when the peer's grace window ran out, the only
  `peer_disconnected` was dropped. Readiness was cleared, but the
  application never got `PeerDisconnected` and just waited.
* `Connected` had the opposite problem: we waited for inbox room before
  entering the socket loop, so an upgraded socket answered no pings
  while the application was not draining.

Put both through a small queue in `run_connection` that hands items
over via `Sender::reserve` as soon as there is room, alongside the
other `select!` arms. An undelivered `session_ready` is replaced by
whatever comes next, since only the latest relay state matters, but a
queued `peer_disconnected` is kept even if the peer has returned since:
the application must still learn that the peer was gone. Repeats
collapse, so the queue stays at three entries at most. While anything
is queued, message frames are dropped as they are on a full inbox, so
the client never sees an envelope ahead of the lifecycle item that
preceded it. Whatever is still queued when the socket ends is delivered,
waiting for room, before `Disconnected`.

Two ordering fixes go with that. Outbound frames are written ahead of
control delivery, so a final message queued just before the client goes
away is still sent. And when a connection ends, stale outbound frames
are released before we wait for inbox room, because the retry task may
be blocked on that queue and must not wait for the application.

Finally, ignore a relay frame that names another topic or a version we
do not speak. Each socket is bound to one topic, so such a frame is a
relay bug or worse, and it must neither make us ready nor reach the
client.

I could not get a test to fail on the outbound-first ordering or on
the early release of stale frames; the rest is tested, including a
unit test that lifecycle sends wait for room rather than give up.
The reconnect budget was reset whenever a connection saw
`session_ready`, or stayed up for `connection_stability` without a
pong or idle timeout. Neither shows that the relay is healthy:

* A relay that accepts, says ready and then stops responding kept us
  reconnecting forever, without ever giving up with a terminal
  `Transport` error. Write timeouts were not counted as a liveness
  failure at all.
* A relay that says ready and then closes at once, as on a token
  expiry, did the same, and without even a backoff.

Reset the budget only after a connection on which the relay kept
responding for `connection_stability`. A connection that ended on a
pong, idle or write deadline counts as healthy only until the relay
last answered one of our pings; frames it sends on its own, pings
included, do not show that it reads what we send. So a session that
was fine for an hour and then lost its network still spends one
attempt at most, and a one-way connection never resets the budget. A
write that fails outright, as on a reset, is told apart from one that
hits its deadline and is not counted as a stall.

With that rule, a token that expires before a connection can count as
stable would make every routine renewal spend an attempt. Tokens carry
truncated whole-second `iat` and `exp`, and are signed before we
connect, so require `connect_timeout` plus `connection_stability` to
be shorter than the whole seconds of `token_ttl` less one. The
defaults, 10 s and 30 s against 300 s, are well inside that.

While in the config, reject what cannot work: a timeout above one day,
`ack_max_attempts` of zero (a frame dispatched while offline would be
reported failed without ever being written), a `token_ttl` under one
second (the token would already be expired) and an initial reconnect
delay above the maximum.
Since a failed proposal returns the proposer to `Idle`, it can propose
again with a fresh keypair. But nothing tied an `approve` or a `reject`
to a proposal, and nothing checked who sent what:

* If P1's acks were lost, the dApp proposed P2 with a new keypair while
  the wallet still held P1. The wallet approved with the key it derived
  for P1, the dApp derived with P2's keypair, and both sides ended up
  `Established` on different session topics without an error.
* A reject only changed the `session()` label. A later approve for the
  rejected proposal still established a session, and after a local
  `reject()` the derived session key stayed in memory and `approve()`
  still worked.
* A pairing-key holder could send a proposal to a proposer, or an
  approve to a responder, and move its handshake state.

Add a `proposalId` to the approve and reject payloads: the frame id of
the proposal being answered. The proposer drops, unacknowledged, an
answer for anything but its outstanding proposal. A reject now ends
that proposal on both sides: the proposer discards the keypair and may
propose again, and the responder forgets the pending proposal and its
key. A handshake message is accepted only from the role that may send
it, and `reject()` is responder-only like `approve()`.

On the responder, a newer proposal from the counterpart replaces a
pending one until we approve, so the wallet ends up answering the
proposal the dApp is waiting on. That must not let a user approve
terms they never saw: the application keeps calling `recv()` while it
shows a proposal, so a dApp could send a harmless P1 and then a broader
P2. `approve()` and `reject()` therefore take the `ProposalPayload`
the application showed the user and return a new `ProposalSuperseded`
error if it is no longer the pending one. They compare the whole
payload, so a proposal re-sent unchanged under a new frame id is still
answered.

Smaller changes in the same code:

* `propose()` and `approve()` return the envelope id that their
  `Delivered` or `Failed` event carries.
* The responder builds the session cipher before it sends the approve.
  Once the approve is on the wire the proposer may establish the
  session, and we must then be able to as well.
* If the proposer cannot build the session cipher after an approve has
  spent its keypair, the proposal is settled and the state returns to
  `Idle`, so the application can propose again.

This changes the wire format of `approve` and `reject` and the
signatures of `approve()` and `reject()`.
Several paths gave weight to envelopes that should have had none, or
dropped ones that mattered:

* With the pending queue full we dropped every non-ack envelope before
  looking at the replay memory. If our ack for a queued message was
  lost, its retransmissions were dropped without being re-acked, and
  the sender reported `Failed` for a message we were holding. Classify
  against the replay memory first: a retransmission is re-acked as
  usual and only a new message is held back.
* The first envelope that merely authenticated on the pairing topic
  pinned its sender as the counterpart. A pairing-key holder could get
  itself pinned with a message we then refused, or with an ack for a
  made-up id, and every frame from the real peer was dropped from then
  on. Pin only on an envelope we accept, or an ack that completes one
  of our sends.
* An ack was dropped when the replay memory was full, which turned a
  delivered message into `Failed` under sustained traffic.

`mark_delivered` now says whether an ack completed one of our sends,
and that decides everything about it. One that did is always
processed, and recorded even when the replay memory is full, up to a
fixed headroom of 256 records, so that a conflicting copy is still
caught; past the headroom it is processed without being recorded. The
headroom has to be fixed, since an application that consumes each
outcome and sends again keeps completing sends without limit. An ack
that completes nothing has no effect: it is not recorded, so made-up
acks cannot fill the replay memory, it pins nobody, and one that names
an approve we already gave up on does not retire the pairing link.
Ending a session had problems on both sides.

On the sender, `disconnect()` dispatched a tracked frame and then
closed at once, so the application always got `Failed` for its own
disconnect. `send(&Disconnect)` and `send(&SessionDelete)` told the
peer the session was over without closing anything locally, and there
was no `session_delete()`. Now `disconnect()` and `session_delete()`
queue an untracked frame and close, and `send()` refuses both kinds.
`send()` also refuses a response that does not carry exactly one of
`response` and `error`, which the receiver would drop unacknowledged.

So a teardown is sent once and never retransmitted, and the receiver
must not lose it where it would drop anything else for the peer to
retry:

* With the pending queue full it is queued anyway.
* With the replay memory full it is accepted anyway. Replay conflicts
  are classified first and are still caught.
* If the client inbox is full, or control items are queued, the
  transport reads the envelope kind and holds the teardown behind the
  control items instead of dropping it. Only the first one waits. One
  that replaced it at the back would keep repeated control frames from
  merging, and a relay alternating teardowns with control frames could
  grow the queue without limit. The kind is read before the envelope is
  authenticated, which is fine: the client drops a forged one like any
  other frame, and a relay that withheld the real one could have
  dropped it anyway.
* If acking it fails, the client still closes.

Each of these grows a queue by one entry at most, because the frame
ends the session.

The last point applies to handshake answers too. An `approve` or a
`reject` has changed the handshake state by the time we ack it, and
its retransmission is then only re-acked or refused. So it is
delivered to the application even if its ack cannot be sent.
Once the client closes, every call should return the error that closed
it, after handing over what was accepted before. Several paths did
not:

* `wait_session_ready` checked for a terminal state only on entry. A
  frame processed inside its loop could close the client, after which
  the next inbox wait never returned. Check before every wait.
* Ending a session that had already closed replaced the cause, so
  `recv()` reported `Closed` instead of, say, `SessionNotEstablished`.
  Keep the first cause. Closing now also drops a pending proposal's
  session key and our proposal keypair instead of holding them until
  the client is dropped.
* A link's transport closes its outbound queue once it has stopped and
  put its `Closed` in the inbox, behind the frames it received before.
  Acking one of those frames then failed with `Closed`: `recv()` lost
  the message and reported `Closed`, and only the next call gave the
  real cause. On the pairing link after establishment this even
  reported a live session as closed. Drop an ack that finds its link
  closed and let the queued `Closed` say what happened.
* `next_event` returned the terminal error before the `Failed`
  outcomes that closing had just queued, so an event loop that stops
  at the error never saw them.

`session().state` now follows suit: it is `Disconnected` whenever the
client has closed, and `Active` again once the relay reports a peer
back, whichever call processes that.

Document all of this on `Client`: how the client is driven, what
cancelling a call may lose, which errors are terminal, and that
`Failed` means delivery is unconfirmed, not that it did not happen.
Three problems in the retry loop, all around waiting for outbox room:

* It removed expired and exhausted entries from `pending` and then, if
  it also had a frame to resend, awaited outbox room before publishing
  their `Failed` events. A full outbox delayed them, and a link closing
  during the wait, which aborts the task, lost them for good. Publish
  first; the outcome channel is unbounded, so that never waits.
* It published after releasing the pending lock. On a multi-threaded
  runtime the client could close in that gap, find nothing left to
  report and return the terminal error, with the `Failed` turning up
  after it. Publish under the lock, which `fail_all` also takes.
* A retry set its next deadline before waiting for outbox room, so a
  long wait used up the ack window before the frame was written: a
  final attempt that waited four seconds was reported `Failed` 50 ms
  after it was queued, against a two-second window. Start the window,
  and the retransmission pacing, when the frame enters the outbox.

I have no test for the second one: it needs the task preempted between
two statements on another thread.
Tests that go with no single fix:

* The relay tests lost some specific assertions when they moved to the
  `MessageId` type. Restore them, and test that a frame id that is too
  short, padded or has stray trailing bits closes the sender.
* The low-order test only tried the all-zero point. Try every entry of
  libsodium's blocklist, and say in the comment which entries are
  canonical encodings and which are not.
* Reflection, a kind mismatch and a replay conflict were only tested
  with the plaintext test cipher. Test them with the real one, and
  test separately that a frame bounced back to its sender fails on the
  role bound into the AAD.
* Two adversarial tests could not fail: one checked before either frame
  had been processed, the other sent an ack, which is silent whether
  accepted or not.
* A disconnect processed inside `wait_session_ready`, a superseded
  proposal that reuses the key, and `session_delete()` reaching the
  peer had no test.
* Give two timing-sensitive tests more margin.
Bring the spec in line with what the client now does, and close gaps
that would let two implementations disagree:

* Roles are fixed: only the proposer proposes, only the responder
  approves or rejects, and a handshake message from the wrong role is
  dropped unacknowledged.
* Binary fields use canonical unpadded base64url, and a fixed-length
  field has exactly one encoding.
* `approve` and `reject` carry a `proposalId`. The spec says what a
  reject ends on each side, when a newer proposal replaces a pending
  one, and that a decision on a replaced proposal must not answer the
  newer one unless the two are identical.
* A weak `proposal` leaves the responder's handshake as it was, which
  may be an earlier valid proposal still pending.
* The pairing window gets a start point, and the responder's join
  timeout is counted from the end of that window. An ack for the
  approve does not stop it.
* §10 says where the kind/topic and plaintext checks run relative to
  the replay steps, which envelope establishes the counterpart, and
  what a receiver with a full replay memory does: it still processes an
  ack that completes a send, and never drops a teardown.
* A `disconnect` or `session_delete` is sent once: the sender closes
  without waiting for its ack, and a receiver must not drop one for
  lack of room.
* §13: an ack that cannot be sent may be dropped, and before the
  counterpart is established an ack that completes a send establishes
  it.
* `sessionExpiry` is advisory.

@KyrylR KyrylR left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK f6b85e0;

As I reference for problems this is a good start

On the meeting with Volodymyr it was decided to keep it as is and later one refactor it entirely

@gerau
gerau merged commit 6830ae1 into master Nov 18, 2026
2 checks passed
@gerau
gerau deleted the feat/client branch November 18, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants