Skip to content

feat(payment)!: enforce the close-group price floor on every node - #257

Open
grumbach wants to merge 16 commits into
WithAutonomi:mainfrom
grumbach:price-floor-live-enforcement
Open

grumbach wants to merge 16 commits into
WithAutonomi:mainfrom
grumbach:price-floor-live-enforcement

Conversation

@grumbach

@grumbach grumbach commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

The ADR-0006 close-group price floor has run in shadow mode since v0.17.1. This turns it on for every node at 30%, with nothing to configure. It also fixes what made enforcement unsafe: the tolerance, griefing by neighbours, a bypass through paid-list repair, and, within a measured margin, trust penalties for refusing a write under the floor.

A week of shadow telemetry (ant-prod-01, 2026-10-14 to 2026-10-20, 699 node instances on 72 hosts, node 0.22.0 and 0.23.0) and a review of the enforcement path found four problems:

  • At its 65% default, the floor would have refused 2.5% of admissions (47 of 1,898). 2.1% of chunks would have been refused by every receiver that evaluated them. An earlier window, 2026-09-10 to 2026-09-16, showed 3.4% and 1.5%.
  • Neighbours gossiping inflated commitments could make every receiver refuse a chunk.
  • One accepting node could still replicate a below-floor write, through PaidNotify and paid-list repair.
  • Four existing checks would turn a price refusal into a trust penalty, three on the refuser and one on the node that accepted the write. Shadow mode never refused, so none of them has fired:
    • the storing node's possession check, at weight 5 for a chunk (about a 48% cut in trust) and weight 1 for a pointer;
    • the responsible audit, which challenges a peer on any key it was hinted as a replica, at weight 5;
    • the prune audit, which penalises an absent close-group peer at weight 5;
    • and peers whose quorum check fails for a key only one node hinted penalise that node, at weight 1.

What changes:

  • The reference is capped at the receiver's own price: min(group median, calculate_price(own committed count)). A node never demands more than the tolerance of what it would itself quote. Neighbours therefore cannot make it refuse a payment that clears its own price, however many of them lie. While gossip is honest, receivers above the median price exactly as before.

  • Tolerance 65% → 30%, a deliberate starting point. No measured admission is refused at 30%, or at 40%, in either production window: 1,898 admissions in October (1,801 of them matched for the capped replay), and 1,103 evaluations in September. The worst October admission clears 30% by 1.78x but 40% by only 1.34x, on a light week of 48 upload bursts. So the floor starts at 30%. A follow-up raises it toward 40%, by a later ADR amendment, once enforced traffic shows that 40% refuses no honest payment. A baseline quote (a near-empty or sybil node) is still refused by 91% of receivers, against 98% at 40%.

  • Paid-list admission is floored too. PaidNotify announces a client's write, with the proof the client presented. Repair and fetch never call the verifier. So this reprices no historical receipt, and one accepting node can no longer give a below-floor write the paid-list votes that repair needs.

  • No peer is judged on a contested write. When a node's verifier first admits the payment for a write (the PUT, a fresh offer or a PaidNotify), it prices the proof against its current view; a later admission of the same write from the payment cache reuses that decision. The write is contested when the node cannot price it (no own commitment, or too thin a view), or when its paid price is below 60% of the uncapped group median, twice the floor. An admitted contested write, one the node cannot price or one paid between the floor and that threshold, is still stored and offered, but:

    • the fan-out (chunk or pointer) schedules no possession check;
    • for a chunk, the key's paid-list entry gets a one-byte mark, which survives restarts. The mark follows the decision the node took when it verified the payment: the verifier remembers which admitted chunk payments the floor contested, in a memory of its own, so a write admitted later from the payment cache is recorded by that decision, and the proof attached to it, which anyone can attach to a PaidNotify or fresh offer and nobody checked, can neither set nor veto the mark. The first paid-list record of a key decides its mark, and a fan-out never schedules a possession check for a write that is marked, by the verified decision or by the record that won, which it reads after recording its own;
    • a marked key goes out in paid hints only, never as a replica hint, so no repair proof is recorded for it, no responsible audit challenges a peer on it, and no peer can penalise this node as its sole hinter. Repair still works through paid hints wherever the group accepted the key. Since pressure-gated retention (feat(replication)!: erase out-of-range records only under disk pressure #271), a node with more than 100,000 paid entries builds paid hints only from keys near itself, so a marked key it stores is a paid-hint candidate at any distance, sent to the members of its close group that a replica hint would have reached. The paid list also keeps its marked keys in memory, updated in commit order and cleared by any write that leaves a key unmarked, so building the hints for a peer reads nothing on disk for them but each far stored marked key's paid entry. If the marks cannot be read when the node starts, it still starts and hints no key as a replica until a restart can read them, and every key it stores with a paid entry is a paid-hint candidate at any distance instead;
    • the responsible audit never samples a marked key; a prune audit, which feat(replication)!: erase out-of-range records only under disk pressure #271 sends only under disk pressure, still needs its proofs before deleting one but scores no absent peer; paid-list eviction (feat(replication)!: erase out-of-range records only under disk pressure #271) keeps the entry of every record still in the store, readable or not, and an evicted entry takes its mark with it; and a mark that cannot be read counts as set, with one warning per run of failed reads.

    A merkle proof, a proof that does not parse, or a node with no floor wired does not by itself make a write contested; a remembered contested admission or an existing mark still turns the checks off for that key. The 2x margin covers the largest spread measured between receivers' views of one chunk (1.91x). Replayed over all 5,536 pairs of measured receivers, at payments from 0.2x to 1x the median in 0.02x steps, no refusal would be judged at 2x, against 4 pairs at 1.5x. The cost is that these checks are given up on 1.4% of the 1,801 measured admissions, and separately on the writes the floor cannot price, 2.6% of the 2,117 floor evaluations. It is a margin, not a guarantee. A guarantee needs the refuser to tell the sender, and a node that can say "refused on price" can also dodge ADR-0003's full-node detection that way, so that is a separate decision. The threshold is its own constant, so a later raise takes two releases: the threshold first, then the tolerance once nearly every node runs it.

  • Enforced on every node, nothing to configure. PriceFloorConfig is gone: the tolerance is one public constant, PRICE_FLOOR_TOLERANCE_PERCENT, and the contest threshold a private one, PRICE_FLOOR_CONTESTED_PERCENT (60). ANT_PRICE_FLOOR_ENFORCE and ANT_PRICE_FLOOR_TOLERANCE_PERCENT are no longer read, without a warning. The floor is the same everywhere, because a node set stricter than its peers would be penalised by their checks. Backing it out takes a release. A node whose replication engine fails to start runs without one, as before, and its floor skips every evaluation (skip_reason=no_commitment_cache), as does ant-devnet's.

  • Telemetry adds target, group_median_price, own_price, paid_quote_age_secs and contested to the existing ant_node::payment::price_floor line. enforce is now always true, which, like target, tells upgraded nodes apart from shadow-mode ones during the rollout. Each line carries what a replay at any tolerance needs, and the follow-up uses that to judge 40%.

  • Tests.

    • tests/poc_price_floor_live.rs had stopped exercising the close-group floor: its rejection test failed on main, which now ignores it as stale. It is repaired here, and CI runs it since ci: consolidate PR CI, run every test target, build the shipped binary #269.
    • Unit tests pin the 30% tolerance and the 60% contest threshold to the atto.
    • Four new e2e tests drive the real schedulers: the chunk possession check, mark and hints; the pointer possession check; the prune audit, under disk pressure; and the three record paths (PUT, fresh offer, PaidNotify), each of which must mark a key exactly when its own verified decision was contested, never from the forged proof attached.
    • An env-gated A/B workload test, tests/e2e/price_floor_ab.rs, returns at once unless PRICE_FLOOR_AB=1, so CI only compiles it.
    • A comment documents why the group sample needs no self-filter: the gossip cache cannot hold this node, since ingest accepts only senders in the routing table, which never holds self.

Linear issue

Closes V2-870

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

  • Wire: none. No message changes. Upgraded nodes refuse below-floor payments that older nodes admit, and both interoperate, except that an older node penalises upgraded peers that refuse a below-floor write it fanned out (see Rollout skew).
  • Storage: the paid list's value byte, always empty until now, marks price-contested keys. Releases before this one only test presence, so they read a marked key as an ordinary paid key, and a rollback leaves the marks unread.
  • API: removes PriceFloorConfig (with its from_env, enforce and tolerance_percent), PaymentVerifierConfig::price_floor, PaymentVerifier::price_floor_config, the test-utils PaymentVerifier::set_price_floor_for_tests, PRICE_FLOOR_ENFORCE_ENV and PRICE_FLOOR_TOLERANCE_ENV; adds PRICE_FLOOR_TOLERANCE_PERCENT, five PaidList methods for the mark (insert_price_contested, is_price_contested, counts_as_price_contested, retain_unmarked and split_marked) and paid_list::is_near. neighbor_sync::build_replica_hints_for_peer now takes the paid list, and neighbor_sync::build_paid_hints_for_peer the chunk store and the close group size (audit::audit_tick_with_repair_proofs takes the paid list on main already, since feat(replication)!: erase out-of-range records only under disk pressure #271), and ReplicationEngine::replicate_fresh, which nothing in production calls, is now test-only (test-utils). The two ANT_PRICE_FLOOR_* variables are no longer read. ant-client's test support imports PriceFloorConfig and sets price_floor: PriceFloorConfig::default(); both go when it moves to this release.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

Production replay, October. The source is shadow telemetry from ant-prod-01, 2026-10-14 to 2026-10-20: 2,117 floor lines from 699 instances on 72 hosts. Replaying the logged formula reproduces all 53 of the nodes' own below_floor=true lines (47 distinct admissions). That leaves 1,898 admissions (each a distinct chunk and receiver) over 536 chunks, of which 1,801, over 521 chunks, are matched to the receiver's own committed count. Refused admissions out of those 1,801:

tolerance median only (today) capped (this PR)
30% (new) 0 0
40% 0 0
50% 1 0
55% 11 1
65% (old default) 47 11
  • The lowest settled-to-reference ratio is 0.534 with the cap (0.473 without). That clears 30% by 1.78x and 40% by 1.34x.

  • With zero refusals observed, the 95% upper bounds are 0.17% per admission, 0.58% per chunk (521 chunks hold a matched admission) and 6.4% per upload burst (47 bursts hold one).

  • Share of receivers refusing a 3x payment to a quoter, with the cap:

    quoter's count, as a share of the group median 30% 40%
    0 (a baseline quote) 91.4% 98.0%
    a quarter 75.4% 94.6%
    40% 27.9% 75.2%
    half 0% 39.1%

    Every observed receiver refuses a baseline quote on 84.5% of chunks at 30% (95.2% at 40%). Not caught at 30%: a quoter 40% as full as its group is admitted by 72% of receivers and one half as full by all of them, nor a client shopping for the cheapest member of the close group (no lower than about 0.7x the median).

  • If every neighbour gossiped the protocol maximum, all 1,801 admissions would still clear at 30%. Today's rule would refuse all of them.

  • Replaying the contest margin over 5,536 ordered pairs of receivers of the same chunk, at payments from 0.2x to 1x the median in 0.02x steps: 0 pairs would see a refusal judged at 2x, against 4 at 1.5x. Moving the tolerance to 40% without first lifting the threshold would leave 3.

Second production window, September. This is an independent window a month earlier, 2026-09-10 to 2026-09-16: 1,134 floor lines from 557 instances on 71 hosts, with 1,103 evaluated over 271 chunks. It was measured under today's median-only rule, which refuses at least everything the capped rule does. At 50% that rule refused nothing. At 55% it refused 0.93% of admissions, and at 65% 3.43%, with 1.48% of chunks refused by every observed receiver. September's worst admission therefore sat at 0.50x or above under the stricter rule. October's sat at 0.473x under that rule and 0.534x capped. Neither window has a refusal at 30% or 40%. Own commitment counts were not joined for that window, and its raw dump was not kept, so these figures are from the analysis done at the time.

Live A/B, tests/e2e/price_floor_ab.rs. The same workload ran on main (shadow mode, 2026-10-20) and on this change at 30% (on 375d62f, this PR's code before its rebase onto main). The main run used an earlier revision of the driver, which asserted the shadow outcome and checked trust only on the peers that refused a direct PUT; it was not re-run with this driver. Setup: 24 nodes seeded to median committed counts of 3,949 for this change (range 3,809–5,785) and 4,536 for main (range 3,841–5,818), with real payments on Anvil and a 20–30 s possession delay.

main (shadow) this change (30%)
12 honest uploads 84/84 stores, 0 floor refusals 84/84 stores, 0 floor refusals in 387 floor evaluations
8 near-empty-quoter uploads (paid at baseline) 56/56 mature storers accept 56/56 mature storers refuse on the floor; 48 fresh offers and 152 PaidNotify also refused
near-empty write after the tail 8–9 holders, 17–20 paid-list votes 1 holder (the quoter), 1 vote
possession penalties 0 0: all 8 fan-outs were contested and scheduled no check, and none of the 10 peers that refused or that the quoter fanned the writes out to took a penalty-sized trust drop in the quoter's view (lowest ratio 0.768, against about 0.52 after a possession penalty)

Tests.

  • cargo test --lib --bins --features test-utils: 1,276 pass in the library, and 10 and 5 in the two binaries, on 8d12c17, whose code is the head's, and cargo test --lib --no-default-features 1,219.
  • The floor tests pin, at the default tolerance:
    • the 30% tolerance and the 60% contest threshold, to the atto, and that the contest reads the uncapped median when the cap binds;
    • the cap;
    • the griefing and disarming bounds. Seven of fifteen overstating neighbours flip nothing and eight flip the verdict. Six understating neighbours leave the floor standing against a baseline quote and seven disarm it, against eight at 40%;
    • all 15 neighbours forged to the maximum still unable to lift a receiver above its own price;
    • the cheapest group member admitted, and a near-empty quoter refused;
    • paid-list and pointer payments floored;
    • the contest decision, including a view that changes after admission;
    • a node with no floor wired contesting nothing;
    • the mark: the verifier remembers a contested payment it admitted, in a memory a small payment cache cannot shrink, and a later cache hit reads that decision, a refused payment is never remembered, the first paid-list record of a key decides, and a marked key is kept across a restart, left out of replica hints and responsible-audit sampling, and, once stored, a paid-hint candidate at any distance; eviction takes an evicted entry's mark with it, and keeps the entry of a record that cannot be read for the moment;
    • the in-memory copy of the marks: filled on opening, changed in the order writes take (a marked insert and a batch removal wait for the write order), cleared for a key by a later plain insert or removal of it when a failed commit left its mark in memory alone, used to build replica hints while the paid list cannot be read, and, when the marks cannot be read on opening (a test points the database root past the file), the paid list still opens and hints no replica, and every stored key with a paid entry is a paid-hint candidate at any distance;
    • a mark that cannot be read counts as set in the responsible audit and a write's fan-out, and a run of such reads logs one warning and a recovery line (the tests make the read fail by holding the thread's one LMDB reader slot).
  • e2e, each with a control: possession_check_skips_a_write_the_floor_cannot_price (a PUT through the handler schedules no possession check when the storer's verified decision was contested, when only the proof it forwards is, or when a marked record of the key won first; a marked key gets a paid hint only), a_fresh_pointer_write_the_floor_cannot_price_is_not_used_to_judge_peers, prune_audit_scores_no_peer_for_a_price_contested_key (its passes now run under disk pressure, since feat(replication)!: erase out-of-range records only under disk pressure #271 sends prune audits only then), and every_record_path_marks_a_key_from_its_own_verified_decision. They settle nothing on chain, so where a test needs a verified contested admission it stands one in; unit tests pin that a real verification records it.
  • Mutations: 54 mutations of the floor, the contest decision, the verifier's memory, the mark, its in-memory copy and every check that reads the mark, run on 375d62f against the unit and e2e tests above: 53 fail at least one test. The one that survives makes prune-audit scoring count a mark that cannot be read as unset. Only the e2e prune test reaches that check, and it cannot make a node's paid-list read fail. Two of the 54 targeted the paid-pruning rule feat(replication)!: erase out-of-range records only under disk pressure #271 made unnecessary, which is gone. Five more were run on 8d12c17, whose code is the head's, one for each check the merge added or ported, and each fails: an eviction that leaves an evicted entry's mark in memory fails four unit tests; leaving far stored marked keys out of paid hints, treating unknown marks as no marks, and keeping only readable records' entries through eviction each fail their unit test; and prune-audit scoring that ignores the mark fails the e2e prune test.
  • cargo test --test poc_price_floor_live --features test-utils: 2/2 against a real Anvil vault. On main it is 2 of 2 with the stale test ignored.
  • cargo test --test e2e --features test-utils -- --test-threads=1: 126 passed, 3 ignored (main's live_testnet tests), on 8d12c17.

The head, 06eea79, is the previous head 16e1fc7 (five commits on main at 02c8682, reviewed before and unchanged), a merge of main at f37429a, efa4a57, and ten commits after it. The merge brings in pressure-gated retention (#271, ADR-0028), which rewrote the paid list, pruning and paid hints the mark hooks into: the mark, its in-memory copy and its readers now sit on #271's paid list; the rule that kept a marked entry through paid pruning is gone with that pruning, since eviction keeps the entry of every record still stored; prune audits still score no one for a marked key, and the e2e test of that now runs under disk pressure; and tests/poc_price_floor_live.rs keeps this PR's repair, not main's ignore of the stale test. After it, ce05871 pins that eviction takes an evicted entry's mark, and 813b103 brings ADR-0006 in line with ADR-0028. Reviews of the merged code then found where the two designs meet. Three fixes each come with a test that fails without them: ada05dd makes a stored marked key a paid-hint candidate at any distance, since above 100,000 paid entries #271 reads only near keys for paid hints and such a record would otherwise go out in neither hint set; 7bbe0e1 does the same for every stored key while the marks are unknown; and 6051640 makes eviction keep the entry of any record still in the store's index, since a read fault turns exists false and could cost a retained record its mark. Two more bound the cost of the first two: 04513a8 finds far marked keys among the stored keys instead of copying every mark on each hint build, and 7b66a6c sends those far paid hints only to the close group a replica hint would reach, so a sync message grows by no more than replica hints of the same keys would make it. Their unit test checks which keys are read, not which peers they go to, since that needs a routing table. e9728f5, 8d12c17 and 06eea79 change only ADR-0006: they record the repair-source limit below and the final mutation count.

Gates, on 8d12c17, whose code is the head's (06eea79 changes only ADR-0006's mutation count), with CI's RUSTFLAGS="-D warnings", on Rust 1.99.0 on macOS: main's consolidated CI apart from its Linux-only steps, the run-directory tests under sudo, storage_scale and the storage jobs on ext4, xfs and btrfs, which ran in hosted CI only:

  • cargo fmt --all -- --check;
  • cargo doc --all-features --no-deps with -D warnings;
  • cargo check --lib --no-default-features --locked;
  • cargo +1.95.0 check --all-targets --all-features --locked (MSRV);
  • cargo build --release, and the shipped-binary failpoint check on it;
  • cargo test --doc --all-features: 1 passes, 2 ignored;
  • every integration target, each with --features test-utils -- --test-threads=1: e2e as above; build_script_git_watch 3; chunk_store_crash_safety 2 (its 1 ignored test is the crash test's child process); devnet_lifecycle 3; poc_audit_handler_live 16; poc_bootstrap_stall 3; poc_commitment_audit_attacks 19; poc_d1_bounded_queues 7; poc_price_floor_live 2, against a real Anvil vault; pointer_convergence 17; webrtc_direct_devnet 6, and its ignored five-node devnet test 1;
  • scripts/adr-governance.py against main, also on the head;
  • clippy: hosted Quick checks pass. Locally on macOS, the local stable (1.98) cargo clippy --all-targets --all-features lints every target and reports one warning, unnecessary_wraps on group_writers in src/devnet/run_dir.rs (main's fix(ant-devnet): stop cleanly on SIGTERM and use a per-run data dir by default #263; on Linux that function can return an error). -D warnings makes it the one error of cargo +1.99.0 clippy --all-targets --all-features -- -D warnings, and cargo clippy -- -D warnings also flags main's unused_self on skips_closeness_without_p2p in src/payment/verifier.rs. src/devnet/run_dir.rs is byte-identical to main, and that function is identical to main's; this PR changes other parts of verifier.rs.

cargo audit reports one vulnerability, RUSTSEC-2026-0387 (crypto-bigint 0.5.5, reached through alloy's k256 and webrtc's dtls, with no fixed release of either yet), and the same 6 allowed warnings as before. main at f37429a has the same Cargo.lock, which this PR leaves unchanged. So the hosted Security Audit check fails here on the same vulnerability, as on every PR; since #269 it sits outside the required CI Gate, until those dependencies move or the maintainers decide on an ignore, a decision that is open and not taken here.

Hosted checks on 06eea79: 15 of 16 pass, the required CI Gate among them (CI run 41981942807, plus the ADR governance and PR-check runs): Quick checks, Rust 1.95 MSRV, Third-party notices, Test on Ubuntu, macOS and Windows, Storage on ext4, xfs and btrfs, Build release (x86_64-unknown-linux-musl), ADR validation, linear-link, pr-template and the PR-check self-test. Security Audit fails on RUSTSEC-2026-0387 alone, with the same six allowed warnings, as cargo audit reports locally; RUSTSEC-2026-0382 and 0383, which the last formal review named, are fixed on main by #258.

The dev-testnet run that T3 asks for has not happened yet.

Residual risks, also recorded in the ADR.

  • Rollout skew. Until the fleet upgrades, an older node admits below-floor writes and still runs possession checks on them. Each one penalises every upgraded peer that refused it: weight 5 for a chunk, which takes a neutral peer from 0.5 to about 0.26 in that node's view, below the 0.35 swap threshold, for about 9.5 hours (weight 1 for a pointer). The score is local (for peers in that node's persisted close group it survives a restart), and below 0.35 a peer only becomes replaceable when that node's routing bucket is full. An attacker can buy such writes for little more than gas by paying its own near-empty node within the 20 closest to the key, and aim them at old nodes. Keys an older node stored this way stay on it unmarked, before and after it upgrades, so its replica hints and responsible and prune audits keep judging the refusers on them while it holds them, and the refusers can penalise it at weight 1. On the v0.17.0 rollout the managed fleet went from 24 of 73 hosts in the first hour to 65 in six and 71 within about 17; community nodes follow on their own schedule.
  • The contest margin is empirical. It reads the uncapped median, so neighbours that overstate their commitments can widen the set of contested writes in their group, and turn checks off there. A minority lifts the median only as far as the dearest honest members, and overstating needs fake leaves that fail the commitment spot-checks (ADR-0004). Seven of fifteen understating neighbours at the storer narrow it instead, to about 0.33x of the honest median, the same minority that disarms the floor.
  • Contested keys lose coverage. The node that marked a key does not audit peers on it, and the key repairs through paid hints only. That is 1.4% of the 1,801 measured admissions, and separately every write the floor cannot price (2.6% of the 2,117 floor evaluations), and the mark stays while the node holds the key, so it also covers every key a node records while it cannot price: its first hour before a commitment, or while its gossip view refills after a restart. The other holders of those keys still check them, unless a quoter front-ran the mark (below).
  • Gaps on nodes running this release. A key repaired from a majority's paid-list votes arrives unmarked, since repair carries no proof to price; a refuser whose own quorum check fails while its neighbours' passes can be judged on it. On the PUT path the record is stored a moment before its paid-list entry is written, so a sync in that moment can hint the key once and a refuser can charge the storer weight 1 once, and a crash in that moment leaves the key unrecorded, so unmarked unless a later verified PaidNotify records it. A PUT served from the cache with no proof attached stores the record without any paid-list entry, as before. A paid-list insert that fails before its commit leaves the key out of the paid list, so unmarked. When a marked insert's commit is what failed, the in-memory copy keeps the mark, since the commit may still have reached the disk, so this node does not hint the key as a replica while the mark stays. A later plain insert that adds the key, or a removal of it, drops the mark once that write commits, and a restart rebuilds the copy from the database. The verifier's memory of contested admissions loses a decision before its key is recorded only if 100,000 newer contested admissions arrive in between. A record admitted between paid-list eviction's scan and its delete would lose its entry and so its mark; eviction takes only entries far outside the node's range, where with evenly spread IDs neither store path admits a record.
  • The first paid-list record decides a key's mark. A quoter in the 20 closest can pay itself for an honest key, for gas only, and announce it before the client's write lands. At a price between the floor and the contest threshold, or at any price on nodes that cannot price, the nodes it reaches first leave that key unchecked, though they still store and repair it; those among the client's storers skip their possession check too, while storers it did not reach first still run theirs. Paying above the threshold first does the opposite: those nodes record the key unmarked and can judge peers that later refuse the client's cheaper write, until those peers repair it from the votes the front-run created.
  • A marked copy outside a key's queried groups is not a repair source. A paid hint makes no possession claim. A node repairing a key fetches from the members of its close group and paid close group that answer present, and from replica hinters, so if none of those holds the key and every other copy is marked on its holder, the key cannot be repaired from those copies. They stay on disk, since deletion still needs the group's proofs. Fixing that without exposing the holder to sole-hinter penalties needs a wire change.
  • Opening the paid list reads every entry once, to fill the in-memory marks, and the node starts after that read. feat(replication)!: erase out-of-range records only under disk pressure #271 lets the list hold 15 million entries before it evicts; a local probe read 4 million in 85 ms with a warm page cache, about a third of a second at that size, plus reading the file on a cold start.
  • No per-node switch. Backing the floor out takes a release, so the release train's canary is where it is stopped.
  • One-quote clients. An honest client whose only reachable quoter is a fresh node is refused. No case was observed.
  • Proof reuse. A proof reused for up to a day is covered only against the uncapped median, where the worst measured payment has 1.58x of headroom.
  • Retry load is new. Refusals are not cached. A refused proof is re-verified, on-chain read included, by every peer a client tries: up to 20 targets over 4 rounds.
  • Merkle batches bypass the floor. Under the default client mode those are uploads of 64 chunks or more. The floor covers single-node payments only.

New dependency

none

ADR

https://github.com/grumbach/ant-node/blob/06eea7957f01c973f2c52554651e3eda7a91daab/docs/adr/ADR-0006-receiver-side-revenue-floor.md#amendment-2-2026-10-20-enforced-on-every-node (amends ADR-0006, which is still Proposed, and updates ADR-0008's status row)

Mitigation / rollback

There is no per-node switch: reverting this PR in a release restores shadow mode. Older releases read the paid-list marks as ordinary paid keys, so a rolled-back node judges peers on those keys again, as it did before.

@grumbach grumbach changed the title feat(payment)!: enforce the close-group price floor by default feat(payment)!: enforce the close-group price floor on every node Oct 20, 2026
@grumbach
grumbach force-pushed the price-floor-live-enforcement branch 9 times, most recently from f17cb86 to bc714ae Compare October 21, 2026 06:38

@dirvine dirvine left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded: original blockers withdrawn

My original REQUEST_CHANGES on bc714aebfae92f6b68f1bb8847f8328e052ea2df overstated the two blocking findings and claimed invalid panel participation. See the corrected, provisional APPROVE review for the reconciled verdict, verified tests, actual panel coverage and remaining human release/strategy decisions. This is a correction of my review, not a change to the code. The original receipt is preserved locally; this older review's historical GitHub state is not its current recommendation.


Corrected code-review verdict: APPROVE (provisional coverage)

This supersedes my earlier REQUEST_CHANGES review on the same code. The late source-review batch prompted a reconciliation. I overstated two findings as blockers. Baseline comparison, a component timing probe, and bounded adversarial reconciliation do not support retaining those blockers. No branch changes were needed for this correction.

Repository: WithAutonomi/ant-node. Reviewed head: bc714aebfae92f6b68f1bb8847f8328e052ea2df. True merge-base: a6e3388e79d0907ef678bcb6ec0fcaa12822e148, not current main.

Findings corrected

  1. Extra synchronous scan: performance follow-up, not a demonstrated blocking DoS. src/replication/neighbor_sync.rs:93-100 really adds price_contested_keys() on the async executor. However, the baseline already synchronously scans the full paid-list via all_keys() on the same sync-request path. A scratch-only probe linked the exact frozen library, preseeded LMDB, and verified both key counts. At 1,000,000 entries the additional scan's median was 12.156 ms with no marks / 71.591 ms with every key marked; the existing scan was 14.940 / 15.510 ms. These are three warm-cache runs per case with the debug-linked library, not release-build, fleet-load, starvation or attack results. The added cost is real and should be reduced or offloaded, but I did not establish the blocking severity claimed earlier.
  2. Audit read-error telemetry: operational follow-up. src/replication/audit.rs:103-112 intentionally treats an unreadable mark as set and skips judgement. The missing local log/metric is real; the fail-safe direction is correct. No paid-list failure-injection experiment was run. Missing observability alone does not justify the blocking correctness claim I made.
  3. First-record-wins is deliberate, not a missing upgrade. ADR-0006:587-597 records the front-running trade-off. Unconditionally marking an existing unmarked key on a later cheap payment would allow audit suppression for gas. Retain the disclosed residual; do not present that upgrade as a safe fix.

Participation, dissent and coverage limits

  • Original full-production first pass: GLM 5.3 requested changes; Qwen 3.8 27B approved. DS4's first response truncated; its sole correctness retry was labelled with a malformed 39-character OID, so it is not a valid exact-revision correctness receipt. My original public body also contained that malformed OID, although GitHub attached the review to the correct commit shown above.
  • Adversarial reconciliation of the disputed claims: GLM 5.3, Qwen 3.8 27B and DS4 Flash all returned substantive APPROVE responses. GLM withdrew its blocking interpretation after the baseline/measurement comparison. GLM's first reconciliation request timed out and DS4's first reconciliation response truncated; each received one narrowed same-model retry. These are clarifications/retries, not extra votes.
  • Qwen's adversarial pass included the full production delta; the successful GLM and DS4 reconciliation retries were focused packets. There are five valid model-role responses: three full-delta responses and two focused reconciliations, not six complete full-scope roles. Full-production first-pass participation is 2/3; whole-PR coverage remains provisional. No 6/6 or full whole-PR consensus is claimed. DS4's missing full correctness coverage is not silently filled by its focused answer.
  • The two ordinary delegated source-tracing reviews found no concrete implementation blocker. They are not additional named-panel votes, and neither independently ran the tests.

Verification actually executed at this head

  • cargo fmt --check: exit 0.
  • cargo test --locked --lib --features test-utils -- --test-threads=4: 1,202 passed, no failures.
  • cargo test --locked --test e2e --features test-utils -- --test-threads=1: 124 passed, 3 ignored, no failures.
  • cargo test --locked --test poc_price_floor_live --features test-utils -- --test-threads=1: 2 passed, no failures. These use real local Anvil settlements, but injected routing/gossip; they are not mixed-version production tests.
  • The opt-in production-wired price_floor_ab workload and the author's telemetry/mutation campaign were not independently reproduced.
  • CI at this exact head has a failed Security Audit for the inherited rustls-webpki advisory; Cargo.lock is unchanged from the merge-base. Other reported checks succeeded, apart from the skipped claude job. I did not enumerate branch-protection requirements, so the earlier claim that “all required checks” passed was unsupported.

Human release/strategy decisions remain open

The 30% constant is an empirical policy, not proof that all honest traffic clears the floor. My earlier extrapolation from a different shadow sample was too broad: that sample does not establish admission safety across changed views, lone fresh-node quotes, or aged proofs.

ADR-0006:598-619 explicitly describes old binaries penalising upgraded refusers, including marks that survive upgrade; new binaries' contested filtering does not protect old auditors. The operator override is removed by design. Disclosure in a Proposed ADR is not human acceptance of this release strategy. That risk and the unmeasured one-quote liveness residual need the proper human sign-off. This provisional implementation approval does not authorise acceptance of the ADR, a strategy decision, merge, release or deployment.

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected code-review verdict: APPROVE (provisional coverage)

This supersedes my earlier REQUEST_CHANGES review on the same code. The late source-review batch prompted a reconciliation. I overstated two findings as blockers. Baseline comparison, a component timing probe, and bounded adversarial reconciliation do not support retaining those blockers. No branch changes were needed for this correction.

Repository: WithAutonomi/ant-node. Reviewed head: bc714aebfae92f6b68f1bb8847f8328e052ea2df. True merge-base: a6e3388e79d0907ef678bcb6ec0fcaa12822e148, not current main.

Findings corrected

  1. Extra synchronous scan: performance follow-up, not a demonstrated blocking DoS. src/replication/neighbor_sync.rs:93-100 really adds price_contested_keys() on the async executor. However, the baseline already synchronously scans the full paid-list via all_keys() on the same sync-request path. A scratch-only probe linked the exact frozen library, preseeded LMDB, and verified both key counts. At 1,000,000 entries the additional scan's median was 12.156 ms with no marks / 71.591 ms with every key marked; the existing scan was 14.940 / 15.510 ms. These are three warm-cache runs per case with the debug-linked library, not release-build, fleet-load, starvation or attack results. The added cost is real and should be reduced or offloaded, but I did not establish the blocking severity claimed earlier.
  2. Audit read-error telemetry: operational follow-up. src/replication/audit.rs:103-112 intentionally treats an unreadable mark as set and skips judgement. The missing local log/metric is real; the fail-safe direction is correct. No paid-list failure-injection experiment was run. Missing observability alone does not justify the blocking correctness claim I made.
  3. First-record-wins is deliberate, not a missing upgrade. ADR-0006:587-597 records the front-running trade-off. Unconditionally marking an existing unmarked key on a later cheap payment would allow audit suppression for gas. Retain the disclosed residual; do not present that upgrade as a safe fix.

Participation, dissent and coverage limits

  • Original full-production first pass: GLM 5.3 requested changes; Qwen 3.8 27B approved. DS4's first response truncated; its sole correctness retry was labelled with a malformed 39-character OID, so it is not a valid exact-revision correctness receipt. My original public body also contained that malformed OID, although GitHub attached the review to the correct commit shown above.
  • Adversarial reconciliation of the disputed claims: GLM 5.3, Qwen 3.8 27B and DS4 Flash all returned substantive APPROVE responses. GLM withdrew its blocking interpretation after the baseline/measurement comparison. GLM's first reconciliation request timed out and DS4's first reconciliation response truncated; each received one narrowed same-model retry. These are clarifications/retries, not extra votes.
  • Qwen's adversarial pass included the full production delta; the successful GLM and DS4 reconciliation retries were focused packets. There are five valid model-role responses: three full-delta responses and two focused reconciliations, not six complete full-scope roles. Full-production first-pass participation is 2/3; whole-PR coverage remains provisional. No 6/6 or full whole-PR consensus is claimed. DS4's missing full correctness coverage is not silently filled by its focused answer.
  • The two ordinary delegated source-tracing reviews found no concrete implementation blocker. They are not additional named-panel votes, and neither independently ran the tests.

Verification actually executed at this head

  • cargo fmt --check: exit 0.
  • cargo test --locked --lib --features test-utils -- --test-threads=4: 1,202 passed, no failures.
  • cargo test --locked --test e2e --features test-utils -- --test-threads=1: 124 passed, 3 ignored, no failures.
  • cargo test --locked --test poc_price_floor_live --features test-utils -- --test-threads=1: 2 passed, no failures. These use real local Anvil settlements, but injected routing/gossip; they are not mixed-version production tests.
  • The opt-in production-wired price_floor_ab workload and the author's telemetry/mutation campaign were not independently reproduced.
  • CI at this exact head has a failed Security Audit for the inherited rustls-webpki advisory; Cargo.lock is unchanged from the merge-base. Other reported checks succeeded, apart from the skipped claude job. I did not enumerate branch-protection requirements, so the earlier claim that “all required checks” passed was unsupported.

Human release/strategy decisions remain open

The 30% constant is an empirical policy, not proof that all honest traffic clears the floor. My earlier extrapolation from a different shadow sample was too broad: that sample does not establish admission safety across changed views, lone fresh-node quotes, or aged proofs.

ADR-0006:598-619 explicitly describes old binaries penalising upgraded refusers, including marks that survive upgrade; new binaries' contested filtering does not protect old auditors. The operator override is removed by design. Disclosure in a Proposed ADR is not human acceptance of this release strategy. That risk and the unmeasured one-quote liveness residual need the proper human sign-off. This provisional implementation approval does not authorise acceptance of the ADR, a strategy decision, merge, release or deployment.

@grumbach
grumbach force-pushed the price-floor-live-enforcement branch 2 times, most recently from 297f587 to 9c2d279 Compare October 21, 2026 11:36

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — re-review of 9c2d2798c2c78d9af5c756efcc877a55b7a83629

Coverage: 1/1 seat reviewed (sol-1, gpt-6.1-sol, xhigh). This is a focused re-review of the delta since bc714aebfae92f6b68f1bb8847f8328e052ea2df; the previous review did not cover this commit. The seat approved but flagged cache-only marks after failed commits. Source tracing confirms that a later successful unmarked insert can make this a persistent cache/database inconsistency.

P2 — Reconcile cache-only marks after successful unmarked writes

src/replication/paid_list.rs:268-274 adds a key to the in-memory mark set before committing a marked insert, then deliberately retains it if the commit returns an error. If that commit did not persist, a later plain insert(key) sees no LMDB row and successfully writes an empty value, but this branch does not remove the cached mark. retain_unmarked then suppresses the now-unmarked key from replica hints, while is_price_contested and the audit paths read the empty LMDB value and treat it as unmarked. That breaks the first-record-wins/cache-consistency invariant and can persist until restart.

After a successful plain insert commit, remove that key from the cached set while still holding write_order; keep the conservative cached mark when the commit fails. Also reconcile cached-only keys on successful batch removal: remove_batch currently removes from the set only keys LMDB reports as deleted (src/replication/paid_list.rs:628-634). Add regression tests for a cache-only mark followed by a successful plain insert and by a successful batch removal of an absent key.

The hosted Security Audit failure is not attributed to this delta: Cargo.lock is unchanged from the merge-base. This code review does not accept the Proposed ADR or authorise merge, release or deployment.

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — validation gates remain

Coverage: 1/1 seat reviewed (sol-1, gpt-6.1-sol, xhigh). The focused re-review covers head 0425e1e07f1a2b93228532df0dd1e5fa659f0fc1. The earlier seat response was uncertain; its single narrowed retry approved the code change. Source inspection confirms the reported stale-mark paths are addressed. I found no new code defect in this delta.

This request is for outstanding required validation, not a code finding:

  1. T3 dev-testnet evidence is still missing. The PR marks this as T3 and says the required dev-testnet run remains outstanding. The successful hosted “WebRTC Direct five-node devnet” check is not that run. Please attach the required dev-testnet result.
  2. Security Audit is failing on this head. The log reports RUSTSEC-2026-0382 (rustls-webpki 0.103.15, upgrade to at least 0.103.16) and RUSTSEC-2026-0383 (webpki-root-certs 1.0.9, upgrade to at least 1.0.10). Please resolve through the dependency/lockfile update and rerun this check, or provide the repository-approved disposition for this inherited advisory gate.

The targeted local paid-list test did not finish: compilation exceeded the available run window and was terminated before test output. I do not claim it passed. The code opinion is based on the reviewed source and the seat receipt.

@grumbach
grumbach force-pushed the price-floor-live-enforcement branch from 0425e1e to 668f573 Compare October 22, 2026 10:25
The ADR-0006 floor has run in shadow mode since v0.17.1. A week of
production telemetry (2026-10-14..20, 699 node instances on 72 hosts)
showed that its 65% default would have refused 2.5% of admissions, and
2.1% of chunks at every receiver that saw them. Neighbours gossiping
inflated commitments could refuse a chunk everywhere. One accepting node
could still replicate a below-floor write through PaidNotify and
paid-list repair. And a peer that refused a write on price would be
penalised for it by possession checks, responsible audits and prune
audits. This makes enforcement safe and turns it on for every node, with
nothing to configure.

- Cap the reference at the receiver's own price: min(group median,
  calculate_price(own committed count)). A node never demands more than
  the tolerance of what it would itself quote, so lying neighbours
  cannot make it refuse a payment that clears its own price.
- Lower the tolerance from 65% to 30%, a deliberate starting point. No
  measured admission is refused at 30% or at 40%, but the worst clears
  30% by 1.78x and 40% by only 1.34x, on a week of 48 upload bursts. A
  baseline quote is still refused by 91% of receivers (98% at 40%).
- Floor paid-list admission too. PaidNotify only announces a client's
  write, with the proof the client presented, so this reprices no
  historical receipt. It stops one accepting node from giving a
  below-floor write the paid-list votes that repair needs.
- Judge no peer on a contested write. A payment is contested when the
  node cannot price it, or its paid price is below 60% of the uncapped
  group median (twice the floor). The verifier remembers each contested
  chunk payment it admits, in a memory of its own, and every record path
  (PUT, fresh offer, PaidNotify) marks the key's paid-list entry from
  that decision, never from the unchecked proof a cache hit carries; the
  first paid-list record of a key decides. A marked key goes out in paid
  hints only, so no repair proof is recorded, no responsible audit
  samples it and no peer penalises this node as its sole hinter; the
  paid list keeps its marked keys in memory too, so building hints
  scans nothing on disk for them. A prune audit still needs its proofs
  but scores no absent peer; paid-list pruning keeps the marked entry
  while the record is held; and a mark that cannot be read counts as
  set, and is logged. A chunk fan-out schedules no
  possession check when the write is marked (by the verified decision
  or by the record that won, read after its own is written) or the
  proof it forwards is contested, and a pointer fan-out none when its
  proof is contested. Releases before this one only test paid-list
  presence, so they read a marked key as paid. The 2x margin covers the
  largest spread measured between receivers' views (1.91x); it is a
  margin, not a guarantee.
- One floor for every node. PriceFloorConfig and
  PaymentVerifierConfig::price_floor are removed: the tolerance is the
  constant PRICE_FLOOR_TOLERANCE_PERCENT and the contest threshold a
  private one. ANT_PRICE_FLOOR_ENFORCE and
  ANT_PRICE_FLOOR_TOLERANCE_PERCENT are no longer read, and backing the
  floor out takes a release.
- The telemetry line adds target, group_median_price, own_price,
  paid_quote_age_secs and contested, and keeps enforce=true constant.
- A comment on the group sample says why it needs no self-filter: the
  routing scan includes this node, but the gossip cache never holds it,
  since ingest accepts only senders in the routing table, so its own
  commitment joins the sample exactly once.
- Repair tests/poc_price_floor_live.rs, which no longer exercised the
  close-group floor and failed on main. Pin the chunk and pointer
  possession schedulers, the mark and hints, the prune audit and all
  three record paths with e2e tests, each with a control. Add an
  env-gated A/B workload driver, tests/e2e/price_floor_ab.rs.

ADR-0006 gains Amendment 2, recording the measurement, these decisions
and the residuals. ADR-0008's status row is updated.

BREAKING CHANGE: PriceFloorConfig (with from_env, enforce and
tolerance_percent), PaymentVerifierConfig::price_floor,
PRICE_FLOOR_ENFORCE_ENV and PRICE_FLOOR_TOLERANCE_ENV are removed, and
PRICE_FLOOR_TOLERANCE_PERCENT is added. The floor is enforced at 30% on
every node. neighbor_sync::build_replica_hints_for_peer and
audit::audit_tick_with_repair_proofs take the paid list, and
ReplicationEngine::replicate_fresh is only built with test-utils.
…clears

The paid list keeps its price-contested keys in memory, so building
replica hints reads nothing on disk. A marked insert puts its key in that
copy just before its commit and keeps it if the commit fails, since the
commit may still have reached the disk. When it had not, the copy held a
mark the database lacked, and nothing removed it: a later plain insert of
the key wrote an unmarked entry, and a batch removal cleared only the keys
the database reported deleted. Replica hints then left out a key that
every point read, the audit included, treated as unmarked, until a
restart.

A plain insert that adds the key now drops any in-memory mark for it once
its commit succeeds, still under the write order, and a removal, single or
batched, drops the mark of every key it names, held or not. A failed
commit still leaves the copy as it was, so the copy stays a superset of
the committed marks, and equals them for every key a successful write
touched.

Tests mark a key in memory only, as such a failed commit leaves it, and
check that a plain insert, a batch removal and a single removal of the
key each clear it, while a key whose entry carries the mark keeps it.
The mutation run on the price-floor code now covers the in-memory mark
reconciliation too: 54 mutations, of which 53 fail at least one unit or
e2e test. The one survivor is unchanged: prune-audit scoring reading a
mark that cannot be read as unset, which only the e2e prune test reaches,
and that test cannot make a paid-list read fail.
ADR-0006's validation list said the contest decision follows the view
when a write is recorded, and the unit test pinning it said the same.
That holds for the proof a fan-out forwards, chunk or pointer, which is
priced against the view when the write is fanned out. The paid-list mark
a chunk is recorded with comes from the decision its verification made,
which the verifier remembers, not from the view at record time. Both
texts now say so. No code changes.
The paid-list helper that holds the write order spelled out
tokio::sync::oneshot and tokio::task::JoinHandle inline, and the
verifier test for a node with no floor wired built its gossip cache as
tokio::sync::RwLock inline. The paid-list tests now import oneshot and
JoinHandle, the helper waits for the order with an assertion instead of
expect, and the verifier test uses CommitmentCache::default(). No
behaviour changes.
@grumbach
grumbach force-pushed the price-floor-live-enforcement branch from 668f573 to 16e1fc7 Compare October 29, 2026 01:41
Brings the branch up to main f37429a. The conflicts came from
pressure-gated retention (WithAutonomi#271, ADR-0028), which rewrote the paid list,
pruning and paid hints that the price-contested mark hooks into:

- paid_list.rs: the mark, its in-memory copy, the write order and the mark
  reads now sit on main's paid list, with its saved record clocks, its
  MapFull reporting (a marked insert reports a full store the same way)
  and its range reads. The paid out-of-range timers and the prune cursor
  stay removed. Eviction goes through remove_batch, which already drops
  the in-memory mark of every key it was asked to remove.
- pruning.rs: paid pruning by rank is gone, so the rule that kept a
  price-contested entry through it while its record was held goes too:
  eviction keeps the entry of every record still stored, marked or not.
  Prune audits still score no absent peer for a price-contested key.
- neighbor_sync.rs: main's paid-hint reads and the replica-hint filter
  for marked keys are both kept.
- tests: the e2e test that a prune audit scores no one for a
  price-contested key now runs its passes under disk pressure, since
  without it no prune audit is sent. The price-floor PoC keeps this
  branch's rewrite, which attaches a commitment source, rather than main's
  ignore of the old test that had none.
…h its entry

Paid-list eviction (ADR-0028) removes entries through remove_batch. The
new test marks two far entries, evicts one and keeps the other, and
checks that the evicted entry's mark leaves the in-memory copy while the
kept entry stays marked on disk and in memory, so no mark is held in
memory alone after eviction.
ADR-0028 replaced paid-list pruning by rank with eviction above a
threshold, and sends prune audits only under disk pressure. Amendment 2
described the mark against the old pruning, so it now says:

- paid-list eviction keeps the entry of every record still held, marked
  or not, and an evicted entry takes its mark with it; the narrow gap
  where a record admitted mid-pass loses its mark is restated for
  eviction;
- prune audits, and the e2e test that a marked key's audit scores no one,
  run under disk pressure;
- of the 54 mutations run before the merge, two targeted the paid-pruning
  rule that is now gone, and two more run on the merged code both fail;
- opening the paid list reads every entry once to fill the in-memory
  marks, which a list at the 15 million entry eviction threshold makes
  about a third of a second with a warm page cache.
A key marked price-contested goes out as a paid hint only, never as a
replica hint (ADR-0006). Since pressure-gated retention (ADR-0028), a
node with more than 100,000 paid entries builds paid hints only from keys
near itself, while replica hints have no distance limit. So a marked
record it retained far from itself reached its close group in neither
hint set, where an unmarked one still goes out as a replica hint.

Paid-hint candidates now also take every marked key the node stores and
still holds a paid entry for, however far it is. The marks and the
store's index are in memory, so each far marked key costs one paid-list
read. build_paid_hints_for_peer takes the chunk store for this, PaidList
gains marked_keys and paid_list gains is_near. ADR-0006 says so, and
lists the new unit test.
… are unknown

When the price-contested marks cannot be read as the paid list opens,
the node hints no key as a replica, since any of them may be marked.
Before ADR-0028 its paid hints still read every paid entry, so its stored
records reached their close groups that way. Above 100,000 entries paid
hints now read only near keys, so in that state a record the node
retained far from itself went out in neither hint set.

While the marks are unknown, every key the node stores that has a paid
entry is now a paid-hint candidate at any distance, as a marked key is
when the marks are known. That moves the per-key work the replica hints
no longer do onto the paid hints. PaidList::marked_keys says when the
marks are unknown, and the unit test covers that state. ADR-0006 says
so.
… for now

Paid-list eviction (ADR-0028) kept an entry only while ChunkStore::exists
held for its record, and exists turns false while a record is out of
service after a failed read, although its file stays in the index. An
eviction pass in that moment removed the entry and, with it, the
record's price-contested mark (ADR-0006). Once the record read again it
went out as an unmarked replica hint, so peers that had refused it under
the floor could be judged on it.

Eviction now keeps an entry while its record is in the store's index,
readable or not, through the named keeps_paid_entry. It only keeps more
entries than before. A unit test makes a record unreadable and checks
that its entry stays while an entry with no record does not. ADR-0006
says so.
…ery mark

The far price-contested keys a hint build adds to its paid hints were
found by copying every mark and checking each against the store. Marks
also sit on paid entries whose records this node never stored, and stay
until eviction, so with millions of them every hint build copied and
checked them all, however few records the node holds.

A hint build now reads the stored keys once, as it already did for
replica hints, and splits them against the in-memory marks: the unmarked
ones become replica-hint candidates, the marked ones far paid-hint
candidates. While the marks are unknown every stored key counts as
marked. The inbound sync handler builds both hint sets from that one
read. PaidList::split_marked replaces marked_keys.
A paid hint makes no possession claim, so a node holding a marked record
is never asked for its bytes. If a key's current close group holds no
copy and every copy outside it is marked, the group cannot repair the key
from them. The amendment's consequences now say so, and that fixing it
without exposing the holder to sole-hinter penalties needs a wire change.
…nt would go

A far price-contested key this node stores went out as a paid hint to
every member of its paid close group it knows (20 peers), where a replica
hint of the same key reaches its close group (7). So turning such keys
from replica hints into paid hints could grow a sync message about three
times past what main sends, in the worst case past the 10 MiB message
limit, and the whole exchange would be dropped.

Paid-hint candidates now keep the near keys read from the paid list apart
from the far marked keys: the first still go to the paid close group, the
second only to the close group, so they add no more to a message than
replica hints of the same keys would. build_paid_hints_for_peer takes the
close group size for this. ADR-0006 says so.
… the queried groups

Verification asks a key's close group and its paid close group, and
takes every member that answers present as a fetch source, marked or
not. Only copies outside both groups, known to repairers through paid
hints alone, are never fetched from. The consequence now says so.
Five targeted mutations were run on the code merged with ADR-0028, one
for each check the merge added or ported, and each fails its test. The
validation list now names all five instead of the first two.

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — eviction race and open release gates

Repository: WithAutonomi/ant-node. Reviewed head: 06eea7957f01c973f2c52554651e3eda7a91daab.

Coverage: 2/2 seats reviewed (sol-1, gpt-6.1-sol xhigh; glm-1, z-ai/glm-5.3). GLM approved. Sol-1 remained uncertain after its one narrowed retry and identified the eviction race below. I checked both concerns against source: the alleged mark-set deep copy is not real (price_contested stores an Arc<HashSet<_>>, so the snapshot clones the Arc); the race is real.

P2 — Coordinate paid-entry eviction with record admission

src/replication/paid_list.rs:619-654 scans candidates and calls the keep closure under a read transaction, then releases that scan and calls remove_batch in a separate write transaction. The eviction closure in src/replication/pruning.rs:699-706 checks storage.is_indexed(key) only during the scan.

A concurrent fresh-offer path in src/replication/mod.rs:6623-6635 records the paid entry (including its contested mark) and then stores the record. If eviction sees the key as not indexed, the fresh offer can then commit the marked entry and record before remove_batch runs. That later delete removes the entry and in-memory mark while the record remains stored. Hint construction can then treat the record as unmarked and send a replica hint, exposing peers that refused the contested write to possession/audit penalties.

The ADR discloses this scan-to-delete race, but its rarity argument does not prevent the execution path or establish human acceptance. Coordinate admission with eviction so a record and its mark cannot be orphaned by a stale eviction candidate; add a deterministic interleaving regression test. A second uncoordinated is_indexed check alone would leave another race window.

Required T3 and strategy gates

The required T3 dev-testnet result and human sign-off on the rollout-skew and one-quote residuals are still outstanding. Please attach the testnet evidence and record the authorised sign-off before approval. These are open release/strategy gates, separate from the code finding.

The Security Audit still fails on RUSTSEC-2026-0387, but the current CI Gate succeeds and the base/head Cargo.lock objects match; I am not treating that shared advisory as a blocker for this review. Hosted test checks on this head succeed. I did not run the local test suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants