Skip to content

Keep replaced withdrawals pending until their RBF replacement confirms - #642

Open
markettes wants to merge 13 commits into
mainfrom
fix/failed-rbf-original-confirmed
Open

markettes wants to merge 13 commits into
mainfrom
fix/failed-rbf-original-confirmed

Conversation

@markettes

@markettes markettes commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

A withdrawal request's status now only describes its own transaction. Creating, executing or abandoning an RBF replacement never writes the request it replaces; the replacement is reachable through the existing BumpingWalletWithdrawalRequestId link, only a confirmation changes a chain, and Bumped becomes terminal: written only by the monitor once a replacement confirms. The API reports the stored status and nothing derived, so WITHDRAWAL_SETTLED, WITHDRAWAL_BUMPED, WITHDRAWAL_CANCELLED and WITHDRAWAL_FAILED are final; while a chain is in flight a client follows replaced_by_request_id.

Why: Bumped meant two things at once ("replacement in flight" and "superseded, done"). A bumped original was not polled, so its confirmation was only noticed through the replacement's row; whenever the replacement was abandoned without a revert (RPC MarkWithdrawalRequestCancelled, the UI's dismissed confirmation modal, a failed revert write) the original sat Bumped forever with a transaction that could still confirm, by-id clients saw WITHDRAWAL_BUMPED forever, and its inputs stopped counting as locked. The revert itself was a cross-row write with a race against the broadcast.

Event Original in DB Original by gRPC id Replacement
Replacement created (cold: awaiting approval) OnChainConfirmationPending WITHDRAWAL_PENDING_CONFIRMATION, replaced_by_request_id set live
Replacement broadcast untouched WITHDRAWAL_PENDING_CONFIRMATION, replaced_by_request_id set OnChainConfirmationPending
Replacement abandoned untouched WITHDRAWAL_PENDING_CONFIRMATION, no next hop terminal
Replacement confirms Bumped / Cancelled + reason, by the monitor WITHDRAWAL_BUMPED / WITHDRAWAL_CANCELLED OnChainConfirmed
Original mined instead OnChainConfirmed, the monitor polls it itself WITHDRAWAL_SETTLED Failed "Withdrawal request A was mined instead", every replacement, a cold one awaiting approval included

Changed after review: the first version reported WITHDRAWAL_BUMPED / WITHDRAWAL_CANCELLED as soon as the replacement was in the mempool, which flipped to WITHDRAWAL_SETTLED when the original was mined instead (an integrator refunding on CANCELLED would have paid twice). Clients that switched to the replacement on WITHDRAWAL_BUMPED must follow replaced_by_request_id instead. A transaction that drops out of the mempool is not failed for that: FAILED only comes from a confirmation that proves it can never confirm; alerting on a chain that left the mempool is a follow-up.

  • Monitor: null-safe NBXplorer result, per-row errors instead of aborting the batch, then ReconcileReplacementChainsAsync gives each live RBF-linked row its final status from the database alone. Every member of a chain spends the original's inputs, so one confirming decides the rest: a confirmed replacement of a request makes it Bumped (or Cancelled when a cancellation sits on the path), any other confirmed member (what it replaced, or a sibling branch from a double submit) fails it. Each row writes only itself, so a failed write is retried next run.
  • Replacements: GetReplacementsAsync(ids) returns the replacements of a set of requests in one query; RbfReplacements answers "live" (gates a further replacement) and "newest" (the next hop) over them. No navigation on the entity, so a caller cannot forget to load it.
  • Guard: creating a replacement is refused while one is live (FAILED_PRECONDITION). MarkOriginalAsReplacedAsync, RevertOriginalAsync and the "Cancelled by RBF replacement request" marker comparison are gone.
  • gRPC: GetWithdrawalsRequestStatusByReferenceIds returns every request carrying the reference id, oldest first, each with its own status. WithdrawalRequest gains replaced_by_request_id (newest non-abandoned replacement, set even while awaiting approval). WITHDRAWAL_BUMPED / WITHDRAWAL_CANCELLED are reported only once stored, i.e. once the replacement confirmed; until then the replaced request reads WITHDRAWAL_PENDING_CONFIRMATION with replaced_by_request_id set. Confirmations are looked up in parallel.
  • UI: bump / cancel gated on "pending and no live replacement"; the status cell shows the stored status plus "replaced by #N"; dismissing a replacement's confirmation modal reloads the grid so the original gets its actions back; the mark / revert paths are gone.
  • Infra: TRANSACTION_CONFIRMATION_MINIMUM_BLOCKS is clamped to at least one at its source (an evicted transaction sits at 0 and must never settle); the generic repository's AddAsync returns the database's message like AddRangeAsync already did; the AutoMapper self-map ignores the replaced-request navigation so Update never writes a neighbour.

Migration BumpedIsTerminalOnly

Data only, no schema change. A Bumped or RBF-cancelled request none of whose replacements (recursively) confirmed goes back to OnChainConfirmationPending; rows whose replacement confirmed are already final and are left alone.

What it does in prod (checked read-only through Teleport on 2026-10-20, with the migration's own query run as a SELECT): no replacement in flight, no RBF cancellation ever used, 14 rows stored Bumped. 13 have a confirmed replacement and are left alone. One is revived: request 1489, marked Bumped on 2025-08-21 when its bump, request 1492, was created; that bump was cancelled before it was ever broadcast, so nothing reverted 1489, and its own transaction confirmed half an hour later in block 911022. It has been reported as WITHDRAWAL_BUMPED ever since, which is the bug this PR fixes. After the deploy the first MonitorWithdrawalsJob run stores it OnChainConfirmed; the log line to look for is Updating wallet withdrawal: 1489 to status: OnChainConfirmed.

Tests

  • Unit: 863 passing (Category!=E2E), just check-migrations clean. Coverage for the guard, the chain reconciliation (bump of a bump, bump of a cancellation, cancellation of a bump, sibling branch, cold replacement awaiting approval, finalizing rows skipped, both directions confirmed), the stored statuses and next hop over gRPC, the by-reference chain response, the replacements lookup and the mapper.
  • E2E (WithdrawalRbfBumpE2ETests): fee bump, RBF cancel, the original mined instead of its replacement (bump and cancel), bump of a bump with the first mined (A settles, B and C fail), and the by-reference chain. Every in-flight poll expects WITHDRAWAL_PENDING_CONFIRMATION with replaced_by_request_id set, and the final statuses only after mining. Runs in CI under the run-e2e-tests label.

@markettes
markettes requested review from Jossec101 and imclvr and a balanced review from Copilot and removed request for Jossec101 October 14, 2026 14:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Persistence failures can still leave originals terminally stuck, and one settlement path reports success after an incomplete update.

3 open findings
What changed in this PR

Improves RBF withdrawal handling when an original transaction confirms instead of its replacement.

Changes:

  • Detects and settles confirmed originals while failing losing replacements.
  • Safely reverts originals after replacement failures.
  • Adds repository, job, service, and E2E coverage.
File Description
src/​Services/​WithdrawalRequestService.cs Adds confirmed-original settlement logic.
src/​Jobs/​MonitorWithdrawalsJob.cs Checks unconfirmed replacements for confirmed ancestors.
src/​Jobs/​PerformWithdrawalJob.cs Reverts originals after execution failures.
src/​Pages/​Withdrawals.razor Reorders replacement marking and scheduling.
src/​Data/​Repositories/​WalletWithdrawalRequestRepository.cs Resolves references to the transaction that settled.
test/​NodeGuard.Tests/​TestHelpers/​TestEnvironment.cs Configures deterministic confirmation depth.
test/​NodeGuard.Tests/​Services/​WithdrawalRequestServiceTests.cs Tests settlement and replacement chains.
test/​NodeGuard.Tests/​Jobs/​PerformWithdrawalJobTests.cs Tests failed replacement execution.
test/​NodeGuard.Tests/​Jobs/​MonitorWithdrawalsJobTests.cs Tests monitoring decisions.
test/​NodeGuard.Tests/​E2E/​WithdrawalRbfBumpE2ETests.cs Covers mined-original RBF races end-to-end.
test/​NodeGuard.Tests/​Data/​Repositories/​WalletWithdrawalRequestRepositoryTests.cs Tests reference-ID chain resolution.
docker/​e2e/​README.md Documents the new E2E scenarios.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/Jobs/PerformWithdrawalJob.cs Resolved
Comment thread src/Pages/Withdrawals.razor Resolved
Jossec101
Jossec101 previously approved these changes Oct 14, 2026

@Jossec101 Jossec101 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM check the comments.

This comment was marked as outdated.

This comment was marked as outdated.

@markettes
markettes force-pushed the fix/failed-rbf-original-confirmed branch from e97e922 to ecdcba2 Compare October 16, 2026 14:01
@markettes
markettes requested a balanced review from Copilot October 16, 2026 14:02

This comment was marked as outdated.

Comment on lines +110 to +118
if (request.Status == WalletWithdrawalRequestStatus.OnChainConfirmed) return request;

// A replacement is always newer than what it replaces, so ids only go down and the walk ends.
request = request.BumpingWalletWithdrawalRequestId is { } replacedId && replacedId < request.Id
? byId.GetValueOrDefault(replacedId)
: null;
}

return latest;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could lead to a situation where, while the winner tx is being confirmed, the failed replacement (the latest) can be returned, which in elenpay side would be difficult to differentiate between "the whole settlement failed" and just the replacement failed....

What about just serving everything as an array instead of replacing? The grpc is already built that way. If all the txs participants in a rbf are returned in a array per refid, we could just listen until one is confirmed

Another option is to change the status to be OnchainConfirmationPending instead of OnchainConfirmed but in my opinion, the array one is the most healthy and less risky one (because you are not overwriting)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you guys want my opinion, I think that this is correct in the way @markettes announced in the EP channel: What it is returned, is the tx mined (or failed, if both fails). That's the part done in lines 112-115 right? 🤔

To us that would suit us well: The latest request returned by the grpc call is actually what it gets promoted if that's the case.

Said that, architectural wise is a bit confusing (philosoraptor from this point on; just thinking out loud but not blocking the PR):

  • WD request is not actually a request in the way it's a registered petition from a caller. It is actually a WD transaction.
  • Returning the history is a way to track what happens for that tx and having multiple "requests" on the response means, there could be a jump that generated a new tx.
  • What it is actually blocking us is the current "jump" happening on the tx replaced, that is position on top of the history. Following the list approach would require that such change in the WD ordering does not happen, so that the replaced remains before the replacement.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have changed few things, this was a mess tbh. rn it is returning every request with the reference Id as you requested. I have clarify a bit the flow.

A withdrawal request's status now only describes its own transaction.
Creating, executing or abandoning an RBF replacement never writes the
request it replaces: the API derives WITHDRAWAL_BUMPED / _CANCELLED from
a broadcast replacement, and the monitor stores Bumped / Cancelled only
once a replacement confirms, or settles the request itself when it was
mined instead. Bumped is terminal.

- MonitorWithdrawalsJob: null-safe, per-row errors, then
  ReconcileReplacementChainsAsync gives each live RBF-linked row its final
  status from its chain alone: Failed when what it replaced confirmed or
  a sibling branch did, Bumped / Cancelled when a replacement of it did
- The replaced request is never marked or reverted; creating a
  replacement is refused while one is live; the replacements of a request
  come from one repository lookup, not a navigation
- GetWithdrawalsRequestStatusByReferenceIds returns the whole chain, and
  WithdrawalRequest gains replaced_by_request_id
- Migration repairs legacy Bumped / RBF-cancelled rows with no confirmed
  replacement back to pending
- Withdrawals page gates bump / cancel on no live replacement and names
  the replacement next to the stored status
- E2E: the original mined instead of its replacement (bump, cancel, bump
  of a bump) and the by-reference chain
@markettes
markettes force-pushed the fix/failed-rbf-original-confirmed branch from c06d7b8 to 01b77a0 Compare October 19, 2026 17:32
@markettes markettes changed the title Fix withdrawal process for confirmed original transactions Keep replaced withdrawals pending until their RBF replacement confirms Oct 19, 2026
@markettes
markettes requested a balanced review from Copilot October 19, 2026 17:35
@markettes
markettes removed this pull request from stack #646 October 19, 2026 17:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/Pages/Withdrawals.razor Resolved
@markettes
markettes requested a review from daliclovr October 20, 2026 14:06
@markettes
markettes marked this pull request as ready for review October 20, 2026 14:10
@imclvr

imclvr commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Hey @markettes !

One thing, in the table on the PR description, you mentioned that a fee bump where the original tx is mined instead of the replacement, this change will return "failed but other mined" .

I think it'd be better to be returned as a success but including the promoted hash, as eitherway, from the caller perspective, the requested withdrawal is ongoing and failed could lead to misinterpretation.

In any case, I applaude your initiative on this 💪 , but unfortunately this topic requires strong consensus from EP and custody. From EP perspective, it's a really important thing that could bring us relief if we get a real, comprehensible snapshot of what happened with the withdrawal, and our processing depends on this.

Is that ok for you guys to have a meeting @daliclovr @markettes @Jossec101 @RodriFS about this? Sorry if it sounds too bureaucratic, but we could benefit a lot. "

Comment thread src/Proto/nodeguard.proto Resolved
@markettes
markettes requested a review from Jossec101 October 22, 2026 06:54
Comment on lines +29 to +49
WITH RECURSIVE roots AS (
SELECT w.""Id"" FROM ""WalletWithdrawalRequests"" w
WHERE w.""Status"" = 8
OR (w.""Status"" = 1 AND EXISTS (
SELECT 1 FROM ""WalletWithdrawalRequests"" r
WHERE r.""BumpingWalletWithdrawalRequestId"" = w.""Id"" AND r.""IsRbfCancellation""))
), descendants AS (
SELECT r.""Id"" AS root_id, r.""Id"" AS node_id FROM roots r
UNION
SELECT d.root_id, c.""Id"" FROM descendants d
JOIN ""WalletWithdrawalRequests"" c ON c.""BumpingWalletWithdrawalRequestId"" = d.node_id
), settled AS (
SELECT DISTINCT d.root_id FROM descendants d
JOIN ""WalletWithdrawalRequests"" n ON n.""Id"" = d.node_id
WHERE n.""Id"" <> d.root_id AND n.""Status"" = 5
)
UPDATE ""WalletWithdrawalRequests"" w
SET ""Status"" = 4,
""RejectCancelDescription"" = CASE WHEN w.""Status"" = 1 THEN NULL ELSE w.""RejectCancelDescription"" END,
""UpdateDatetime"" = now()
WHERE w.""Id"" IN (SELECT ""Id"" FROM roots) AND w.""Id"" NOT IN (SELECT root_id FROM settled);";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This migration feels risky to me. It rewrites the status of past requests and no test covers its SQL. I see you ran it as a SELECT in prod, which helps, but it will also run on every other environment against whatever data is there.

But going beyond and stepping back The original goal was fixing what GetWithdrawalRequest reports. Could we leave stored statuses alone and derive the status on read? Replacements already copy the ReferenceId, so NG can load the whole chain and work out the final status on each call, asking NBXplorer which tx actually confirmed.

For fixing those recursives we could just trigger a one-off (automated or not, but better automated 😬 ) query apart of this PR. But that would be an optional thing, because NG would be able to read and understand properly the real status of the psp ref id.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can move it to a follow up pr and there do so tests but also you should prove that its risky or where does that feeling come from

@imclvr imclvr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for being picky on here, as this may impact on the way we are handling withdrawals 🙏

I see two risks:

  1. Complexity. We're changing data on prod here, monitors and a lot of things. IMHO the API part is just a matter of wrapping data: when building the response, a single function could read the history of withdrawal requests and their statuses and decide the ultimate status of the whole thing. The rest of the changes could go in separate, smaller PRs.

  2. Impact surface: We're updating existing GetWithdrawalRequest. In order to migrate properly and not exposing a gap on ep-onchain, I would create a new method GetWithdrawalRequestV2 . That would leave us margin to migrate.

@Jossec101

Jossec101 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Sorry for being picky on here, as this may impact on the way we are handling withdrawals 🙏

I see two risks:

  1. Complexity. We're changing data on prod here, monitors and a lot of things. IMHO the API part is just a matter of wrapping data: when building the response, a single function could read the history of withdrawal requests and their statuses and decide the ultimate status of the whole thing. The rest of the changes could go in separate, smaller PRs.
  2. Impact surface: We're updating existing GetWithdrawalRequest. In order to migrate properly and not exposing a gap on ep-onchain, I would create a new method GetWithdrawalRequestV2 . That would leave us margin to migrate.

Sorry for being picky on here, as this may impact on the way we are handling withdrawals 🙏

I see two risks:

  1. Complexity. We're changing data on prod here, monitors and a lot of things. IMHO the API part is just a matter of wrapping data: when building the response, a single function could read the history of withdrawal requests and their statuses and decide the ultimate status of the whole thing. The rest of the changes could go in separate, smaller PRs.
  2. Impact surface: We're updating existing GetWithdrawalRequest. In order to migrate properly and not exposing a gap on ep-onchain, I would create a new method GetWithdrawalRequestV2 . That would leave us margin to migrate.
  1. It can be moved to a follow up pr, but please confirm what's the risk with actual proof pls.
  2. NACK, it's a bug, why would a bug trigger a V2 RPC creation? bugs get fixed, and integrations act accordignly. It's also a rare edge case, not something fundamental to trigger a whole v2 rpc, that's actual complexity.

@imclvr

imclvr commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor
  1. It can be moved to a follow up pr, but please confirm what's the risk with actual proof pls.

Overwriting a WD that ep-onchain is monitoring and getting any inconsistency. True that the surface layer is minimum (depending on the release window) and as it is a correction it will happen sooner or later. But better separately and with an ad-hoc warn to ops, IMHO.

  1. NACK, it's a bug, why would a bug trigger a V2 RPC creation? bugs get fixed, and integrations act accordignly. It's also a rare edge case, not something fundamental to trigger a whole v2 rpc, that's actual complexity.

But this changes the contract right? That's my point. In particular the semantics of results that we're actively consuming this, so the way we process withdrawals could start having different (hence unexpected) results we'd need to tackle suddenly if you release.

As an example and digging a little bit, this could even lead to open some cases where we could pay twice from ep-onchain in some batching race conditions. We usually retry failed wd requests. If we query without adaptation to this new semantic, the elenpay tx with tx hash assigned from replacement may be FAILED. Then it will retry and make another request => paid twice💥

That's mainly why we need a V2, to adapt and enable after your release.

@Jossec101

Jossec101 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor
  1. It can be moved to a follow up pr, but please confirm what's the risk with actual proof pls.

Overwriting a WD that ep-onchain is monitoring and getting any inconsistency. True that the surface layer is minimum (depending on the release window) and as it is a correction it will happen sooner or later. But better separately and with an ad-hoc warn to ops, IMHO.

  1. NACK, it's a bug, why would a bug trigger a V2 RPC creation? bugs get fixed, and integrations act accordignly. It's also a rare edge case, not something fundamental to trigger a whole v2 rpc, that's actual complexity.

But this changes the contract right? That's my point. In particular the semantics of results that we're actively consuming this, so the way we process withdrawals could start having different (hence unexpected) results we'd need to tackle suddenly if you release.

As an example and digging a little bit, this could even lead to open some cases where we could pay twice from ep-onchain in some batching race conditions. We usually retry failed wd requests. If we query without adaptation to this new semantic, the elenpay tx with tx hash assigned from replacement may be FAILED. Then it will retry and make another request => paid twice💥

That's mainly why we need a V2, to adapt and enable after your release.

How has the contract changed? What's changed is statuses for security reasons but thats not a v2, status cant be v1 an v2 and the same time, ng rbf was a flawed design and you should control that on your side now with the fix

@imclvr

imclvr commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

and you should control that on your side now with the fix

Yes indeed! And that's the point 😄 ! But would be lovely to do in a way that you can release at your own pace, while we fix at our own pace.

The thing is that we don't have a window where we are exposed to such scenario of possible double spending.

A FF, a temporary parameter on the request... whatever. Any idea is welcome.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants