workloads+smite-scenarios: patch CLN to sync with new blocks faster - #156
Merged
Merged
Conversation
That is difficult to answer with just the plots. While the plots provide a qualitative way of comparing the efficacy of two approaches, the statistical tests performed in the evaluation script are supposed to be the ground-truth oracles to determine answers to questions of this nature. Could you provide the summary statistics from the evaluation report? It should be a markdown table similar to this:
|
Contributor
Author
|
The experiment is definitely statistically under-powered (needs longer and more trials). But solely from the results here, the experimental configuration (which I assume is the Bottom line, this is the strongest possible result we can observe using our framework! The |
|
The code looks fine to me, but I think it would be better to store patches in a separate directory like fuzzamoto does instead of alongside Docker files. Maybe in a |
Collaborator
|
BTW, can we now remove the previous CLN 30s -> 2s patch? |
ekzyis
approved these changes
Jul 21, 2026
| rpc_port: self.bitcoind_rpc_port, | ||
| p2p_port: self.bitcoind_p2p_port, | ||
| extra_args: vec
erickcestari
suggested changes
Jul 21, 2026
Contributor
There was a problem hiding this comment.
I don't think coverage is the most meaningful metric for evaluating this change. Since the goal of this PR is to maintain the same coverage while improving execution throughput, I think the benchmark from #160 is more appropriate, as it repeatedly executes the same input directly with NYX.
I ran the benchmark 6 times, and in each run I executed the following input 100 times:
v0 = LoadPrivateKey(0x00…01)
v1 = DerivePoint(v0)
v2 = LoadPrivateKey(0x00…02)
v3 = DerivePoint(v2)
v4 = LoadPrivateKey(0x00…03)
v5 = DerivePoint(v4)
v6 = LoadPrivateKey(0x00…04)
v7 = DerivePoint(v6)
v8 = LoadPrivateKey(0x00…05)
v9 = DerivePoint(v8)
v10 = LoadPrivateKey(0x00…06)
v11 = DerivePoint(v10)
v12 = LoadChannelId(0x1111…11)
v13 = LoadChainHashFromContext()
v14 = LoadAmount(100000)
v15 = LoadAmount(0)
v16 = LoadAmount(546)
v17 = LoadAmount(10000000)
v18 = LoadAmount(1000)
v19 = LoadAmount(1)
v20 = LoadFeeratePerKw(2500)
v21 = LoadU16(144)
v22 = LoadU16(483)
v23 = LoadU8(1)
v24 = LoadShutdownScript(Empty)
v25 = LoadChannelType(Anchors)
v26 = BuildOpenChannel(v13, v12, v14, v15, v16, v17, v18, v19, v20, v21, v22, v1, v3, v5, v7, v9, v11, v23, v24, v25)
v27 = SendOpenChannel(v26)
v28 = RecvAcceptChannel(v27)
v29 = ExtractFundingPubkey(v28)
v30 = CreateFundingTransaction(v1, v29, v14, v20)
v31 = SendFundingCreated(v30, v0, v12)
v32 = RecvFundingSigned(v31)
BroadcastTransaction(v30)
MineBlocks(8)
v35 = LoadPrivateKey(0x00…07)
v36 = DerivePoint(v35)
v37 = LoadShortChannelId(1x1x0)
SendChannelReady{include_alias=false}(v32, v36, v37)
RecvChannelReady()
For the CLN JSON-RPC implementation, I got:
Smite Nyx benchmark
target: cln/ir
sharedir: /tmp/smite-nyx
input size: 410 bytes
iterations: 100
repeats: 6
run 1/6:
snapshot creation (first exec): 338.79 ms
steady-state (snapshot restore + target run):
execs/sec: 4.4
wall time: 22.631 s
latency: min 215.34 ms mean 226.31 ms median 221.34 ms p99 273.81 ms max 290.26 ms
input execution: mean 203.76 ms median 199.32 ms (guest runtime)
nyx overhead: mean 22.55 ms median 21.85 ms (restore + reset + ipc; 58686 dirty pages/exec)
non-normal results: 0 / 100
run 2/6:
snapshot creation (first exec): 285.37 ms
steady-state (snapshot restore + target run):
execs/sec: 4.5
wall time: 22.453 s
latency: min 214.58 ms mean 224.53 ms median 224.42 ms p99 234.49 ms max 238.29 ms
input execution: mean 202.42 ms median 202.32 ms (guest runtime)
nyx overhead: mean 22.11 ms median 21.92 ms (restore + reset + ipc; 58224 dirty pages/exec)
non-normal results: 0 / 100
run 3/6:
snapshot creation (first exec): 321.92 ms
steady-state (snapshot restore + target run):
execs/sec: 4.4
wall time: 22.698 s
latency: min 215.03 ms mean 226.98 ms median 220.90 ms p99 265.48 ms max 276.45 ms
input execution: mean 204.73 ms median 199.26 ms (guest runtime)
nyx overhead: mean 22.25 ms median 21.64 ms (restore + reset + ipc; 58087 dirty pages/exec)
non-normal results: 0 / 100
run 4/6:
snapshot creation (first exec): 315.64 ms
steady-state (snapshot restore + target run):
execs/sec: 4.4
wall time: 22.657 s
latency: min 215.28 ms mean 226.57 ms median 224.91 ms p99 252.71 ms max 268.01 ms
input execution: mean 204.30 ms median 202.43 ms (guest runtime)
nyx overhead: mean 22.27 ms median 22.06 ms (restore + reset + ipc; 58112 dirty pages/exec)
non-normal results: 0 / 100
run 5/6:
snapshot creation (first exec): 284.72 ms
steady-state (snapshot restore + target run):
execs/sec: 4.2
wall time: 23.837 s
latency: min 220.04 ms mean 238.37 ms median 232.66 ms p99 287.95 ms max 294.65 ms
input execution: mean 214.49 ms median 209.38 ms (guest runtime)
nyx overhead: mean 23.88 ms median 22.94 ms (restore + reset + ipc; 59717 dirty pages/exec)
non-normal results: 0 / 100
run 6/6:
snapshot creation (first exec): 312.56 ms
steady-state (snapshot restore + target run):
execs/sec: 4.0
wall time: 24.873 s
latency: min 217.83 ms mean 248.73 ms median 245.15 ms p99 292.91 ms max 304.05 ms
input execution: mean 224.14 ms median 220.37 ms (guest runtime)
nyx overhead: mean 24.59 ms median 24.71 ms (restore + reset + ipc; 58166 dirty pages/exec)
non-normal results: 0 / 100
aggregate over 6 runs:
execs/sec: mean 4.3 stddev 0.2 min 4.0 max 4.5
snapshot: mean 309.83 ms
latency (mean across runs):
min 216.35 ms mean 231.91 ms median 228.23 ms p99 267.89 ms max 278.62 ms
input execution (mean across runs): mean 208.97 ms median 205.51 ms
nyx overhead (mean across runs): mean 22.94 ms median 22.52 ms
non-normal results: 0 / 600
With the CLN polling implementation using a 50 ms interval, the results were:
Smite Nyx benchmark
target: cln/ir
sharedir: /tmp/smite-nyx
input size: 410 bytes
iterations: 100
repeats: 6
run 1/6:
snapshot creation (first exec): 164.29 ms
steady-state (snapshot restore + target run):
execs/sec: 6.2
wall time: 16.231 s
latency: min 148.71 ms mean 162.31 ms median 159.20 ms p99 200.43 ms max 254.28 ms
input execution: mean 152.66 ms median 150.25 ms (guest runtime)
nyx overhead: mean 9.65 ms median 9.53 ms (restore + reset + ipc; 21166 dirty pages/exec)
non-normal results: 0 / 100
run 2/6:
snapshot creation (first exec): 141.56 ms
steady-state (snapshot restore + target run):
execs/sec: 7.5
wall time: 13.368 s
latency: min 129.52 ms mean 133.68 ms median 133.42 ms p99 138.12 ms max 145.71 ms
input execution: mean 124.27 ms median 124.09 ms (guest runtime)
nyx overhead: mean 9.41 ms median 9.42 ms (restore + reset + ipc; 21929 dirty pages/exec)
non-normal results: 0 / 100
run 3/6:
snapshot creation (first exec): 134.82 ms
steady-state (snapshot restore + target run):
execs/sec: 7.5
wall time: 13.370 s
latency: min 111.58 ms mean 133.70 ms median 133.85 ms p99 145.51 ms max 160.73 ms
input execution: mean 124.18 ms median 124.25 ms (guest runtime)
nyx overhead: mean 9.52 ms median 9.52 ms (restore + reset + ipc; 22083 dirty pages/exec)
non-normal results: 0 / 100
run 4/6:
snapshot creation (first exec): 136.57 ms
steady-state (snapshot restore + target run):
execs/sec: 7.5
wall time: 13.325 s
latency: min 127.18 ms mean 133.25 ms median 132.65 ms p99 143.67 ms max 165.25 ms
input execution: mean 123.89 ms median 123.55 ms (guest runtime)
nyx overhead: mean 9.36 ms median 9.29 ms (restore + reset + ipc; 21880 dirty pages/exec)
non-normal results: 0 / 100
run 5/6:
snapshot creation (first exec): 120.91 ms
steady-state (snapshot restore + target run):
execs/sec: 6.1
wall time: 16.383 s
latency: min 156.31 ms mean 163.82 ms median 162.39 ms p99 181.66 ms max 189.13 ms
input execution: mean 154.25 ms median 152.75 ms (guest runtime)
nyx overhead: mean 9.57 ms median 9.53 ms (restore + reset + ipc; 21705 dirty pages/exec)
non-normal results: 0 / 100
run 6/6:
snapshot creation (first exec): 142.95 ms
steady-state (snapshot restore + target run):
execs/sec: 7.7
wall time: 12.946 s
latency: min 112.60 ms mean 129.46 ms median 129.22 ms p99 141.65 ms max 168.44 ms
input execution: mean 120.29 ms median 120.22 ms (guest runtime)
nyx overhead: mean 9.17 ms median 9.11 ms (restore + reset + ipc; 21266 dirty pages/exec)
non-normal results: 0 / 100
aggregate over 6 runs:
execs/sec: mean 7.1 stddev 0.7 min 6.1 max 7.7
snapshot: mean 140.18 ms
latency (mean across runs):
min 130.98 ms mean 142.70 ms median 141.79 ms p99 158.51 ms max 180.59 ms
input execution (mean across runs): mean 133.26 ms median 132.52 ms
nyx overhead (mean across runs): mean 9.45 ms median 9.40 ms
non-normal results: 0 / 600
Compared to the JSON-RPC implementation, the 50 ms polling interval increased execution throughput by roughly 81% (4.3 -> 7.8 execs/sec) while reducing average end-to-end latency from 230 ms to 129 ms.
I think it's worth considering using the 50 ms polling interval. This difference is only noticeable when running with NYX.
My guess is that spawning processes dirty a lot of memory (page tables, loaded binaries, and RPC buffers). NYX must restore every dirty page on the next snapshot reset, so an exec with many spawns makes every subsequent reset more expensive.
Edit: I've updated the benchmark code to calculate the NYX snapshot overhead. My guess wasn't entirely correct. The polling accounts for half of the nyx JSON-RPC overhead. However, this doesn't explain why the raw input execution was faster than the clean JSON-RPC. I'll try running it on a different computer.
Collaborator
|
The fuzzing results show significantly higher coverage for the notification variant, which also implies more execs/sec. But the benchmark shows more execs/sec for the polling variant. Some possible explanations:
I haven't reviewed #160 yet, so there could be other explanations. Just wanted to share some initial thoughts. |
Contributor
One possible explanation for the higher level of coverage is that it uses JSON-RPC and Lightning-CLI, which access code that the polling variant cannot/will access. |
Collaborator
|
We should be able to test these hypotheses empirically.
Insert various sleeps before snapshot creation to offset the block-sync intervals, rebenchmark and compare results. At most can explain 50ms of the ~90ms difference.
Benchmark the exact same program without the
Run the benchmark input through both variants and subtract the coverage difference. Adjust the coverage numbers from the fuzzing result accordingly. |
Contributor
I've tested with an input that never calls InputResults: json-rpc notification50ms pollingClaude reason why that polling was faster based on the results:
|
Contributor
|
This is true for For an input that never calls inputjson rpc notification:50ms polling: |
Contributor
The difference is negible, only +10 lines. I think it would be valuable to run I don't want this to block the PR. Since it has already been approved and is clearly an improvement over the previous 20-second polling interval, I think it's worth merging as-is so we can keep moving forward. We can always follow up with a more thorough evaluation afterward to validate the overall fuzzing performance. |
CLN's only polls for new blocks periodically, so scenarios still wait up to the poll interval after mining a block. Add a `syncblocks` RPC that forces CLN to poll immediately, and wire it to bitcoind's `-blocknotify` so CLN syncs as soon as a block is mined. The RPC complements periodic polling rather than replacing it: concurrent requests are coalesced, and polling still catches any blocks missed while CLN is processing a previous chain update. Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
NishantBansal2003
force-pushed
the
cln-syncwallet-rpc
branch
from
July 22, 2026 12:54
6da3fb7 to
c6f15cf
Compare
Contributor
Author
|
I used the script from #160 on my machine with the same input shared in #156 (review). Here are the results I got: For the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref: #143
I tried two approaches for CLN:
syncblocksRPC that triggers an immediate chain sync, and tie it to bitcoind’s-blocknotifyhookIn both cases, I ran campaign for 2 days and didn’t observe any target-side issues, so both approaches appear to be stable. To determine which approach performs better, I compared their coverage using the smite evaluation script. I ran five 1h trials for each approach, then replayed the generated corpus from each trial in local mode while measuring the CPU overhead for both CLN and
bitcoind. Here are the resulting plots for reference (Let me know if running longer campaign trials would provide a better comparison?):Target: cln
Median Coverage Over Time
Distribution Comparisons
CPU Overhead (median)
With the 50ms polling interval approach:
With the
-blocknotifyhook approach: