smitebot: add bench command to measure Nyx execution speed by erickcestari · Pull Request #160 · lnfuzz/smite

erickcestari · 2026-07-21T13:20:44Z

The goal is to tell whether a change concretely improved the target's throughput or not: run bench before and after and compare execs/sec.

It runs a single input through the target's Nyx VM many times, timing the snapshot-restore-plus-target-run loop. Each exec is one snapshot restore plus one target run, so the numbers isolate VM/snapshot and target speed from AFL++'s mutation and scheduling overhead.

Like start, it builds the image and sets up the sharedir by default (pass --no-build to reuse an existing one), and loads libnyx.so at runtime the same way afl-fuzz does. --repeat averages over several fresh VM boots so a real change stands out from boot/snapshot variance, and --timeout bounds a single execution (default 2s).

Chand-ra

Looks like a great addition overall. Following are some improvement suggestions.

erickcestari · 2026-07-23T15:14:52Z

Thanks for the review @Chand-ra ! I've added the coverage metric, which is a nice feature to have. Specially when we start using the bedrock as one of the hypervisor.

Ashish-Kumar-Dash

I had a few thoughts/nits, otherwise its a solid PR! The only real concern I have is the order in which we merge status, bench and print-ir since one of us will have to rebase, but I think it'd be much better if bench,print-ir could get merged and then we land status. I don't think it will be an issue on your part, since I'd have to do the rebase then :)

morehouse · 2026-07-21T17:35:10Z

+smitebot bench campaign.toml --input testcase.bin --iterations 5000
+```
+
+- `--input`: Input file to execute repeatedly. Defaults to a single `0x00` byte.


I think --input should be required -- it should be rare that anyone wants to benchmark an empty program.

It's useful for quickly checking performance improvements. I've used it to quickly test JVM optimization because the executor checks if the target is live, which does a ping-pong.

I think it is worth mentioning in the documentation then: the default input can be used to quickly check performance improvements due to factors like target optimization.

morehouse · 2026-07-23T17:05:39Z

+    /// Start of the `result` section within the buffer.
+    const RESULT: usize = 896;
+    /// `result.dirty_pages` (`u32`): pages restored for the last exec.
+    pub const DIRTY_PAGES: usize = RESULT + 16;
+    /// `result.runtime_usec` (`u32`): microsecond part of guest payload runtime.
+    pub const RUNTIME_USEC: usize = RESULT + 28;
+    /// `result.runtime_sec` (`u32`): second part of guest payload runtime.
+    pub const RUNTIME_SEC: usize = RESULT + 32;


These offsets are all correct AFAICT. But it seems fragile -- it would be nice if Nyx provided a clearer interface.

erickcestari · 2026-07-23T21:09:49Z

Thanks @morehouse and @Ashish-Kumar-Dash for the review.

Chand-ra

GitHub started acting up so I couldn't use the "suggest change" functionality🤦. Looks mostly good to me, following are a couple of suggestions.

Chand-ra · 2026-07-24T05:57:40Z

+smitebot bench campaign.toml --input testcase.bin --iterations 5000
+```
+
+- `--input`: Input file to execute repeatedly. Defaults to a single `0x00` byte.


I think it is worth mentioning in the documentation then: the default input can be used to quickly check performance improvements due to factors like target optimization.

Chand-ra

Took a quick look and verified that all the pointers from the previous feedback were addressed, looks good to me now!

Ashish-Kumar-Dash · 2026-07-30T13:51:37Z

@erickcestari I pushed some recent commits, you might have to rebase

erickcestari · 2026-07-30T14:29:01Z

@erickcestari I pushed some recent commits, you might have to rebase

Thanks! I've rebased it now.

morehouse · 2026-08-04T17:46:33Z

+    /// Maximum input buffer size in bytes for the Nyx VM. Defaults to the input
+    /// size rounded up to a 4 KiB page; pass 1048576 to match AFL++'s 1 MiB buffer.
+    #[arg(long)]
+    max_input_size: Option<u32>,


Is there ever a reason to use this flag now that we automatically calculate the buffer size?

Actually I'll remove the adaptive buffer size, since when running with AFL++ the value used is always the 1MiB MAX_FILE. So I believe it makes more sense to bench in the same way AFL++ runs. Also I'll keep an option to change the max buffer size to test if changing it makes any difference on performance.

Move setup_nyx out of the start command and into utils so other commands can reuse it to prepare the Nyx sharedir. Pure refactor, no behavior change.

The goal is to tell whether a change concretely improved the target's throughput or not: run bench before and after and compare execs/sec. It runs a single input through the target's Nyx VM many times, timing the snapshot-restore-plus-target-run loop. Each exec is one snapshot restore plus one target run, so the numbers isolate VM/snapshot and target speed from AFL++'s mutation and scheduling overhead. Like start, it builds the image and sets up the sharedir by default (pass --no-build to reuse an existing one), and loads libnyx.so at runtime the same way afl-fuzz does. --repeat averages over several fresh VM boots so a real change stands out from boot/snapshot variance, and --timeout bounds a single execution (default 2s). Adds FFI bindings to libnyx (loaded via dlopen) and a small stats module for latency percentiles and mean/stddev aggregation.

erickcestari · 2026-08-06T15:41:38Z

Thanks @morehouse. It was an insightful review. It's ready for another one.

morehouse · 2026-08-07T15:47:06Z

+/// # Errors
+/// Returns an error if `cpu` is out of range or is not an online CPU this
+/// process is allowed to run on.
+pub fn pin_to_cpu(cpu: usize) -> Result<(), String> {


This should not be added in the commit labeled "pure refactor". It also looks like it's introduced before we even add the needed libc dependency.

morehouse · 2026-08-07T15:58:20Z

+    // `size_of::<cpu_set_t>()` bytes from the pointer. A pid of 0 means the
+    // calling thread.
+    let failed = unsafe {
+        let mut set: libc::cpu_set_t = std::mem::zeroed();


I think we should probably use CPU_ZERO rather than assuming a manual zero-init does the same.

morehouse · 2026-08-07T16:06:21Z

+    true
+}
+
+/// Pins the calling process, and every process it later spawns, to `cpu`.


Suggested change

/// Pins the calling process, and every process it later spawns, to `cpu`.

/// Pins the calling thread, and every process it later spawns, to `cpu`.

morehouse · 2026-08-07T16:33:33Z

+        // Affinity is per-thread, and this runs on the test's own thread, so
+        // pinning here leaves the rest of the suite alone. Pin to a CPU the
+        // process is already allowed on so a restricted cpuset does not fail it.
+        let mut set: libc::cpu_set_t = unsafe { std::mem::zeroed() };


Should also be using CPU_ZERO in this test.

morehouse · 2026-08-07T16:59:31Z

+/// asserts `shared_payload_buffer_size % x86_64_PAGE_SIZE == 0`
+/// (`nyx_mode/QEMU-Nyx/nyx/memory_access.c`), so any buffer size we pass must be
+/// page-rounded or the VM aborts. AFL++ never trips this because its buffer is a
+/// page-multiple 1 MiB; a `--max-input-size` that is not is rejected up front.


Not sure what the last part is about.

Suggested change

/// page-multiple 1 MiB; a `--max-input-size` that is not is rejected up front.

/// page-multiple 1 MiB.

morehouse · 2026-08-07T17:10:31Z

+    /// Returns `false` (rather than erroring) if not: benchmarking can still run,
+    /// only the target/overhead split is unreliable.


This seems like it would be confusing to the user. If we're reading garbage data, we should fail loudly rather than expecting the user to figure out that the data is garbage.

morehouse · 2026-08-07T17:14:50Z

+    /// Defaults to AFL++'s 1 MiB, the buffer a real campaign runs with. Lower it
+    /// to measure what a smaller buffer buys: the guest allocates the whole
+    /// buffer on every execution and the snapshot resets every page of it
+    /// dirtied, so buffer size shows up directly in the Nyx overhead and dirty
+    /// pages per exec.


I don't think this logic checks out. Assume the input size is X pages and the buffer size is Y > X pages. Then regardless of what Y is, only X pages ever get dirtied and need to be reset.

Have you actually observed differences in performance with varying buffer sizes?

morehouse · 2026-08-07T18:31:41Z

+        // A zero timeout reaches Nyx as "no timeout" rather than "expire
+        // immediately", which silently turns off the timing the benchmark exists
+        // to measure.
+        if args.timeout_secs == 0 {
+            log::error!("--timeout must be at least 1 second; Nyx reads zero as no timeout");
+            return false;
+        }


This is confusing to the user, who may very well want "no timeout" (a.k.a. super high timeout). The actual issue is that we lose benchmark timings .

Suggested change

// A zero timeout reaches Nyx as "no timeout" rather than "expire

// immediately", which silently turns off the timing the benchmark exists

// to measure.

if args.timeout_secs == 0 {

log::error!("--timeout must be at least 1 second; Nyx reads zero as no timeout");

return false;

}

// A zero timeout causes Nyx to silently turn off the timing the

// benchmark exists to measure.

if args.timeout_secs == 0 {

log::error!("Nyx requires --timeout to be at least 1 second");

return false;

}

morehouse · 2026-08-07T18:35:42Z

+        // Pin before booting: libnyx spawns QEMU as a child, so the VM inherits
+        // this affinity mask. Doing it here also pins the timing loop itself.
+        let cpu = args.cpu.unwrap_or(args.worker_id as usize);
+        if let Err(e) = pin_to_cpu(cpu) {
+            log::error!("{e}");
+            return false;
+        }
+        log::info!("pinned to CPU {cpu}");


Perhaps we should pin closer to where we actually start benchmarking. Currently we still load and validate a whole bunch of stuff after pinning.

morehouse · 2026-08-07T19:21:26Z

+  steady-state (snapshot restore + target run):
+    execs/sec: 1234.5
+    exec time: 810.20 ms
+    latency:   min 0.7 µs  mean 0.8 µs  median 0.8 µs  p99 1.2 µs  max 3.4 ms
+    input execution: mean 0.5 µs  median 0.5 µs   (guest runtime)
+    nyx overhead:    mean 0.3 µs  median 0.3 µs   (restore + reset + ipc; 42 dirty pages/exec)
+    failed iterations: 0 / 1000
+    coverage determinism: 98.4% stable (12 / 750 edges fluctuated)


These stats are inconsistent. exec time 810 ms implies 810 us latency per iteration.

Let's regenerate with data from a real run.

erickcestari mentioned this pull request Jul 21, 2026

workloads+smite-scenarios: patch CLN to sync with new blocks faster #156

Merged

erickcestari force-pushed the bench-exec-s branch from 5170540 to 528ffd7 Compare July 21, 2026 15:14

Chand-ra suggested changes Jul 22, 2026

View reviewed changes

Comment thread smitebot/src/commands/start.rs Outdated

Comment thread smitebot/src/commands/bench.rs Outdated

Comment thread smitebot/README.md Outdated

Comment thread smitebot/src/commands/bench_exec.rs

erickcestari mentioned this pull request Jul 22, 2026

smite-scenarios: warm up and freeze Eclair's JIT before the snapshot #161

Open

erickcestari force-pushed the bench-exec-s branch 2 times, most recently from 15cf725 to c2c557f Compare July 23, 2026 15:13

Ashish-Kumar-Dash suggested changes Jul 23, 2026

View reviewed changes

Comment thread smitebot/src/commands/bench_exec.rs

Comment thread smitebot/README.md Outdated

Comment thread smitebot/src/libnyx.rs Outdated

Comment thread smitebot/src/commands/bench_exec.rs Outdated

Comment thread smitebot/Cargo.toml Outdated

morehouse reviewed Jul 23, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 3 times, most recently from d6d5659 to 931a8b2 Compare July 23, 2026 20:53

Chand-ra suggested changes Jul 24, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 3 times, most recently from bfb96ea to 470fce8 Compare July 24, 2026 17:27

Chand-ra approved these changes Jul 28, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 2 times, most recently from c80f707 to 76314ac Compare July 30, 2026 14:28

erickcestari requested a review from morehouse July 31, 2026 14:14

morehouse requested changes Aug 4, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch from 76314ac to 01a9562 Compare August 6, 2026 15:35

erickcestari added 3 commits August 6, 2026 12:37

smitebot: hoist setup_nyx from start into utils

24a3f6e

Move setup_nyx out of the start command and into utils so other commands can reuse it to prepare the Nyx sharedir. Pure refactor, no behavior change.

smitebot: document bench-exec in README

28fe6ec

erickcestari force-pushed the bench-exec-s branch from 01a9562 to 28fe6ec Compare August 6, 2026 15:37

erickcestari requested a review from morehouse August 6, 2026 15:40

morehouse reviewed Aug 7, 2026

View reviewed changes

	/// Pins the calling process, and every process it later spawns, to `cpu`.
	/// Pins the calling thread, and every process it later spawns, to `cpu`.

	/// page-multiple 1 MiB; a `--max-input-size` that is not is rejected up front.
	/// page-multiple 1 MiB.

		/// Returns `false` (rather than erroring) if not: benchmarking can still run,
		/// only the target/overhead split is unreliable.

Uh oh!

Conversation

erickcestari commented Jul 21, 2026

Uh oh!

Chand-ra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erickcestari commented Jul 23, 2026

Uh oh!

Ashish-Kumar-Dash left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

erickcestari commented Jul 23, 2026

Uh oh!

Chand-ra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chand-ra left a comment

Choose a reason for hiding this comment

Uh oh!

Ashish-Kumar-Dash commented Jul 30, 2026

Uh oh!

erickcestari commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erickcestari commented Aug 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!