smite-ir/mutators: Add `SpliceMutator` by Chand-ra · Pull Request #135 · lnfuzz/smite

Chand-ra · 2026-06-29T06:40:34Z

Add SpliceMutator for smite-IR. Mutates a given program by inserting a spliced input at a random point in the said program.

Chand-ra · 2026-06-29T06:47:14Z

+        for instr in &self.splice.instructions {
+            let shifted_inputs: Vec<usize> = instr
+                .inputs
+                .iter()
+                .map(|&input| input + insert_idx)
+                .collect();
+            builder.append(instr.operation.clone(), &shifted_inputs);


This logic is the only logical difference between this mutator and GeneratorInsertion. It could also be implemented as:

Suggested change

for instr in &self.splice.instructions {

let shifted_inputs: Vec<usize> = instr

.inputs

.iter()

.map(|&input| input + insert_idx)

.collect();

builder.append(instr.operation.clone(), &shifted_inputs);

for instr in &self.splice.instructions {

let mut inputs = vec![];

for var_type in instr.operation.input_types() {

inputs.push(builder.pick_variable(var_type, rng));

}

builder.append(instr.operation.clone(), &inputs);

}

which I guess would "wire it stronger" to the rest of the program, but I'm not sure if that's a worthwhile tradeoff for the simplicity of the current approach.

It's an interesting idea, but I think the current simpler approach may be better. By doing pick_variable, we end up losing the specific values that the splice program used (which were selected as interesting after many random mutations), which probably hurts fuzzing effectiveness. We'd also need to figure out how to handle variable types for which pick_variable currently panics.

We also can get some of the same behavior already if InputSwapMutator is stacked after the splice.

We'd also need to figure out how to handle variable types for which pick_variable currently panics.

Would we? pick_variable() currently panics if we try to generate_fresh() a type that cannot be generated (messages, affine types, etc.).

If the corpus always has valid programs and every mutation preserves validity, won't both the input programs to SpliceMutator always be valid? That is, wouldn't every panicking type already have at least one matching candidate?

Yes, right -- we only have panic issues with generate_fresh, which shouldn't be called in this case.

We should do an experiment with the following configs to evaluate: (1) baseline, (2) this mutator, (3) a pick_variable splice mutator, (4) this mutator + a pick_variable splice mutator.

morehouse · 2026-06-30T18:52:15Z

+        for instr in &self.splice.instructions {
+            let shifted_inputs: Vec<usize> = instr
+                .inputs
+                .iter()
+                .map(|&input| input + insert_idx)
+                .collect();
+            builder.append(instr.operation.clone(), &shifted_inputs);


It's an interesting idea, but I think the current simpler approach may be better. By doing pick_variable, we end up losing the specific values that the splice program used (which were selected as interesting after many random mutations), which probably hurts fuzzing effectiveness. We'd also need to figure out how to handle variable types for which pick_variable currently panics.

We also can get some of the same behavior already if InputSwapMutator is stacked after the splice.

morehouse · 2026-06-30T18:58:11Z

+        }
+
+        // Insert the spliced program.
+        for instr in &self.splice.instructions {


I think it would be interesting to do an experiment where we compare full-program insertion with only inserting a random subset (prefix) of the spliced program.

Full-program insertion will tend to create longer (and slower) programs, so it's possible that prefix insertion actually performs better.

I think it would be interesting to do an experiment where we compare full-program insertion with only inserting a random subset (prefix) of the spliced program.

Hmm, we're inserting the prefix (instead of a random slice) so that we don't have to worry about variable dependencies. Should be easily implementable but I'm not fully sold on the "performs better" part, although I agree that it is an interesting experiment.

I think a more interesting (and perhaps advanced) version of this would be FlowInsertionMutator that selects a random Act instruction in the spliced program, yanks out its entire lineage, and inserts it at a random point in the given program.

I suspect a prefix insertion mutator and a flow insertion mutator will perform similarly in practice, since we have the DeadCodeEliminator that will also periodically do the dependency analysis and dead instruction removal that FlowInsertionMutator would. So either one would be interesting to evaluate IMO.

It may also be interesting to combine this FlowInsertionMutator idea with the pick_variable rewiring you suggested as an experiment.

morehouse

Code LGTM. This is ready for experimental evaluation.

morehouse · 2026-07-06T16:45:05Z

+        for instr in &self.splice.instructions {
+            let shifted_inputs: Vec<usize> = instr
+                .inputs
+                .iter()
+                .map(|&input| input + insert_idx)
+                .collect();
+            builder.append(instr.operation.clone(), &shifted_inputs);


Yes, right -- we only have panic issues with generate_fresh, which shouldn't be called in this case.

We should do an experiment with the following configs to evaluate: (1) baseline, (2) this mutator, (3) a pick_variable splice mutator, (4) this mutator + a pick_variable splice mutator.

morehouse · 2026-07-06T18:03:28Z

+        }
+
+        // Insert the spliced program.
+        for instr in &self.splice.instructions {


I suspect a prefix insertion mutator and a flow insertion mutator will perform similarly in practice, since we have the DeadCodeEliminator that will also periodically do the dependency analysis and dead instruction removal that FlowInsertionMutator would. So either one would be interesting to evaluate IMO.

It may also be interesting to combine this FlowInsertionMutator idea with the pick_variable rewiring you suggested as an experiment.

erickcestari · 2026-07-06T18:01:37Z

+    fn mutate_stacked(&mut self, program: &mut Program, splice: Option<Program>) {
        self.last_sequence.clear();
        // Power-of-two stack count: 1, 2, 4, 8, or 16 mutations.
        let stack = 1u32 << self.rng.random_range(0..=4);
+        let splice_mutator = splice.map(SpliceMutator::new);
+        // Only roll up to 6 if we actually have a splice input.
+        let upper_bound = if splice_mutator.is_some() { 6 } else { 5 };
        for _ in 0..stack {
            // Uniform pick between the available mutators.
-            let name = match self.rng.random_range(0..5) {
+            let name = match self.rng.random_range(0..upper_bound) {
                0 => {
                    OperationParamMutator.mutate(program, &mut self.rng);
                    "op-param"
                }
                1 => {
                    InputSwapMutator.mutate(program, &mut self.rng);
                    "input-swap"
                }
                2 => {
                    InstructionDeleteMutator.mutate(program, &mut self.rng);
                    "instr-delete"
                }
                3 => {
                    InstructionReorderMutator.mutate(program, &mut self.rng);
                    "instr-reorder"
                }
                4 => {
                    let generator = *AnyGenerator::ALL
                        .iter()
                        .choose(&mut self.rng)
                        .expect("AnyGenerator::ALL is non-empty");
                    let mutator = GeneratorInsertionMutator::new(generator);
                    mutator.mutate(program, &mut self.rng);
                    "gen-insert"
                }
+                5 => {
+                    splice_mutator
+                        .as_ref()
+                        .expect("splice present")
+                        .mutate(program, &mut self.rng);
+                    "splice"
+                }
                _ => unreachable!("random_range() bound out of sync with match arms"),
            };
            self.last_sequence.push(name);
        }
    }


nit: We could refactor to avoid hardcoding numbers that would need changing if we added or removed mutators.

Suggested change

/// Mutators available to [`MutatorState::mutate_stacked`]. The pool of

/// candidates is built per call, so optional entries (splice) are included by

/// pushing a variant.

#[derive(Clone, Copy)]

enum StackedMutator {

OpParam,

InputSwap,

InstrDelete,

InstrReorder,

GenInsert,

Splice,

}

fn mutate_stacked(&mut self, program: &mut Program, splice: Option<Program>) {

self.last_sequence.clear();

// Power-of-two stack count: 1, 2, 4, 8, or 16 mutations.

let stack = 1u32 << self.rng.random_range(0..=4);

let splice_mutator = splice.map(SpliceMutator::new);

let mut pool = vec![

StackedMutator::OpParam,

StackedMutator::InputSwap,

StackedMutator::InstrDelete,

StackedMutator::InstrReorder,

StackedMutator::GenInsert,

];

if splice_mutator.is_some() {

pool.push(StackedMutator::Splice);

}

for _ in 0..stack {

// Uniform pick between the available mutators.

let choice = *pool

.iter()

.choose(&mut self.rng)

.expect("pool is non-empty");

let name = match choice {

StackedMutator::OpParam => {

OperationParamMutator.mutate(program, &mut self.rng);

"op-param"

}

StackedMutator::InputSwap => {

InputSwapMutator.mutate(program, &mut self.rng);

"input-swap"

}

StackedMutator::InstrDelete => {

InstructionDeleteMutator.mutate(program, &mut self.rng);

"instr-delete"

}

StackedMutator::InstrReorder => {

InstructionReorderMutator.mutate(program, &mut self.rng);

"instr-reorder"

}

StackedMutator::GenInsert => {

let generator = *AnyGenerator::ALL

.iter()

.choose(&mut self.rng)

.expect("AnyGenerator::ALL is non-empty");

let mutator = GeneratorInsertionMutator::new(generator);

mutator.mutate(program, &mut self.rng);

"gen-insert"

}

StackedMutator::Splice => {

splice_mutator

.as_ref()

.expect("Splice is only pooled when a splice input exists")

.mutate(program, &mut self.rng);

"splice"

}

};

self.last_sequence.push(name);

}

}

So basically the AnyGenerator variant for mutators. It does improve readability, but I feel like the change doesn't fit well within this PR. I'll follow up with a separate one implementing it.

The following commit will implement `SpliceInsertionMutator`, which uses the decoded program from this buffer. Additionally, add a test to ensure parsing a valid `add_buf` via postcard doesn't panic or corrupt the state.

Refactor the common logic from `GeneratorInsertionMutator` tests that we will need for implementing `SpliceInsertionMutator` tests.

Chand-ra · 2026-07-27T16:54:20Z

Finally got to evaluating this mutator (thanks to the new orchestration script). Here are the results:

Fuzzing Evaluation Report

Configuration A (Baseline): baseline
Configuration B (Experimental): baseline-plus-splice

1. Summary Statistics

Target	Duration (h)	n (Baseline)	n (Exp.)	Median Cov. (Baseline)	Median Cov. (Exp.)	Adj. p-value (Cov.)	Â12 (Cov.)	Median AUC (Baseline)	Median AUC (Exp.)	Adj. p-value (AUC)	Â12 (AUC)	Union Cov. (Baseline)	Union Cov. (Exp.)	Execs/s (Baseline)	Execs/s (Exp.)
cln	12	20	20	3617.5	3641.5	0.0494718	0.7225	42704.7	43344.8	0.000666608	0.8425	5750	5071	49.025	43.57
ldk	12	20	20	12611.5	12628.5	1	0.50125	144049	139023	0.523233	0.395	14552	14983	170.48	97.04
lnd	12	20	20	25675.5	25675	0.771048	0.41875	307312	307591	0.635945	0.455	25753	25764	88.195	81.885

A comprehensive version of this table including raw P-values and Interquartile Ranges (IQRs) is available in evaluation_metrics.csv.

2. Interpretation Guide

Use the generated matrix above to objectively evaluate the experimental configuration. For full methodology, see the Smite Fuzzing Evaluation Framework.

Key Metrics

Adj. p-value: Mann-Whitney U test corrected for multiple targets via Holm-Bonferroni. Controls false-positive rate to ≤ 5% across all targets.
Â12: Probability that a random B trial outperforms a random A trial. 0.5 = no difference; 0.7 = B wins 70% of pairings. Always read alongside the p-value.
IQR: Spread of the middle 50% of trials. A much larger IQR in B suggests a few outlier runs may be inflating the median.
AUC: Coverage speed — how much was discovered and how early. Useful when final coverage is similar between configurations.
Union Coverage: Union of all trial bitmaps; the coverage ceiling for a multi-core deployment. Descriptive only, cannot be statistically tested.
Execs/s: A large drop in B without a coverage gain means the new feature is too expensive.

Reading the Results

Adj. p	`Â12`	Conclusion
< 0.05	> 0.5	Meaningful improvement. Check IQRs are comparable, then merge.
< 0.05	~0.5	Significant but negligible. Check if worth the added complexity.
> 0.05	> 0.6	Promising but underpowered. Re-run with more trials (e.g., 50).
> 0.05	~0.5	No effect. Try an advanced snapshot or ground-truth evaluation.
any	< 0.5	B underperforms A. If significant, reject or redesign the feature.

Time-series caveat: If the IQR bands overlap for most of the campaign and only diverge near the end, treat the final-coverage result cautiously — late divergence may reflect noise rather than a sustained advantage.

3. Visualizations

Note: In the box plots below, the central box represents the Interquartile Range (IQR, the middle 50% of trials), demonstrating the consistency of the fuzzer's performance. The internal line represents the median.

Target: cln

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

Target: ldk

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

Target: lnd

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

By the way, I had also included Eclair in the evaluation but I think I messed up the orchestrator for it. Here is the summary table including Eclair:

1. Summary Statistics

Target	Duration (h)	n (Baseline)	n (Exp.)	Median Cov. (Baseline)	Median Cov. (Exp.)	Adj. p-value (Cov.)	Â12 (Cov.)	Median AUC (Baseline)	Median AUC (Exp.)	Adj. p-value (AUC)	Â12 (AUC)	Union Cov. (Baseline)	Union Cov. (Exp.)	Execs/s (Baseline)	Execs/s (Exp.)
cln	12	20	20	3617.5	3641.5	0.0659624	0.7225	42704.7	43344.8	0.00088881	0.8425	5750	5071	49.025	43.57
eclair	12	20	20	7285.5	7241	1	0.5625	81812.3	83814.8	0.272722	0.6575	8455	8541	12.72	17.82
ldk	12	20	20	12611.5	12628.5	1	0.50125	144049	139023	0.523233	0.395	14552	14983	170.48	97.04
lnd	12	20	20	25675.5	25675	1	0.41875	307312	307591	0.635945	0.455	25753	25764	88.195	81.885

A comprehensive version of this table including raw P-values and Interquartile Ranges (IQRs) is available in evaluation_metrics.csv.

Chand-ra commented Jun 29, 2026

View reviewed changes

morehouse reviewed Jun 30, 2026

View reviewed changes

Chand-ra force-pushed the splice branch from cbba9e2 to a96cd41 Compare July 1, 2026 06:25

morehouse reviewed Jul 6, 2026

View reviewed changes

erickcestari reviewed Jul 6, 2026

View reviewed changes

Chandra Pratap added 4 commits July 7, 2026 04:43

smite-ir-mutator: Make AFL++ populate add_buf

3788d62

The following commit will implement `SpliceInsertionMutator`, which uses the decoded program from this buffer. Additionally, add a test to ensure parsing a valid `add_buf` via postcard doesn't panic or corrupt the state.

smite-ir/mutators: Add SpliceInsertionMutator

40acf43

smite-ir/tests: Refactor GeneratorInsertionMutator tests

c58f118

Refactor the common logic from `GeneratorInsertionMutator` tests that we will need for implementing `SpliceInsertionMutator` tests.

smite-ir/tests: Add tests for SpliceInsertionMutator

bc3691a

Chand-ra force-pushed the splice branch from a96cd41 to bc3691a Compare July 7, 2026 04:45

Chand-ra mentioned this pull request Aug 6, 2026

Effectiveness Evaluation Framework for Different Mutator Configurations #115

Open

+/// Mutators available to [`MutatorState::mutate_stacked`]. The pool of
+/// candidates is built per call, so optional entries (splice) are included by
+/// pushing a variant.
+#[derive(Clone, Copy)]
+enum StackedMutator {
+    OpParam,
+    InputSwap,
+    InstrDelete,
+    InstrReorder,
+    GenInsert,
+    Splice,
+}
+fn mutate_stacked(&mut self, program: &mut Program, splice: Option<Program>) {
+        self.last_sequence.clear();
+        // Power-of-two stack count: 1, 2, 4, 8, or 16 mutations.
+        let stack = 1u32 << self.rng.random_range(0..=4);
+        let splice_mutator = splice.map(SpliceMutator::new);
+        let mut pool = vec![
+            StackedMutator::OpParam,
+            StackedMutator::InputSwap,
+            StackedMutator::InstrDelete,
+            StackedMutator::InstrReorder,
+            StackedMutator::GenInsert,
+        ];
+        if splice_mutator.is_some() {
+            pool.push(StackedMutator::Splice);
+        }
+        for _ in 0..stack {
+            // Uniform pick between the available mutators.
+            let choice = *pool
+                .iter()
+                .choose(&mut self.rng)
+                .expect("pool is non-empty");
+            let name = match choice {
+                StackedMutator::OpParam => {
+                    OperationParamMutator.mutate(program, &mut self.rng);
+                    "op-param"
+                }
+                StackedMutator::InputSwap => {
+                    InputSwapMutator.mutate(program, &mut self.rng);
+                    "input-swap"
+                }
+                StackedMutator::InstrDelete => {
+                    InstructionDeleteMutator.mutate(program, &mut self.rng);
+                    "instr-delete"
+                }
+                StackedMutator::InstrReorder => {
+                    InstructionReorderMutator.mutate(program, &mut self.rng);
+                    "instr-reorder"
+                }
+                StackedMutator::GenInsert => {
+                    let generator = *AnyGenerator::ALL
+                        .iter()
+                        .choose(&mut self.rng)
+                        .expect("AnyGenerator::ALL is non-empty");
+                    let mutator = GeneratorInsertionMutator::new(generator);
+                    mutator.mutate(program, &mut self.rng);
+                    "gen-insert"
+                }
+                StackedMutator::Splice => {
+                    splice_mutator
+                        .as_ref()
+                        .expect("Splice is only pooled when a splice input exists")
+                        .mutate(program, &mut self.rng);
+                    "splice"
+                }
+            };
+            self.last_sequence.push(name);
+        }
+    }

Uh oh!

Conversation

Chand-ra commented Jun 29, 2026

Uh oh!

Chand-ra Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morehouse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chand-ra commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fuzzing Evaluation Report

1. Summary Statistics

2. Interpretation Guide

Key Metrics

Reading the Results

3. Visualizations

Target: cln

Median Coverage Over Time

Distribution Comparisons

Target: ldk

Median Coverage Over Time

Distribution Comparisons

Target: lnd

Median Coverage Over Time

Distribution Comparisons

1. Summary Statistics

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Chand-ra Jun 29, 2026 •

edited

Loading

Chand-ra commented Jul 27, 2026 •

edited

Loading