Add reservation benchmark by jerbaroo · Pull Request #151 · channable/opsqueue

jerbaroo · 2026-07-23T16:33:38Z

This PR adds a benchmark which can be run with just bench-chunks-select. It produces the SVG contained in this PR. Output from the Rust bench is written to a CSV (not committed) which is then read by Python to produce a graph. The Rust bench also prints the information to stdout, which looks like:

SHAPE                          STRATEGY                            SIZE         MEDIAN     P10 / P90
---------------------------------------------------------------------------------------------------------
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=100:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 10
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 100          56.5       51.4 / 65.9
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 100          111.1      103.1 / 130.8
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=500:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 50
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 500          55.8       53.1 / 66.6
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 500          182.6      174.3 / 256.1
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=1000:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 100
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 1000         55.9       52.1 / 63.2
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 1000         344.3      260.8 / 384.3
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=2000:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 200
Beginning benchmark of strategy Random
FewSubmissionsManyChunks       Random                                 2000         55.3       50.9 / 62.7
Beginning benchmark of strategy PreferDistinct { meta_key: "metadata_value", underlying: Oldest }
FewSubmissionsManyChunks       PreferDistinct(metadata_value, Oldest) 2000         499.6      438.2 / 681.2
Seeding database with shape=FewSubmissionsManyChunks and total_chunks=4000:
    Metadata values: 10
    Submissions per metadata value: 1
    Chunks per submission: 400
Beginning benchmark of strategy Random

ReinierMaas

I didn't review opsqueue/benches/chunks_select.rs yet, only the top block comment. But I think I already have some good advice that can be helpful.

ReinierMaas · 2026-07-24T15:27:25Z

        pythonEnv
      ];
+      # Fixes: libstdc++.so.6: cannot open shared object file: No such file or directory
+      LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc.lib ];


Which dependency needed this? And at what time of it lifecycle build or runtime?

Just enabling the benchmarks probably caused it to show-up for you to add it here. What I am thinking is should we also ensure that our packaged opsqueue server has this library path available.

This is only added to the pkgs.mkShell, it fixes the following error from attempting to plot the benchmark results with matplotlib:

Traceback (most recent call last): File "/home/jeremy-barisch-rooney/channable/opsqueue/opsqueue/benches/plot_chunks_select.py", line 10, in <module> import matplotlib.pyplot as plt # noqa: E402 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jeremy-barisch-rooney/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/matplotlib/__init__.py", line 161, in <module> from . import _api, _version, cbook, rcsetup File "/home/jeremy-barisch-rooney/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/matplotlib/cbook.py", line 24, in <module> import numpy as np File "/home/jeremy-barisch-rooney/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/numpy/__init__.py", line 109, in <module> from numpy.__config__ import show_config File "/home/jeremy-barisch-rooney/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/numpy/__config__.py", line 4, in <module> from numpy._core._multiarray_umath import ( ...<3 lines>... ) File "/home/jeremy-barisch-rooney/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/numpy/_core/__init__.py", line 85, in <module> raise ImportError(msg) from exc ImportError: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: * The Python version is: Python 3.13 from "/home/jeremy-barisch-rooney/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin/python" * The NumPy version is: "2.5.1" and make sure that they are the versions you expect. Please carefully study the information and documentation linked above. This is unlikely to be a NumPy issue but will be caused by a bad install or environment on your machine. Original error was: libstdc++.so.6: cannot open shared object file: No such file or directory

Address comments

ReinierMaas

LGTM

I left a few smaller changes you could still make.

ReinierMaas · 2026-07-31T12:03:55Z

+  cd libs/opsqueue_python
+  source "./.setup_local_venv.sh"
+  cd -


It's painfull that we can't source ./libs/opsqueue_python/setup_local_venv.sh directly without cding to the correct working directory...

ReinierMaas · 2026-07-31T14:03:26Z

+    if let Some((
+        current_metadata_values_,
+        current_submissions_per_metadata_value,
+        current_chunks_per_submission,
+    )) = current_layout
+    {
+        // We can only extend if the shape of the data is compatible with the current inserted data.
+        if current_submissions_per_metadata_value == submissions_per_metadata_value
+            && current_chunks_per_submission == chunks_per_submission
+        {
+            we_are_extending = true;
+            current_metadata_values = current_metadata_values_;
+        }
+    }


This can be written with fewer mutations:

Suggested change

if let Some((

current_metadata_values_,

current_submissions_per_metadata_value,

current_chunks_per_submission,

)) = current_layout

{

// We can only extend if the shape of the data is compatible with the current inserted data.

if current_submissions_per_metadata_value == submissions_per_metadata_value

&& current_chunks_per_submission == chunks_per_submission

{

we_are_extending = true;

current_metadata_values = current_metadata_values_;

}

}

let (we_are_extending, current_metadata_values) = if let Some((

current_metadata_values,

current_submissions_per_metadata_value,

current_chunks_per_submission,

)) = current_layout

&& current_submissions_per_metadata_value == submissions_per_metadata_value

&& current_chunks_per_submission == chunks_per_submission

{

(true, current_metadata_values)

} else { (false, 0) }

ReinierMaas · 2026-07-31T14:22:05Z

+    dispatcher: &Dispatcher,
+) -> ChunkId {
+    use tokio::sync::mpsc::unbounded_channel;
+    let (notifier, _) = unbounded_channel();


Using an underscore here _ drop the receiving side instantly. The sending side will receive errors when it tries to send anything over this channel. I haven't looked at the underlying code but does the reservation hold if we drop this receiving side of the channel? We could opt to return the receiving side together with the ChunkId and retain them until we finish our reservation to ensure that we can receive messages from the view of the reserver.

ReinierMaas · 2026-07-31T14:23:34Z

+    if let [(chunk, _)] = reserved.as_slice() {
+        ChunkId::from((chunk.submission_id, chunk.chunk_index))


What are we dropping here? Should we retain that until we finish our chunk reservation?

ReinierMaas · 2026-07-31T14:24:51Z

+    // For each of the samples we want to collect.
+    for _ in 0..SAMPLES_PER_STAT {
+        let mut sample_reservation_durations = Vec::with_capacity(RESERVATIONS_IN_SAMPLE);
+        let mut chunks_reserved = Vec::new();


Suggested change

let mut chunks_reserved = Vec::new();

let mut chunks_reserved = Vec::with_capacity(RESERVATIONS_IN_WARMUP + RESERVATIONS_IN_SAMPLE);

You can also combine this with allocating the vector once and using .drain(..) when looping over the reserved chunks to retain the vectors allocation.

ReinierMaas · 2026-07-31T14:33:28Z

+    let runtime = tokio::runtime::Runtime::new().unwrap();
+    let csv_path = PathBuf::from("benches/chunks_select_bench.csv");
+    if let Some(parent) = csv_path.parent() {
+        let _ = std::fs::create_dir_all(parent);


You probably just want to `.unwrap()`` here. This should succeed if the parents exist and you want to early out if it fails to create the parent directories.

jerbaroo requested a review from ReinierMaas July 23, 2026 16:33

jerbaroo mentioned this pull request Jul 23, 2026

PreferDistinct: faster and fairer #148

Draft

Add reservation benchmark

da93859

jerbaroo force-pushed the jerbaroo-benchmark branch from 7592f8f to da93859 Compare July 23, 2026 20:16

ReinierMaas requested changes Jul 24, 2026

View reviewed changes

jerbaroo force-pushed the jerbaroo-benchmark branch from 46043aa to bde9a83 Compare July 28, 2026 16:28

jerbaroo requested a review from ReinierMaas July 30, 2026 08:46

fixup! Add reservation benchmark

dfed300

Address comments

jerbaroo force-pushed the jerbaroo-benchmark branch from 2dfa1ef to dfed300 Compare July 30, 2026 09:50

ReinierMaas approved these changes Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add reservation benchmark - #151

Add reservation benchmark#151
jerbaroo wants to merge 2 commits into
masterfrom
jerbaroo-benchmark

jerbaroo commented Jul 23, 2026 •

edited

Loading

Uh oh!

ReinierMaas left a comment

Uh oh!

Uh oh!

ReinierMaas Jul 24, 2026

Uh oh!

jerbaroo Jul 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ReinierMaas left a comment

Uh oh!

ReinierMaas Jul 31, 2026

Uh oh!

ReinierMaas Jul 31, 2026

Uh oh!

ReinierMaas Jul 31, 2026

Uh oh!

ReinierMaas Jul 31, 2026

Uh oh!

ReinierMaas Jul 31, 2026

Uh oh!

ReinierMaas Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if let [(chunk, _)] = reserved.as_slice() {
		ChunkId::from((chunk.submission_id, chunk.chunk_index))

	let mut chunks_reserved = Vec::new();
	let mut chunks_reserved = Vec::with_capacity(RESERVATIONS_IN_WARMUP + RESERVATIONS_IN_SAMPLE);

Uh oh!

Conversation

jerbaroo commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ReinierMaas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ReinierMaas Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

jerbaroo Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ReinierMaas left a comment

Choose a reason for hiding this comment

Uh oh!

ReinierMaas Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

ReinierMaas Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

ReinierMaas Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

ReinierMaas Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

ReinierMaas Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

ReinierMaas Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jerbaroo commented Jul 23, 2026 •

edited

Loading

jerbaroo Jul 28, 2026 •

edited

Loading