Skip to content

datagen: labelled time-series API for library use; drop the ERP export - #249

Merged
zzylol merged 2 commits into
mainfrom
feat/datagen-library-drop-erp
Nov 16, 2026
Merged

zzylol merged 2 commits into
mainfrom
feat/datagen-library-drop-erp

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: another workspace (the planner's devtools) needs synthetic labelled time series, but aqpbm-datagen only generates typed tables, and approxbench erp is an export with no consumer.
  • Change: add TimeSeriesDescription to aqpbm-datagen (per-label cardinalities with optional Zipf skew, sample interval, duration, an existing DataDistribution per series, counter mode, one seed), put TableDescription::from_path behind a new files feature, and delete the ERP export with its docs and scripts.
  • Effect: aqpbm-datagen = { git = "https://github.com/ProjectASAP/sketch-bench", rev = "cf8f86735e6f4c59d9827e3ae5d5bd28e8cd3ee1" } builds from an outside workspace with only serde, rand, rand_distr, rand_xoshiro and thiserror (no serde_json, serde_norway, sketch libraries, polars or CLI). Example: 50 series over host (100 values) x endpoint (8 values, Zipf 1.1), every 10 s for 60 s, gives 300 samples; 10^4 series x 360 samples stream at 7.2 M samples/s in 6.9 MB.

Review guide

  • Base: main. Not stacked.
  • Read first: aqpbm-datagen/src/timeseries.rs (new API, the two label-set algorithms, and its tests).
  • Mechanical: ERP deletion in aqpbm-cli (module, subcommand, main match arm); reduce_scores inlined into atomic_costs::reduce_one (same checks, same order); the shared error::bad helper; the files feature and the configs/datagen tests reading YAML through a test helper; doc and comment edits.

Behavior: before → after

Before: an outside crate wanting labelled series had to build a TableDescription per label, dedupe label sets, lay out timestamps and run a counter itself, and taking aqpbm-datagen also pulled in serde_json and serde_norway. After:

let generated = TimeSeriesDescription { labels, series_num: 50, start_ms, sample_interval_ms: 10_000,
    duration_ms: 60_000, value_distribution, counter: false, seed: 42 }.generate()?;
for sample in generated.samples() {
    let (labels, timestamp_ms, value) = sample?; // 300 samples, time-major
}

Why not reuse aqpbm-usecase::data: it already turns a use-case Spec into labelled series, but it depends on aqpbm-datagen with the import feature, so taking it pulls in polars, csv, chrono, flate2 and anyhow; its spec (metrics, groups, phases, {i} templates) is also richer than a library caller needs. The two generators differ in three conventions:

Convention aqpbm-usecase::data TimeSeriesDescription Here
First timestamp start_ms + interval (offsets 1..=duration/interval) start_ms (k * interval < duration) Kept: start_ms names the first sample directly and the grid is the half-open window [start_ms, start_ms + duration_ms); a caller wanting the usecase grid sets start_ms + interval.
Seed source the distribution's own seed seed ^ value_distribution.seed Aligned: both now use the distribution's seed.
Stream mixing seed ^ stream * 0x9E3779B97F4A7C15 splitmix64(seed ^ splitmix64(stream)) Kept: the splitmix finalizer has no linear collisions such as seed 0 / stream i against seed i * φ / stream 0, and switching either side changes recorded outputs.

Follow-up #250 makes usecase's synthetic path draw from these conventions.

Interfaces

aqpbm-datagen/src/timeseries.rs (new; re-exported from lib.rs)

+/// One label of a series: its name, how many values it takes, and how skewed the draws are over them.
+pub struct SeriesLabelSpec {
+    /// The label name, such as `host`.
+    pub name: String,
+    /// How many values the label takes: `{name}_0` to `{name}_{cardinality-1}`.
+    pub cardinality: u64,
+    /// Zipf skewness of each draw over the values by index; 0.0 (the default) is uniform.
+    /// The series returned keep this skew far below the product of the cardinalities
+    /// and approach uniform as `series_num` approaches it.
+    pub skewness: f64,
+}
+
+/// What to generate: which series, at which timestamps, with which values.
+pub struct TimeSeriesDescription {
+    /// The labels every series carries, in output order.
+    pub labels: Vec<SeriesLabelSpec>,
+    /// How many distinct label sets; at most the product of the cardinalities.
+    pub series_num: u64,
+    /// Timestamp of each series' first sample, in ms (default 0).
+    pub start_ms: i64,
+    /// Milliseconds between two samples of one series.
+    pub sample_interval_ms: u64,
+    /// Samples sit at start_ms + k * sample_interval_ms for k * sample_interval_ms < duration_ms.
+    pub duration_ms: u64,
+    /// Drawn independently per series; its own seed is mixed with `seed` (0 leaves `seed` in charge).
+    pub value_distribution: DataDistribution,
+    /// Value at sample k = sum of the first k + 1 draws, each floored at 0 (default false).
+    pub counter: bool,
+    /// Fixes the label sets and, with the distribution's seed, every value.
+    pub seed: u64,
+}
+
+impl TimeSeriesDescription {
+    /// Check the description without drawing; errors are `DataGenError::BadParam` (bad input).
+    pub fn validate(&self) -> Result<(), DataGenError>;
+    /// ceil(duration_ms / sample_interval_ms).
+    pub fn samples_per_series(&self) -> u64;
+    /// Draw the label sets; values come lazily from `samples()`. `BadParam` for bad input, or
+    /// when repeated draws under a very steep skew cannot find `series_num` distinct sets.
+    pub fn generate(&self) -> Result<GeneratedTimeSeries, DataGenError>;
+}
+
+/// The generated series; holds the description it came from (private).
+pub struct GeneratedTimeSeries {
+    /// Series i's (name, value) pairs, in `labels` order.
+    pub labels: Vec<Vec<(String, String)>>,
+}
+
+impl GeneratedTimeSeries {
+    /// Every sample, time-major; calling again replays the same samples.
+    pub fn samples(&self) -> TimeSeriesSamples<'_>;
+}
+
+/// (series labels, timestamp_ms, value).
+pub type TimeSeriesSample<'a> = (&'a [(String, String)], i64, f64);
+
+/// Iterator over the samples. Plain mode never fails; a counter whose sum passes 2^53
+/// yields one `Err(BadParam)` naming the series and timestamp, and the iterator ends.
+pub struct TimeSeriesSamples<'a>;
+impl<'a> Iterator for TimeSeriesSamples<'a> { type Item = Result<TimeSeriesSample<'a>, DataGenError>; }

aqpbm-datagen/src/table.rs, aqpbm-datagen/Cargo.toml

 impl TableDescription {
     /// Load a description from a `.yaml`/`.yml` (serde_norway) or otherwise JSON file.
+    /// Behind the `files` feature.
+    #[cfg(feature = "files")]
     pub fn from_path(path: &Path) -> Result<Self, DataGenError>;
 }

 [features]
 schema = ["dep:schemars"]
+# Load a description from a YAML or JSON file (`TableDescription::from_path`).
+files = ["dep:serde_json", "dep:serde_norway"]
-import = ["dep:anyhow", "dep:chrono", "dep:csv", "dep:flate2", "dep:polars"]
+import = ["files", "dep:anyhow", "dep:chrono", "dep:csv", "dep:flate2", "dep:polars"]
 [dependencies]
-serde_json.workspace = true
-serde_norway.workspace = true
+serde_json = { workspace = true, optional = true }
+serde_norway = { workspace = true, optional = true }

Every in-repo caller of from_path (aqpbm-cli) already enables import, which now implies files.

aqpbm-cli/src/cli.rs, aqpbm-cli/src/lib.rs, aqpbm-cli/src/atomic_costs.rs

 pub enum Cmd {
     Sketchbench(SketchbenchArgs),
     AtomicCosts(atomic_costs_cmd::AtomicCostsArgs),
-    /// Preserve distribution-conditioned error and resource measurements as ERP v1.
-    Erp(erp_cmd::ErpArgs),
     Flatten(flatten_cmd::FlattenArgs),
 }
-pub mod erp;
-pub mod erp_cmd;

 pub fn reduce_one(record: &MergedRecord, accuracy_metric: &str) -> Result<AtomicCostEntry, SkipReason>;
-/// [`reduce_one`] without naming the row's metric (`accuracy_metric` is empty), for a consumer that keeps every score (ERP).
-pub(crate) fn reduce_scores(record: &MergedRecord) -> Result<AtomicCostEntry, SkipReason>;

Removed because nothing reads ERP output: a grep for erp, erp_v1, erp_cmd and erp_shape across the repo found only the deleted files and doc mentions. reduce_scores existed only for ERP; its body now sits in reduce_one.

Usage (the rustdoc example in timeseries.rs, run as a doctest):

use aqpbm_datagen::{DataDistribution, ParetoParameter, SeriesLabelSpec, TimeSeriesDescription};

let description = TimeSeriesDescription {
    // 50 series over host (100 values, uniform) x endpoint (8 values, Zipf 1.1).
    labels: vec![
        SeriesLabelSpec { name: "host".into(), cardinality: 100, skewness: 0.0 },
        SeriesLabelSpec { name: "endpoint".into(), cardinality: 8, skewness: 1.1 },
    ],
    series_num: 50,
    start_ms: 1_700_000_000_000,
    // One sample every 10 s for 60 s: 6 samples per series.
    sample_interval_ms: 10_000,
    duration_ms: 60_000,
    // Heavy-tailed values of at least 1.0; seed 0 here leaves `seed` below in charge.
    value_distribution: DataDistribution::Pareto(ParetoParameter { alpha: 1.5, scale: 1.0, seed: 0 }),
    counter: false,
    seed: 42,
};
// generated.labels: 50 distinct label sets.
let generated = description.generate().unwrap();
let mut samples = 0;
for sample in generated.samples() {
    // Only a counter can fail (its sum passed 2^53); this one is plain.
    let (labels, timestamp_ms, value) = sample.unwrap();
    // labels: [("host", "host_17"), ("endpoint", "endpoint_0")]
    assert!((timestamp_ms - 1_700_000_000_000) % 10_000 == 0 && value >= 1.0);
    samples += 1;
}
assert_eq!(samples, 300);

Algorithm changes

Label sets

  • Problem: label sets are distinct, so near the product of the cardinalities repeated draws hunt for the last rare sets. Measured on the first revision: 4×10^5 of 5×10^5 sets took 2.1 s, 4.99×10^5 took 9.7 s, and 1000 sets out of 50 x 20 at skewness 3 failed with BadParam after 10^6 draws (322 found).
  • After (TimeSeriesDescription::generate):
    1. If the product of the cardinalities is at most 4 x series_num (WEIGHTED_SAMPLING_SETS_PER_SERIES), weighted_label_sets: give every set (i, j, ...) the weight w = (i+1)^-s_1 (j+1)^-s_2 ... and the key u^(1/w), u uniform; keep the series_num largest keys (select, then sort), largest first. Keys are compared as ln(-ln u) - ln w so tiny weights do not underflow.
    2. Otherwise drawn_label_sets: each label draws its index from Zipf(cardinality, skewness) on one seeded stream; keep each new set until series_num exist, giving up with BadParam after 1000 x series_num draws.
  • Correctness: both are weighted sampling without replacement with the same weights (Efraimidis and Spirakis: the key order is the order sequential draws would pick sets in), so the skew is that of the draws: far below the product the series keep it, near the product they approach uniform, and at the product every set is used once. Tests: far_below_the_label_set_count_the_series_keep_the_draw_skew, near_the_label_set_count_heavy_sets_are_taken_first, series_num_equal_to_the_label_set_count_always_succeeds.
  • Complexity: weighted path O(product x labels) time and O(product) memory, used only when the product is under 4 x series_num; drawn path unchanged, O(series_num x labels) expected. Measured (release, one core, labels 1000 x 100 x 5): 4×10^5 series 2.1 s → 0.44 s, 4.99×10^5 series 9.7 s → 0.53 s.
  • Limits: a very steep skew far from the product can still exhaust the draw budget (1000 x 100 at skewness 3, n=2000: BadParam after 2×10^6 draws, 0.9 s), with a message that names the fix.

Values

Series i's values come from its own stream splitmix64(s ^ splitmix64(i)) with s = seed ^ value_distribution.seed, so series are independent and replays are deterministic. Counter mode reuses rule::MonotonicAcc. O(1) per sample, 7.2 M samples/s.

Decisions and open questions

Needs reviewer decision: none.

Decided:

  • 2026-11-15, value seed: mix the distribution's own seed into each series' stream, stream_seed(seed ^ value_distribution.seed(), i). A seed of 0 keeps the output of the first revision. Rejected: ignoring it silently (a validated field doing nothing), refusing a nonzero one.
  • 2026-11-15, counter overflow: samples() yields Result items; a counter past 2^53 yields one Err naming the series and timestamp, then the iterator ends. Rejected: refusing up front, which can only bound bounded distributions (Uniform, Zipf); Pareto and Normal, the usual counter increments, would have to be refused outright or still stop silently.
  • 2026-11-15, label skew: document that the skew is that of the draws, and choose label sets by weighted sampling without replacement when the product is under 4 x series_num.
  • 2026-11-15, names: value → value_distribution, LabelSpec → SeriesLabelSpec (the crate root already uses "label" for a column name); series_num kept to match row_num/column_num.
  • 2026-11-15, label values render as {name}_{index} (for example host_17); the index is the Zipf rank, so _0 is the most frequent value. Rejected: random StringOpts strings, which add nothing for series labels.
  • 2026-11-15, first timestamp and stream mixing differ from aqpbm-usecase::data on purpose (table above); aqpbm-usecase: draw synthetic series from aqpbm-datagen's TimeSeriesDescription conventions #250 follows up.
  • 2026-11-15, from_path moves behind a files feature so a library caller does not take serde_json and serde_norway.

Verification

Scope Result Notes
cargo test -p aqpbm-datagen (default features) 62 unit + 1 doctest passed 11 tests in timeseries.rs
cargo test -p aqpbm-datagen --all-features 69 unit + 1 doctest passed includes import tests
cargo test -p aqpbm-cli 39 passed 27 unit + 12 integration
cargo clippy --workspace --all-targets --all-features -- -D warnings clean also -p aqpbm-datagen -p aqpbm-cli with default features, and -p aqpbm-datagen --no-default-features
cargo fmt --all --check clean
outside workspace, git dep at cf8f86735e6f4c59d9827e3ae5d5bd28e8cd3ee1 builds and runs cargo tree: 22 crates (40 in the first revision): serde, rand, rand_distr, rand_xoshiro, thiserror and their dependencies; no serde_json, serde_norway, polars, sketch library or clap

New tests (written failing first where the behavior changed):

  • the_value_distributions_seed_is_mixed_into_the_values_only: series i's first value is stream i under seed ^ value seed; a value seed of 0 matches seed alone, 3 changes every value and no label set.
  • a_counter_past_two_to_the_53_ends_with_an_error: Pareto(α=0.3) counter, 10 series x 100,000 samples, ends with one Err naming the series and 2^53 as its last item (the first revision returned 41,249 samples and no error).
  • series_num_equal_to_the_label_set_count_always_succeeds: 1000 of 50 x 20 sets at skewness 3 (the first revision failed with 322 found).
  • far_below_the_label_set_count_the_series_keep_the_draw_skew: 200 of 5000 sets, code_0 on about 57% of the series (Zipf 1.5 over 5 values).
  • near_the_label_set_count_heavy_sets_are_taken_first: 100 of 200 sets via the weighted path, heavy code_0 sets mostly taken, light code_4 sets mostly left.
  • Unchanged from the first revision: the_same_description_generates_the_same_samples_and_the_seed_changes_them, every_series_is_distinct_and_sampled_on_the_interval, a_counter_is_the_running_sum_of_the_plain_draws, bad_descriptions_are_refused_by_name, no_labels_means_one_series, a_description_round_trips_through_yaml, and the doctest.

Outside-consumer checks (separate workspace, release):

  • Distinct label sets: n=150 and n=210 over 10 x 7 x 3 give n distinct sets; n=211 is refused.
  • Label skew at 20,000 of 5×10^7 sets: fitted log-log slope 0.006 / 0.817 / 1.088 / 1.516 for skewness 0 / 0.8 / 1.1 / 1.5. Near the product (50 x 10 at 1.1): slope 0.50 at n=250, 0.10 at n=450, 0.00 at n=500, as documented.
  • Grid: (interval, duration) = (7,100), (10,100), (10,101), (1000,1), (10,9) give 15, 10, 11, 1, 1 samples per series, time-major, exact size_hint.
  • Counter, Normal(0.5, 2): never decreases, equals the running sum of the draws floored at 0. Pareto(0.3): 45,342 Ok samples, then one Err ("the counter of series 2 passed 2^53 at timestamp 46340 ms ...") as the last item.
  • Values (2×10^5 each) pass KS at 1%: Uniform[2,5) D=0.0019, Normal(10,3) D=0.0024, Pareto(3,2) D=0.0019; Zipf(1.1,100) χ²=105.7 (critical 135.8).
  • Seeds: same seed bit-identical and replayable; seed 5 vs 6 changes labels and all 1200 values; changing the distribution's seed keeps the labels and changes all 1200 values.
  • Throughput (one core, Pareto values, 3 labels): 10^4 x 360 at 7.2 M samples/s (counter 7.1 M/s), peak 6.7 MB; 10^5 x 100 at 7.2 M/s, peak 46.8 MB; 10 x 10^6 at 7.3 M/s, 1.3 MB. Unchanged by the Result items.

Review status

Round 1 (quality and functional reviews of the first revision, 74ef8d0) found nothing blocking. Addressed here:

Finding Resolution
A1: a second generator beside aqpbm-usecase::data, conventions differ "Why not reuse" and the convention table above; seed source aligned; #250 filed
A2: reduce_scores left over from ERP inlined into reduce_one, "(ERP)" doc gone
A3: GeneratedTimeSeries copied five description fields holds the description
A4 / F1: counter stopped silently on overflow Result items; test with Pareto(α=0.3)
A5: duplicated bad() one pub(crate) fn bad in error.rs
A6: MonotonicAcc primed with push(0.0) kept, as the review suggested; comment corrected
F2: skew flattens near the product documented; E2 weighted sampling below 4 x series_num
F3: very steep skew can fail unchanged far from the product (clear error); fixed near it by E2
F4: "(ERP)" comment gone with A2
E1 / E3: label memory, per-sample divide not done: they matter only at 10^6+ series, and the per-sample cost is dominated by the draw
E2: weighted sampling near the product done
L1: dependency list wrong in lib.rs and docs corrected, per feature
L2: serde_json / serde_norway needed only for from_path files feature
L3: crate description mentions time series
L4: names value_distribution, SeriesLabelSpec
samples() comment about the first counter draw says the first sample is the first draw floored at 0
Open question: value seed decided, mixed
Open question: {name}_{index} decided, kept

Follow-ups

  • aqpbm-usecase: draw synthetic series from aqpbm-datagen's TimeSeriesDescription conventions #250: aqpbm-usecase's synthetic path draws from TimeSeriesDescription conventions (grid and stream seed in one place).
  • aqpbm-planeval is left unchanged here; its executor driver will be replaced later.
  • Deleted: aqpbm-cli/src/erp.rs, aqpbm-cli/src/erp_cmd.rs, docs/erp-v1-workloads.md, scripts/sketch_runs/run_erp_v1_figure1.sh, scripts/sketch_runs/analyze_erp_v1_figure1.py, the ERP section of docs/aqpbm-cli.md, and the ERP-provenance text in configs/workloads/README.md and burst.rs.

@zzylol zzylol added enhancement New feature or request documentation Improvements or additions to documentation labels Nov 15, 2026
@zzylol
zzylol requested a review from GordonYuanyc November 16, 2026 16:00
Add aqpbm_datagen::TimeSeriesDescription: series_num distinct label sets
drawn from per-label cardinalities with optional Zipf skew, sampled every
sample_interval_ms over duration_ms, values from an existing
DataDistribution, optional counter mode, one seed. samples() yields
(labels, timestamp_ms, value) time-major. The crate's default features
need only serde, rand and thiserror, so another workspace can depend on
it by git rev.

Delete approxbench erp (erp.rs, erp_cmd.rs, CLI wiring), its methodology
doc, its run/analysis scripts and the doc references: nothing consumes it.
…weighted label sets

- Series values mix the distribution's own seed: stream_seed(seed ^ value_distribution.seed(), i).
  A distribution seed of 0 keeps the previous output.
- samples() yields Result items: a counter past 2^53 yields one error naming the series and
  timestamp, then the iterator ends, instead of stopping silently.
- Label skew is documented as the skew of the draws. When the product of the cardinalities is
  at most 4 x series_num, label sets come from weighted sampling without replacement
  (keys u^(1/w)), which always succeeds and is 4-18x faster near the product.
- GeneratedTimeSeries holds its description; one shared bad() helper.
- Renames: value -> value_distribution, LabelSpec -> SeriesLabelSpec.
- TableDescription::from_path, serde_json and serde_norway move behind a `files` feature
  (implied by `import`); dependency lists and crate description corrected.
- atomic_costs: reduce_scores inlined into reduce_one.
@zzylol
zzylol force-pushed the feat/datagen-library-drop-erp branch from cf8f867 to 6236d43 Compare November 16, 2026 20:39
@zzylol
zzylol merged commit 7bb6af2 into main Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant