Skip to content

datagen: one sample grid and one stream seed for every generator - #255

Merged
zzylol merged 2 commits into
mainfrom
datagen/one-sample-grid
Nov 16, 2026
Merged

zzylol merged 2 commits into
mainfrom
datagen/one-sample-grid

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: aqpbm-usecase and aqpbm-datagen's TimeSeriesDescription generate different samples from the same kind of description. With a 10 s interval over 60 s, TimeSeriesDescription samples at 0, 10, …, 50 s, so in left-open evaluation windows (t − 10 s, t] at t = 10, …, 60 s the sample at 0 falls in no window and the window ending at 60 s is empty: 5 of 6 samples are counted. The use-case path samples at 10, …, 60 s, and the two paths seed their per-series streams differently.
  • Change: one sample grid, (start, start + duration] (sample_offsets_ms), and one per-series stream seed, splitmix64(seed ^ splitmix64(stream)) (series_stream_seed), both public in aqpbm-datagen. aqpbm-usecase draws its synthetic series through them and keeps its own label templates and phases on top.
  • Effect: a use-case group of 3 series from Normal(0.45, 0.08) with seed 22 and a TimeSeriesDescription with one label of cardinality 3, seed: 0 and the same distribution now give the same timestamps and bit-identical values per series index. Every synthetic use-case output changes (table below).

Closes #250.

Review guide

  • Base: main (datagen: labelled time-series API for library use; drop the ERP export #249 is merged). Two commits: 213a13e adds the failing tests (the two helpers are todo!() stubs there), adc863e is the implementation.
  • Read first: aqpbm-datagen/src/timeseries.rs (sample_offsets_ms, series_stream_seed, validate, samples_per_series, the iterator's timestamp), then aqpbm-usecase/src/data.rs (mixed_seed deleted; the no-rate branch of offsets).
  • Careful review: the new uc1 hash in aqpbm-usecase/tests/uc1.rs, and the use-case output changes below.
  • Mechanical: doc text in docs/aqpbm-datagen.md.

Behavior: before → after

// One label of cardinality 3, 500 ms samples over 5 s from Normal(0.45, 0.08) with seed 22.
let description = TimeSeriesDescription {
    labels: vec![SeriesLabelSpec { name: "host".into(), cardinality: 3, skewness: 0.0 }],
    series_num: 3,
    start_ms: 1_767_225_600_000,
    sample_interval_ms: 500,
    duration_ms: 5_000,
    value_distribution: DataDistribution::Normal(NormalParameter { mean: 0.45, standard_deviation: 0.08, seed: 22 }),
    counter: false,
    seed: 0,
};

The use-case spec with "count": 3, "sample_interval_ms": 500, "duration_ms": 5000 and the same baseline distribution:

before after
TimeSeriesDescription series 0 timestamps start + 0, +500, …, +4500 (10 samples) start + 500, …, +5000 (10 samples)
use-case series 0 timestamps start + 500, …, +5000 unchanged
series 0 values differ: seed ^ stream · 0x9E37… (use case) vs splitmix64(seed ^ splitmix64(stream)) bit-identical, both through series_stream_seed(22, 0)
duration_ms 9 at sample_interval_ms 10 1 sample at start_ms Err: duration_ms (9) is shorter than one sample_interval_ms (10), so no sample fits

This serves comparisons between engines: they are only comparable on identical data, and the harness evaluates over left-open windows.

Interfaces

aqpbm-datagen/src/timeseries.rs, aqpbm-datagen/src/lib.rs

-/// The seed of one independent stream of draws under the description's seed.
-fn stream_seed(seed: u64, stream: u64) -> u64;
+/// The seed of stream `stream`'s random generator under `seed`: `splitmix64(seed ^ splitmix64(stream))`.
+/// Every generator in the workspace seeds its per-series streams through this.
+pub fn series_stream_seed(seed: u64, stream: u64) -> u64;
+/// Offsets in ms from the window start of one series' samples: one interval, two intervals, ...,
+/// up to and including `duration_ms`, so the samples cover `(start, start + duration]`.
+/// Example: interval 10_000, duration 60_000 -> 10_000, 20_000, ..., 60_000.
+pub fn sample_offsets_ms(sample_interval_ms: u64, duration_ms: u64) -> impl Iterator<Item = u64>;

 pub struct TimeSeriesDescription {
-    /// The timestamp of each series' first sample, in milliseconds.
+    /// The start of the generated window, in milliseconds (default 0). The first sample is one interval after it.
     pub start_ms: i64,
-    /// Samples sit at `start_ms + k * sample_interval_ms` for every `k` with `k * sample_interval_ms < duration_ms`.
+    /// Samples sit at `start_ms + k * sample_interval_ms` for every `k >= 1` with `k * sample_interval_ms <= duration_ms`. At least one interval.
     pub duration_ms: u64,
 }
 impl TimeSeriesDescription {
-    /// How many samples each series has: `ceil(duration_ms / sample_interval_ms)`.
+    /// How many samples each series has: `floor(duration_ms / sample_interval_ms)`.
     pub fn samples_per_series(&self) -> u64;
 }

 // lib.rs
 pub use timeseries::{
-    GeneratedTimeSeries, SeriesLabelSpec, TimeSeriesDescription, TimeSeriesSample,
-    TimeSeriesSamples,
+    sample_offsets_ms, series_stream_seed, GeneratedTimeSeries, SeriesLabelSpec,
+    TimeSeriesDescription, TimeSeriesSample, TimeSeriesSamples,
 };

aqpbm-usecase/src/data.rs: private only. mixed_seed is deleted; replaced by series_stream_seed.

Usage (from aqpbm-usecase/tests/datagen_conventions.rs):

// The use-case spec above, read the way the systems runner reads it.
let usecase = aqpbm_usecase::data::generate(&spec, Path::new(".")).unwrap();
// The same three series through aqpbm-datagen; samples are time-major, so series i is every third sample.
let generated = description.generate().unwrap();
let mut by_series: Vec<Vec<(i64, u64)>> = vec![Vec::new(); 3];
for (index, sample) in generated.samples().enumerate() {
    let (_, t, v) = sample.unwrap();
    by_series[index % 3].push((t, v.to_bits()));
}
// Same timestamps, bit-identical values; only the label names differ (`host=h0` in the use case, a drawn `host=host_<n>` in the description).
assert_eq!(usecase[0].points.iter().map(|&(t, v)| (t, v.to_bits())).collect::<Vec<_>>(), by_series[0]);

Algorithm changes

None in the sense of search or cost. The sample grid and the seed mix change as described above; rate phases keep their own offsets, which already start after the window start.

Use-case outputs that change

Every synthetic spec, digested with aqpbm_usecase::data::generate before (7bb6af2) and after (adc863e). FNV-1a over timestamps and over value bits. _real specs read imported tables and do not change.

spec series samples timestamps values mean value before → after
uc1_syn_flood 1 → 1 11,340,033 → 11,340,033 unchanged changed (hash 0x0ab5_805d_18d8_d702 → 0xd704_cd23_c966_dead) 1,032,468,599.40 → 1,032,936,255.24
uc2_autoscaling 400 → 400 5,760,000 → 5,760,000 unchanged changed 430,428,530.08 → 430,463,478.55
uc3_video 360 → 360 3,599,825 → 3,599,825 unchanged changed 56.6753 → 56.7128
uc4_canary 6 → 6 6,315,788 → 6,315,788 changed (requests splits samples by the drawn code) changed 0.538212 → 0.538253
uc5_top_clients 98,860 → 98,798 3,600,000 → 3,600,000 changed (series are split by the drawn client_id) unchanged (all 1.0) 1.0 → 1.0

The use-case grid was already (start, start + duration], so no use-case timestamp moves except through label_from_value splits. The phase 2 and phase 3 results recorded under docs/ were measured on the old samples and are stale until rerun; they are not regenerated here.

Decisions and open questions

  • Decided: the first sample is one interval after start_ms, so samples fill (start, start + duration], matching left-open evaluation windows (rejected: keep TimeSeriesDescription's [start, start + duration) and move every evaluation time one interval earlier). The stream seed is TimeSeriesDescription's splitmix mix; mixed_seed is deleted (rejected: keep the linear mix, which gives (seed 0, stream 1) and (seed 0x9E37_79B9_7F4A_7C15, stream 0) the same seed).
  • Needs reviewer decision: None.

Verification

Scope Result Newly failing vs base Notes
cargo test -p aqpbm-datagen 68 passed none 6 new tests, 3 updated
cargo test -p aqpbm-datagen --all-features 75 passed + 1 doctest none
cargo test -p aqpbm-usecase 7 passed none 2 new, 1 renamed with a new hash
cargo fmt --all --check pass
cargo clippy -p aqpbm-datagen -p aqpbm-usecase --all-targets --all-features -- -D warnings pass

New tests:

  • sample_offsets_are_one_interval_apart_and_end_at_the_duration: 10 s over 60 s gives 10 … 60 s.
  • a_duration_that_is_not_a_multiple_of_the_interval_stops_at_the_last_whole_interval: (7, 100) gives 14 offsets ending at 98; (10, 101) gives 10 ending at 100.
  • a_duration_shorter_than_one_interval_is_refused_by_name: (1000, 1) and (10, 9) are refused, naming both fields.
  • time_series_samples_sit_on_the_left_open_grid: 50 series, 6 samples each at start + 10 s … start + 60 s.
  • every_sample_falls_in_exactly_one_left_open_window: each window (t − 10 s, t] holds exactly one sample of each series.
  • series_stream_seed_is_the_splitmix_mix_and_has_no_linear_collision: pins the mix and the collision the old use-case seed had.
  • usecase_and_time_series_description_draw_the_same_samples: the example above.
  • a_usecase_series_draws_from_the_shared_stream_seed: series 0's first value is the first draw from series_stream_seed(22, 0).
  • the_use_case_1_spec_draws_a_fixed_sample_stream (renamed): still 11,340,033 samples, new hash.

Updated: every_series_is_distinct_and_sampled_on_the_interval and no_labels_means_one_series (95 ms at 10 ms is now 9 samples at 1010 … 1090, not 10 at 1000 … 1090), and bad_descriptions_are_refused_by_name (adds the short-duration case).

Failing output at the tests commit 213a13e:

$ cargo test -p aqpbm-datagen --lib
test timeseries::tests::series_stream_seed_is_the_splitmix_mix_and_has_no_linear_collision ... FAILED   (todo!)
test timeseries::tests::sample_offsets_are_one_interval_apart_and_end_at_the_duration ... FAILED        (todo!)
test timeseries::tests::a_duration_that_is_not_a_multiple_of_the_interval_stops_at_the_last_whole_interval ... FAILED (todo!)
test timeseries::tests::the_value_distributions_seed_is_mixed_into_the_values_only ... FAILED           (todo!)
test timeseries::tests::no_labels_means_one_series ... FAILED                 left: 10, right: 9
test timeseries::tests::bad_descriptions_are_refused_by_name ... FAILED       duration 9 at interval 10 accepted
test timeseries::tests::a_duration_shorter_than_one_interval_is_refused_by_name ... FAILED  accepted
test timeseries::tests::every_sample_falls_in_exactly_one_left_open_window ... FAILED  sample at 1700000000000 is in windows []
test timeseries::tests::every_series_is_distinct_and_sampled_on_the_interval ... FAILED  left: 300, right: 270
test timeseries::tests::time_series_samples_sit_on_the_left_open_grid ... FAILED  first sample at start_ms
test result: FAILED. 58 passed; 10 failed

$ cargo test -p aqpbm-usecase --test datagen_conventions
test usecase_and_time_series_description_draw_the_same_samples ... FAILED
  left:  [(1767225600500, 4600895163899444392), (1767225601000, ...), ...]   use case
  right: [(1767225600000, 4601161766480322653), (1767225600500, ...), ...]   TimeSeriesDescription
test a_usecase_series_draws_from_the_shared_stream_seed ... FAILED   (todo!)

$ cargo test -p aqpbm-usecase --test uc1
test the_use_case_1_spec_draws_a_fixed_sample_stream ... FAILED
  left: 771664048501085954     (0x0ab5_805d_18d8_d702, the old stream)
  right: 15493734171647860397  (0xd704_cd23_c966_dead)

Not run: aqpbm-planeval and aqpbm-systemeval. They call aqpbm_usecase::data::generate and compare exact against planned answers on the same data, so they see the new samples but have no golden values.

Follow-ups

The samples of TimeSeriesDescription and of aqpbm-usecase's synthetic
series should sit on the left-open grid (start, start + duration], and
both should seed per-series streams through series_stream_seed. The two
helpers are todo!() stubs here; the next commit implements them. Updates
the grid expectations of existing tests (95 ms at 10 ms is now 9 samples)
and records the new uc1 sample-stream hash.
TimeSeriesDescription now samples at start + k * interval for k >= 1 and
k * interval <= duration, the left-open grid (start, start + duration]
that evaluations use, and refuses a duration below one interval.
sample_offsets_ms and series_stream_seed are public, and aqpbm-usecase's
synthetic path draws through them instead of its own grid and mixed_seed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aqpbm-usecase: draw synthetic series from aqpbm-datagen's TimeSeriesDescription conventions

1 participant