diff --git a/.gitignore b/.gitignore index 3bf36d2..fcb4082 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ build/ .vscode/ .venv/ benchmarks/results/ +/.worktrees/ diff --git a/README.md b/README.md index 58dd6c7..31d2d0a 100644 --- a/README.md +++ b/README.md @@ -162,39 +162,37 @@ One exception is deliberate. The `training` flag stays a plain runtime boolean, ## Benchmarks -All figures below come from the same CNN, the same CIFAR-10 batches, the same seed, and the same hardware (AMD RX 9070 XT, Intel Core Ultra 265K). Only the dispatch path differs. +All figures below come from the same CNN, the same seed, the same fp16 precision policy, and the same hardware (AMD RX 9070 XT, Intel Core Ultra 265K). The dispatch path differs, and with it Conv2d's precision: only the fused path can run Conv2d in fp16, so the other two compute it in fp32. The scripts and saved results are in [`benchmarks/`](benchmarks/README.md). ### Convergence -Four paths are tested; NumPy on CPU, legacy pre-refactor GPU implementation, vectorized CuPy without custom kernels, and the fused kernel path. +Three paths are tested; NumPy on CPU, vectorized CuPy without custom kernels, and the fused kernel path. - - - Backend convergence + + + Backend convergence ### Throughput -Per training step: **3.17 ms** on the fused kernel path vs **52.6 ms** on vectorized CuPy on the same GPU, with **326 ms** for NumPy on CPU. With the GPU paths, the kernel work shows a **17x speedup against identical hardware running array operations**. +Per training step: **4.85 ms** on the fused kernel path vs **202 ms** on vectorized CuPy on the same GPU, with **1,865 ms** for NumPy on CPU. With the GPU paths, the kernel work shows a **42x speedup against identical hardware running array operations**. - - - Backend throughput + + + Backend throughput ### Transient VRAM -Transient VRAM footprint over two training steps. Unfused CuPy dispatches allocate independent temporary buffers in VRAM for intermediate steps, resulting in transient spikes up to **180 MiB**. Custom fused kernels eliminate the need to create temporary buffers by executing multiple actions in the same kernel within registers and shared memory, reducing peak transient VRAM by **5.6×** (32.0 MiB peak). +Transient VRAM footprint over two training steps. Unfused CuPy dispatches allocate independent temporary buffers in VRAM for intermediate steps, resulting in transient spikes up to **420 MiB**. Custom fused kernels eliminate the need to create temporary buffers by executing multiple actions in the same kernel within registers and shared memory, reducing peak transient VRAM by **5.3×** (80.0 MiB peak). - - - Transient VRAM usage + + + Transient VRAM usage -**On the legacy series.** "Legacy CuPy" contains a `float64` type promotion bug that was found during the rewrite. GPUs often have very few or no FP64 ALUs, which explains the slowdown. The pathway is still included as an honest before-and-after, and not as a competitive baseline. - --- ## Building from Source diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..0f3331e --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,61 @@ +# Benchmarks + +The figures in the main README and the release notes come from the scripts here, on one +CNN: `build_model` in `cifar10_breakdown.py`, the same model as +`examples/cifar10/v1_cifar10.py`. + +| Script | What it measures | Figures | +|---|---|---| +| `cifar10_breakdown.py` | ms/step, split by training-step phase, on the normal GPU path | `benchmark_release` | +| `cifar10_paths.py` | ms/step and loss over one epoch, on NumPy, fallback CuPy and fused kernels | `benchmark_throughput`, `benchmark_convergence` | +| `cifar10_vram.py` | live VRAM over two steps, on fallback CuPy and fused kernels | `benchmark_vram` | +| `compare.py` | prints what changed between two `cifar10_breakdown.py` results | none | +| `plot.py` | draws the figures from saved results | all of the above | + +The scripts need a GPU and CuPy. They also need CIFAR-10 in `data/cifar-10`: create it with +`examples/cifar10/cifar-setup.ipynb`. `plot.py` needs matplotlib instead, which isn't a +dependency of the package. + +## Before and after a change + + python3 benchmarks/cifar10_breakdown.py benchmarks/results/before.json + python3 benchmarks/cifar10_breakdown.py benchmarks/results/after.json + python3 benchmarks/compare.py benchmarks/results/before.json benchmarks/results/after.json + +`benchmarks/results/` is ignored by git, so it's the place for working numbers. + +## Regenerating the figures for a release + +Published numbers live in `benchmarks/history//`, so a figure can be redrawn +without rerunning anything. Measure from clean worktrees: the scripts stay in your +checkout, `PYTHONPATH` picks the `aether` being measured, and each result records that +tree's commit. `--relative-paths` lets the worktree resolve inside the devcontainer. + + git worktree add --relative-paths --detach .worktrees/old v1.1.0 + git worktree add --relative-paths --detach .worktrees/new + +Run from the repo root on each GPU, naming the file after that GPU: + + PYTHONPATH=.worktrees/old python3 benchmarks/cifar10_breakdown.py benchmarks/history/v1.1.0/breakdown_rtx3050.json + PYTHONPATH=.worktrees/new python3 benchmarks/cifar10_breakdown.py benchmarks/history/v1.2.0/breakdown_rtx3050.json + +On a GPU other than the one the README names, also add its fused path to the convergence +figure: + + PYTHONPATH=.worktrees/new python3 benchmarks/cifar10_paths.py --fused-only benchmarks/history/v1.2.0/paths_fused_rtx3050.json + +Then once, on the machine the README names, for the path figures: + + PYTHONPATH=.worktrees/new python3 benchmarks/cifar10_vram.py benchmarks/history/v1.2.0/vram.json + PYTHONPATH=.worktrees/new python3 benchmarks/cifar10_paths.py benchmarks/history/v1.2.0/paths.json + +Draw them, then remove the worktrees: + + python3 benchmarks/plot.py v1.1.0 v1.2.0 + git worktree remove .worktrees/old + git worktree remove .worktrees/new + +`cifar10_paths.py` runs NumPy once and each GPU path three times, which takes about 15 +minutes. The cross-version figure uses only `cifar10_breakdown.py`, so older versions need nothing +else from this folder. Each GPU gets its own panel and scale on it: comparing versions is +only a fair A/B on one machine. diff --git a/benchmarks/cifar10_breakdown.py b/benchmarks/cifar10_breakdown.py index 5748562..c46d442 100644 --- a/benchmarks/cifar10_breakdown.py +++ b/benchmarks/cifar10_breakdown.py @@ -36,8 +36,10 @@ from pathlib import Path import cupy as cp +import numpy as np import aether as ae +import aether.config as config TRAIN_DIR = Path("data") / "cifar-10" / "cifar-10_train.npz" @@ -48,10 +50,22 @@ BREAKDOWN_STEPS = 100 -def build_model(X_train): +def load_train(): + """CIFAR-10 training images and labels, as host arrays.""" + if not TRAIN_DIR.exists(): + sys.exit( + f"[aether] {TRAIN_DIR} not found. Run from the repo root after creating it " + "with examples/cifar10/cifar-setup.ipynb." + ) + with np.load(TRAIN_DIR, allow_pickle=False) as data: + return data["X_train"], data["y_train"] + + +def build_model(X_train, device="cupy"): """Same model, pipeline and precision as examples/cifar10/v1_cifar10.py.""" + config.set_backend(device) feature_pipeline = ae.Compose([ - ae.ToTensor(dtype='float32', target_device="cupy"), + ae.ToTensor(dtype='float32', target_device=device), ae.Rescale(factor=1.0 / 255.0), ae.StandardScaler() ]) @@ -93,7 +107,7 @@ def build_model(X_train): preprocessor=feature_pipeline ) - model.to('cupy') + model.to(device) model.set_precision(compute_dtype=COMPUTE_DTYPE) model.finalize(input_shape=X_train.shape[1:]) feature_pipeline.fit(X_train) @@ -218,17 +232,36 @@ def spread(samples): def git_state(): - """(commit, dirty) of the checkout, or (None, None) outside git.""" + """(commit, dirty) of the checkout aether was imported from, or (None, None) + outside git, so a run against a worktree of an older tag records that tag.""" + source = Path(ae.__file__).parent try: commit = subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, - text=True, check=True).stdout.strip() + text=True, check=True, cwd=source).stdout.strip() status = subprocess.run(["git", "status", "--porcelain"], capture_output=True, - text=True, check=True).stdout + text=True, check=True, cwd=source).stdout except (OSError, subprocess.CalledProcessError): return None, None return commit, bool(status.strip()) +def run_meta(): + """What a result file needs to say about where it was measured.""" + commit, dirty = git_state() + return { + "commit": commit, + "dirty": dirty, + "date": datetime.datetime.now().isoformat(timespec="seconds"), + "gpu": cp.cuda.runtime.getDeviceProperties(0)["name"].decode(), + "cupy": cp.__version__, + "python": platform.python_version(), + "batch_size": BATCH_SIZE, + "compute_dtype": COMPUTE_DTYPE, + "warmup_steps": WARMUP_STEPS, + "baseline_steps": BASELINE_STEPS, + } + + def bound_paths(model): """Forward function each layer is bound to, keyed like '03 Conv2d'.""" paths = {} @@ -249,14 +282,7 @@ def main(): if args.repeats < 1: parser.error("--repeats must be at least 1") - if not TRAIN_DIR.exists(): - sys.exit( - f"[aether] {TRAIN_DIR} not found. Run from the repo root after creating it " - "with examples/cifar10/cifar-setup.ipynb." - ) - with cp.load(TRAIN_DIR, allow_pickle=False) as data: - X_train = data["X_train"] - y_train = data["y_train"] + X_train, y_train = (cp.asarray(array) for array in load_train()) model, feature_pipeline = build_model(X_train) transforms = feature_pipeline.transforms @@ -271,19 +297,9 @@ def main(): timed_train(model, X_train, y_train, BASELINE_STEPS) * 1e3 / BASELINE_STEPS) breakdown_runs.append(breakdown(model, transforms, X_train, y_train)) - commit, dirty = git_state() result = { "meta": { - "commit": commit, - "dirty": dirty, - "date": datetime.datetime.now().isoformat(timespec="seconds"), - "gpu": cp.cuda.runtime.getDeviceProperties(0)["name"].decode(), - "cupy": cp.__version__, - "python": platform.python_version(), - "batch_size": BATCH_SIZE, - "compute_dtype": COMPUTE_DTYPE, - "warmup_steps": WARMUP_STEPS, - "baseline_steps": BASELINE_STEPS, + **run_meta(), "breakdown_steps": BREAKDOWN_STEPS, "repeats": args.repeats, "warmup_s": warmup_s, diff --git a/benchmarks/cifar10_paths.py b/benchmarks/cifar10_paths.py new file mode 100644 index 0000000..b6980ca --- /dev/null +++ b/benchmarks/cifar10_paths.py @@ -0,0 +1,155 @@ +""" +Train the same CIFAR-10 CNN on three dispatch paths, for the README's throughput and +convergence figures. + +Run from the repo root (see benchmarks/README.md for the full release workflow): + + python3 benchmarks/cifar10_paths.py benchmarks/history/v1.2.0/paths.json + +--fused-only skips the slower paths, for adding another GPU to the convergence figure. + +The paths, all built by cifar10_breakdown.build_model: + + numpy-cpu the framework on the CPU fallback path + cupy-fallback tensors on the GPU, but every layer and the optimizer on that same + fallback path, so vectorized CuPy array ops and no custom kernels + cupy-fused the normal model.to("cupy") path + +The precision policy is the same everywhere, but only the fused path's Conv2d can honor +fp16; the fallback paths warn and compute Conv2d in fp32. + +Each run warms up, records loss and elapsed time for one epoch, then times BASELINE_STEPS +with no per-step sync. The epoch syncs after every step so each loss gets a timestamp, +which costs time on a fast step, so throughput comes from the unsynced block. The first +full-length train() runs slow (see cifar10_breakdown.main), and the epoch absorbs that. +""" +import argparse +import gc +import json +import time +from pathlib import Path + +import cupy as cp +from cifar10_breakdown import ( + BASELINE_STEPS, + BATCH_SIZE, + WARMUP_STEPS, + bound_paths, + build_model, + load_train, + run_meta, + spread, + timed_train, +) + +LABELS = { + "numpy-cpu": "NumPy CPU", + "cupy-fallback": "CuPy, no custom kernels", + "cupy-fused": "CuPy, fused kernels", +} +NUMPY_REPEATS = 1 # about 11 minutes a run, and the CPU is the least noisy path + + +def use_fallback_kernels(model): + """Point a finalized GPU model's dispatch at the fallback path, leaving tensors on + the GPU. The fallbacks take their array module from the data, so they run as CuPy.""" + for component in (*model.layers, model.loss, model.accuracy): + if hasattr(component, "_compile_for_device"): + component._compile_for_device("numpy") + # Not Adam's _compile_for_device, which also moves its buffers to the host. + model.optimizer._step_impl = model.optimizer._step_fallback + # BatchNorm just lost its fused activation, so the skipped ReLUs have to come back. + model._bind_fusion() + + +def prepare(key, X): + model, _ = build_model(X, device="numpy" if key == "numpy-cpu" else "cupy") + if key == "cupy-fallback": + use_fallback_kernels(model) + return model + + +def trace_epoch(model, X, y): + """Per-step data loss and cumulative seconds over one epoch of the real train().""" + calculate, step_optimizer = model.loss.calculate, model._step_optimizer + pending_loss = None + losses, elapsed_s = [], [] + total = 0.0 + + def remember_loss(*args, **kwargs): + nonlocal pending_loss + pending_loss = calculate(*args, **kwargs) + return pending_loss + + def end_step(): + nonlocal total, step_start + step_optimizer() + cp.cuda.Device().synchronize() + total += time.perf_counter() - step_start + elapsed_s.append(total) + losses.append(float(pending_loss)) + step_start = time.perf_counter() # the host readback above isn't step time + + model.loss.calculate = remember_loss + model._step_optimizer = end_step + try: + step_start = time.perf_counter() + model.train(X, y, epochs=1, batch_size=BATCH_SIZE, shuffle=True, verbose=0, + fit_preprocessor=False) + finally: + del model.loss.calculate # it was a plain class method + model._step_optimizer = step_optimizer + return losses, elapsed_s + + +def run_path(key, X, y, repeats): + steps_per_epoch = len(X) // BATCH_SIZE + epoch_X, epoch_y = X[:steps_per_epoch * BATCH_SIZE], y[:steps_per_epoch * BATCH_SIZE] + ms_per_step = [] + for _ in range(repeats): + model = prepare(key, X) + timed_train(model, X, y, WARMUP_STEPS) + losses, elapsed_s = trace_epoch(model, epoch_X, epoch_y) + ms_per_step.append(timed_train(model, X, y, BASELINE_STEPS) * 1e3 / BASELINE_STEPS) + paths = bound_paths(model) + del model + gc.collect() + cp.get_default_memory_pool().free_all_blocks() + return { + "key": key, + "label": LABELS[key], + "repeats": repeats, + "ms_per_step": spread(ms_per_step), + "loss": losses, # from the last repeat + "elapsed_s": elapsed_s, + "paths": paths, + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[1]) + parser.add_argument("output", type=Path, help="results file, for benchmarks/plot.py") + parser.add_argument("--repeats", type=int, default=3, + help=f"runs per GPU path (default: 3); NumPy runs {NUMPY_REPEATS}") + parser.add_argument("--fused-only", action="store_true", + help="run only cupy-fused, for a second GPU on the convergence figure") + args = parser.parse_args() + if args.repeats < 1: + parser.error("--repeats must be at least 1") + + X, y = load_train() + X_gpu, y_gpu = cp.asarray(X), cp.asarray(y) + results = [run_path("cupy-fused", X_gpu, y_gpu, args.repeats)] + if not args.fused_only: + results = [ + run_path("numpy-cpu", X, y, NUMPY_REPEATS), + run_path("cupy-fallback", X_gpu, y_gpu, args.repeats), + *results, + ] + + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps({"meta": run_meta(), "results": results}, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/cifar10_vram.py b/benchmarks/cifar10_vram.py new file mode 100644 index 0000000..be8d7e7 --- /dev/null +++ b/benchmarks/cifar10_vram.py @@ -0,0 +1,154 @@ +""" +Record transient VRAM over two training steps, fused kernels against the fallback path, +for the README's VRAM figure. + +Run from the repo root (see benchmarks/README.md for the full release workflow): + + python3 benchmarks/cifar10_vram.py benchmarks/history/v1.2.0/vram.json + +A CuPy memory hook logs live pool bytes at every malloc and free inside the real +model.train(). "Transient" is the delta above the bytes already live after warm-up +(parameters, AdamW state, the uploaded batches). The train() call records one extra step +that is thrown away, which leaves its one-off setup out of the trace and gives the first +kept step a start mark. The batch gather and preprocessing land in the forward phase. + +Allocation order doesn't depend on timing, so repeats must produce identical traces; +anything else means the path is nondeterministic and the run fails. +""" +import argparse +import gc +import itertools +import json +from pathlib import Path + +import cupy as cp +import cupy.cuda.memory_hook +from cifar10_breakdown import BATCH_SIZE, WARMUP_STEPS, load_train, run_meta +from cifar10_paths import LABELS, prepare + +RECORDED_STEPS = 2 +REPEATS = 3 +# Share of a step's x axis per phase. Layout only: these are not durations. +PHASE_WIDTHS = {"forward": 0.4, "loss": 0.1, "backward": 0.4, "optimizer": 0.1} + + +class LivePoolTrace(cupy.cuda.memory_hook.MemoryHook): + """Live bytes since the hook was entered, appended at every malloc and free.""" + + name = "AetherLivePoolTrace" + + def __init__(self): + super().__init__() + self.live = 0 + self.trace = [] + + def malloc_postprocess(self, **kwargs): + self.live += kwargs["mem_size"] + self.trace.append(self.live) + + def free_postprocess(self, **kwargs): + self.live -= kwargs["mem_size"] + self.trace.append(self.live) + + +def record_steps(model, X, y): + """(trace, seams): the live-bytes trace of train(), and the trace index at the end + of every phase. The phases end at the last call of each, in PHASE_WIDTHS order.""" + recorder = LivePoolTrace() + seams = [] + + def marked(func): + def wrapper(*args, **kwargs): + out = func(*args, **kwargs) + seams.append(len(recorder.trace)) + return out + return wrapper + + instance_attrs = ((model, "forward"), (model.loss, "backward"), (model, "backward"), + (model, "_step_optimizer")) + originals = [(obj, attr, vars(obj).get(attr)) for obj, attr in instance_attrs] + for obj, attr in instance_attrs: + setattr(obj, attr, marked(getattr(obj, attr))) + try: + n = (RECORDED_STEPS + 1) * BATCH_SIZE + with recorder: + model.train(X[:n], y[:n], epochs=1, batch_size=BATCH_SIZE, shuffle=False, + verbose=0, fit_preprocessor=False) + finally: + for obj, attr, original in originals: + if original is None: + delattr(obj, attr) # it was a plain class method + else: + setattr(obj, attr, original) + return recorder.trace, seams + + +def to_phase_axis(trace, seams): + """Keep the recorded steps and spread each phase's events evenly over its width, so + both paths share an x axis. Returns (x, transient bytes, phase) per event.""" + phases = len(PHASE_WIDTHS) + kept = seams[phases - 1:] # from the end of the discarded step + x, transient, phase_of = [], [], [] + bands = itertools.cycle(PHASE_WIDTHS.items()) + position = 0.0 + for (lo, hi), (phase, width) in zip(itertools.pairwise(kept), bands): + for i in range(lo, hi): + x.append(position + width * (i - lo) / (hi - lo)) + transient.append(trace[i]) + phase_of.append(phase) + position += width + return x, transient, phase_of + + +def measure(key, X, y): + model = prepare(key, X) + model.train(X, y, epochs=1, batch_size=BATCH_SIZE, shuffle=False, verbose=0, + fit_preprocessor=False) + cp.cuda.Device().synchronize() + baseline = cp.get_default_memory_pool().used_bytes() + + trace, seams = record_steps(model, X, y) + x, transient, phase_of = to_phase_axis(trace, seams) + peak = max(range(len(transient)), key=transient.__getitem__) + + del model + gc.collect() + cp.get_default_memory_pool().free_all_blocks() + return { + "key": key, + "label": LABELS[key], + "baseline_bytes": baseline, + "peak_transient_bytes": transient[peak], + "peak_x": x[peak], + "peak_phase": phase_of[peak], + "x": x, + "transient_bytes": transient, + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[1]) + parser.add_argument("output", type=Path, help="results file, for benchmarks/plot.py") + args = parser.parse_args() + + # Only what the run consumes goes on the GPU, or the baseline is mostly dataset. + n = WARMUP_STEPS * BATCH_SIZE + X, y = (cp.asarray(array[:n]) for array in load_train()) + + results = [] + for key in ("cupy-fused", "cupy-fallback"): + runs = [measure(key, X, y) for _ in range(REPEATS)] + if any(run["transient_bytes"] != runs[0]["transient_bytes"] for run in runs): + raise RuntimeError(f"[aether] {key}: VRAM traces differ across {REPEATS} repeats.") + results.append(runs[0]) + + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps({ + "meta": {**run_meta(), "recorded_steps": RECORDED_STEPS, "repeats": REPEATS}, + "phase_widths": PHASE_WIDTHS, + "results": results, + }, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/history/v1.1.0/breakdown_rtx3050.json b/benchmarks/history/v1.1.0/breakdown_rtx3050.json new file mode 100644 index 0000000..a30bb49 --- /dev/null +++ b/benchmarks/history/v1.1.0/breakdown_rtx3050.json @@ -0,0 +1,829 @@ +{ + "meta": { + "commit": "cbaa753a3abf612ef7d8a60eb1193a973e5f42fd", + "dirty": false, + "date": "2026-11-01T01:56:09", + "gpu": "NVIDIA GeForce RTX 3050 6GB Laptop GPU", + "cupy": "14.2.0", + "python": "3.14.10", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128, + "breakdown_steps": 100, + "repeats": 3, + "warmup_s": 3.5643876999999975 + }, + "paths": { + "00 Conv2d": "_forward_gpu compute=float16", + "01 BatchNorm": "_forward_gpu", + "02 ReLU": "forward", + "03 Conv2d": "_forward_gpu compute=float16", + "04 BatchNorm": "_forward_gpu", + "05 ReLU": "forward", + "06 MaxPool2d": "_forward_gpu", + "07 SpatialDropout": "_forward_gpu", + "08 Conv2d": "_forward_gpu compute=float16", + "09 BatchNorm": "_forward_gpu", + "10 ReLU": "forward", + "11 Conv2d": "_forward_gpu compute=float16", + "12 BatchNorm": "_forward_gpu", + "13 ReLU": "forward", + "14 MaxPool2d": "_forward_gpu", + "15 SpatialDropout": "_forward_gpu", + "16 Conv2d": "_forward_gpu compute=float16", + "17 BatchNorm": "_forward_gpu", + "18 ReLU": "forward", + "19 MaxPool2d": "_forward_gpu", + "20 SpatialDropout": "_forward_gpu", + "21 GlobalAvgPool": "_forward_gpu", + "22 Dense": "forward" + }, + "baseline_ms": { + "median": 49.89947500000014, + "min": 49.76996562499991, + "max": 50.026520312499656 + }, + "gpu_total_ms": { + "median": 51.09320569279218, + "min": 51.06470774640941, + "max": 51.351779436824295 + }, + "host_total_ms": { + "median": 12.169427272735879, + "min": 12.168137373737869, + "max": 12.398943434338362 + }, + "rows": [ + { + "name": "preprocess ToTensor", + "stage": "preprocess", + "gpu_ms": { + "median": 0.23127886803463252, + "min": 0.22718771720173384, + "max": 0.23497729364669684 + }, + "host_ms": { + "median": 0.20822626263199912, + "min": 0.20474343433315972, + "max": 0.2154191919328356 + } + }, + { + "name": "preprocess Rescale", + "stage": "preprocess", + "gpu_ms": { + "median": 0.1513868286603629, + "min": 0.15028622242236378, + "max": 0.1514925244781706 + }, + "host_ms": { + "median": 0.10935151514253577, + "min": 0.10759595959072356, + "max": 0.10980505049527056 + } + }, + { + "name": "preprocess StandardScaler", + "stage": "preprocess", + "gpu_ms": { + "median": 0.18016420237042688, + "min": 0.17748298917454902, + "max": 0.1835733326998624 + }, + "host_ms": { + "median": 0.1679474747455823, + "min": 0.16490202019575909, + "max": 0.16856161616690063 + } + }, + { + "name": "forward 00 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.7828738587071197, + "min": 0.7809690513996163, + "max": 0.7871663799189558 + }, + "host_ms": { + "median": 0.3119181818222231, + "min": 0.30879797978888396, + "max": 0.3153242424431106 + } + }, + { + "name": "forward 01 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 1.1363607163381095, + "min": 1.1355435980690851, + "max": 1.1365779337256845 + }, + "host_ms": { + "median": 0.4685969696871472, + "min": 0.4597989899102318, + "max": 0.4728939393937088 + } + }, + { + "name": "forward 02 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.5488019417030643, + "min": 0.5485847315402946, + "max": 0.5488640051899534 + }, + "host_ms": { + "median": 0.051109090893280355, + "min": 0.04992828281051562, + "max": 0.056212121204556424 + } + }, + { + "name": "forward 03 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 3.4301621143264, + "min": 3.4130126851977725, + "max": 3.4569516061532375 + }, + "host_ms": { + "median": 0.18369393938478928, + "min": 0.1836535353628609, + "max": 0.18913636364746395 + } + }, + { + "name": "forward 04 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 1.1350160801049434, + "min": 1.1343541133283364, + "max": 1.1363503788456772 + }, + "host_ms": { + "median": 0.4080626262697883, + "min": 0.3965414141544677, + "max": 0.41068383838036326 + } + }, + { + "name": "forward 05 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.5484812946030588, + "min": 0.5484399223568464, + "max": 0.5485433562837466 + }, + "host_ms": { + "median": 0.04164646464122405, + "min": 0.040220201999729414, + "max": 0.04316464646257777 + } + }, + { + "name": "forward 06 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.4166542273579222, + "min": 0.41647839215066695, + "max": 0.41692315598930973 + }, + "host_ms": { + "median": 0.10121010102397343, + "min": 0.09945353534967948, + "max": 0.10134646463989662 + } + }, + { + "name": "forward 07 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 1.0513170036402615, + "min": 1.044552399052514, + "max": 1.0611639600811582 + }, + "host_ms": { + "median": 0.05739595958586524, + "min": 0.05658080807173827, + "max": 0.05751212122527479 + } + }, + { + "name": "forward 08 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.9003959587126067, + "min": 0.8952242437035146, + "max": 0.9075846431231258 + }, + "host_ms": { + "median": 0.1583333333343549, + "min": 0.15239595960209576, + "max": 0.15886868688098868 + } + }, + { + "name": "forward 09 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.659218101790457, + "min": 0.6591146678635569, + "max": 0.6606041193008423 + }, + "host_ms": { + "median": 0.3763313131292472, + "min": 0.3760272727320607, + "max": 0.3835303030363587 + } + }, + { + "name": "forward 10 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.27722472765228967, + "min": 0.2771730100867724, + "max": 0.27732816097712276 + }, + "host_ms": { + "median": 0.04048989900652624, + "min": 0.03794040403269232, + "max": 0.040961616164305735 + } + }, + { + "name": "forward 11 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 1.7285430335035228, + "min": 1.7185616144026168, + "max": 1.7422273905590326 + }, + "host_ms": { + "median": 0.15226464646766913, + "min": 0.15224545453095586, + "max": 0.15322626262042666 + } + }, + { + "name": "forward 12 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.6600145474828855, + "min": 0.6593835980001123, + "max": 0.6601903847973756 + }, + "host_ms": { + "median": 0.350388888901679, + "min": 0.3501464646346031, + "max": 0.3702959596039639 + } + }, + { + "name": "forward 13 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.27727644250850486, + "min": 0.2771006039898805, + "max": 0.27748331216850664 + }, + "host_ms": { + "median": 0.04245858584532342, + "min": 0.03686363635611684, + "max": 0.04350404039560273 + } + }, + { + "name": "forward 14 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.21296097097372768, + "min": 0.2127851326056201, + "max": 0.21314715165080447 + }, + "host_ms": { + "median": 0.08104949494157733, + "min": 0.0761808080805893, + "max": 0.08215757574904237 + } + }, + { + "name": "forward 15 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.5298320753405793, + "min": 0.5264290860204985, + "max": 0.5347969206896696 + }, + "host_ms": { + "median": 0.049761616163217545, + "min": 0.047160606052123265, + "max": 0.05543232326071846 + } + }, + { + "name": "forward 16 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.8348703023159143, + "min": 0.830453654732367, + "max": 0.8416555993484728 + }, + "host_ms": { + "median": 0.14506969696550653, + "min": 0.13472525252800183, + "max": 0.14552525253794105 + } + }, + { + "name": "forward 17 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.33603749100608055, + "min": 0.3333171672291226, + "max": 0.34267797825312374 + }, + "host_ms": { + "median": 0.3507393939293656, + "min": 0.34420707071479806, + "max": 0.42328888889412586 + } + }, + { + "name": "forward 18 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.14182917111449772, + "min": 0.14152921164276624, + "max": 0.14182917156604805 + }, + "host_ms": { + "median": 0.03899494949008739, + "min": 0.03795151515261769, + "max": 0.039802020196613296 + } + }, + { + "name": "forward 19 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.11503967800826738, + "min": 0.11475006197438095, + "max": 0.11538101153241263 + }, + "host_ms": { + "median": 0.06667272726080034, + "min": 0.06554040403282882, + "max": 0.0703999999981846 + } + }, + { + "name": "forward 20 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.26812250475690824, + "min": 0.2663537769606619, + "max": 0.27054286815903406 + }, + "host_ms": { + "median": 0.04589090908804351, + "min": 0.045437373728418606, + "max": 0.07878282828660488 + } + }, + { + "name": "forward 21 GlobalAvgPool", + "stage": "forward", + "gpu_ms": { + "median": 0.02381058635585236, + "min": 0.02370715197740179, + "max": 0.023872646945293504 + }, + "host_ms": { + "median": 0.03433434343688449, + "min": 0.033086868694281846, + "max": 0.03435353536670763 + } + }, + { + "name": "forward 22 Dense", + "stage": "forward", + "gpu_ms": { + "median": 0.02052137323401191, + "min": 0.020511029796166855, + "max": 0.020593777298927307 + }, + "host_ms": { + "median": 0.3147767676770368, + "min": 0.3005222222189773, + "max": 0.3233717171850812 + } + }, + { + "name": "loss.calculate", + "stage": "loss.calculate", + "gpu_ms": { + "median": 0.22105761609896266, + "min": 0.21566028230720097, + "max": 0.22624872730235862 + }, + "host_ms": { + "median": 0.8591979798154051, + "min": 0.8482929292928718, + "max": 0.8603252525297262 + } + }, + { + "name": "loss.regularization_loss", + "stage": "loss.regularization_loss", + "gpu_ms": { + "median": 0.04561616177435475, + "min": 0.04004880795349376, + "max": 0.04879224242059269 + }, + "host_ms": { + "median": 0.17677070709304635, + "min": 0.17141414140854921, + "max": 0.20221212121030457 + } + }, + { + "name": "accuracy.calculate", + "stage": "accuracy.calculate", + "gpu_ms": { + "median": 0.0962197980043864, + "min": 0.09449147530878434, + "max": 0.09631450553283546 + }, + "host_ms": { + "median": 0.38682828280936077, + "min": 0.3865979797938788, + "max": 0.40233434341726854 + } + }, + { + "name": "loss.backward", + "stage": "loss.backward", + "gpu_ms": { + "median": 0.05133802040169636, + "min": 0.05083539422526203, + "max": 0.06094545481557196 + }, + "host_ms": { + "median": 0.184549494941515, + "min": 0.18127979796870455, + "max": 0.2105333333295703 + } + }, + { + "name": "backward 22 Dense", + "stage": "backward", + "gpu_ms": { + "median": 0.05914278810072427, + "min": 0.056749252997564545, + "max": 0.06061414196485221 + }, + "host_ms": { + "median": 0.42942929293889337, + "min": 0.4148101010048207, + "max": 0.44154141414765036 + } + }, + { + "name": "backward 21 GlobalAvgPool", + "stage": "backward", + "gpu_ms": { + "median": 0.04491797968218423, + "min": 0.04042763626108868, + "max": 0.045981414145743006 + }, + "host_ms": { + "median": 0.05626565657300002, + "min": 0.05499292929429926, + "max": 0.05923939395163163 + } + }, + { + "name": "backward 20 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.15200258640929906, + "min": 0.15134998072277417, + "max": 0.1520394351747301 + }, + "host_ms": { + "median": 0.05879494950141919, + "min": 0.05686666665928667, + "max": 0.06446060605770926 + } + }, + { + "name": "backward 19 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.2199427852726946, + "min": 0.2193428682859498, + "max": 0.22112193628393037 + }, + "host_ms": { + "median": 0.10904343432432345, + "min": 0.10346161616555515, + "max": 0.10996363638319834 + } + }, + { + "name": "backward 18 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.24486011883827172, + "min": 0.24476702737085748, + "max": 0.24486011898878848 + }, + "host_ms": { + "median": 0.10669595958415352, + "min": 0.10381616163034448, + "max": 0.10809292927771105 + } + }, + { + "name": "backward 17 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.44779830207728377, + "min": 0.447725899291761, + "max": 0.45170811992703064 + }, + "host_ms": { + "median": 0.42370909089856806, + "min": 0.41207979796521926, + "max": 0.4292949494990325 + } + }, + { + "name": "backward 16 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 2.114197991111062, + "min": 2.1049406046819206, + "max": 2.1285753683610396 + }, + "host_ms": { + "median": 0.26469191918680957, + "min": 0.24795353535887674, + "max": 0.2653050505017231 + } + }, + { + "name": "backward 15 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.2804415361447768, + "min": 0.27890036593783984, + "max": 0.2830791115158736 + }, + "host_ms": { + "median": 0.05245858586830662, + "min": 0.05018787878082484, + "max": 0.057268686865574285 + } + }, + { + "name": "backward 14 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.42525995801193545, + "min": 0.42262238083463727, + "max": 0.42875604075614854 + }, + "host_ms": { + "median": 0.08152828283500364, + "min": 0.08093030301281051, + "max": 0.09059797982233865 + } + }, + { + "name": "backward 13 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.4785079630938443, + "min": 0.4784459017141901, + "max": 0.47852865072211836 + }, + "host_ms": { + "median": 0.08500404038671269, + "min": 0.08225353533614704, + "max": 0.09769292928881028 + } + }, + { + "name": "backward 12 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.8218168905287078, + "min": 0.8163038383830677, + "max": 0.8295020623640581 + }, + "host_ms": { + "median": 0.3839606060612116, + "min": 0.3823343434310164, + "max": 0.40011616161708924 + } + }, + { + "name": "backward 11 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 4.297810761615484, + "min": 4.279482181626137, + "max": 4.325531169621631 + }, + "host_ms": { + "median": 0.23948888888344758, + "min": 0.22933434343260636, + "max": 0.24978585858864083 + } + }, + { + "name": "backward 10 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.47841487207798045, + "min": 0.4781252540121175, + "max": 0.47857002387143144 + }, + "host_ms": { + "median": 0.08793131313714092, + "min": 0.0775585858489171, + "max": 0.08816565656905534 + } + }, + { + "name": "backward 09 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.8233270295942673, + "min": 0.8205860190921359, + "max": 0.8284470276399092 + }, + "host_ms": { + "median": 0.36248989899740075, + "min": 0.36165555556034074, + "max": 0.40818383838150757 + } + }, + { + "name": "backward 08 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 3.0126183418312458, + "min": 2.9980858215177904, + "max": 3.032177785430292 + }, + "host_ms": { + "median": 0.22815353536821525, + "min": 0.2275707070754898, + "max": 0.2471232323175788 + } + }, + { + "name": "backward 07 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.5556906612232478, + "min": 0.5516670665355644, + "max": 0.5602728044143831 + }, + "host_ms": { + "median": 0.05489494948060475, + "min": 0.05069494948592631, + "max": 0.057196969701272804 + } + }, + { + "name": "backward 06 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.8498993142686709, + "min": 0.8473237999761948, + "max": 0.8528368461011636 + }, + "host_ms": { + "median": 0.08790404040660683, + "min": 0.08437171716953647, + "max": 0.08923333333392164 + } + }, + { + "name": "backward 05 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.9467862579557631, + "min": 0.9465069849081714, + "max": 0.9467966014688666 + }, + "host_ms": { + "median": 0.08987373736329234, + "min": 0.08544949494792334, + "max": 0.0908363636385316 + } + }, + { + "name": "backward 04 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 1.4596033879000732, + "min": 1.446446540379765, + "max": 1.474384159752817 + }, + "host_ms": { + "median": 0.39471313130438873, + "min": 0.37691111110964837, + "max": 0.4101070707214732 + } + }, + { + "name": "backward 03 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 8.219968641647185, + "min": 8.175191908171684, + "max": 8.269855017613883 + }, + "host_ms": { + "median": 0.2418272727394688, + "min": 0.24173232322891178, + "max": 0.25181414142282693 + } + }, + { + "name": "backward 02 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.9471586268357556, + "min": 0.9471586256316213, + "max": 0.9474275533599082 + }, + "host_ms": { + "median": 0.08379090909651073, + "min": 0.0828717171807879, + "max": 0.09008787877735243 + } + }, + { + "name": "backward 01 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 1.464764770835337, + "min": 1.4513803672308874, + "max": 1.4850896127296216 + }, + "host_ms": { + "median": 0.37992020201722837, + "min": 0.3722292929344831, + "max": 0.3810646464554531 + } + }, + { + "name": "backward 00 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 4.676514890458849, + "min": 4.652218182881673, + "max": 4.706469506928415 + }, + "host_ms": { + "median": 0.2231272727440779, + "min": 0.21761212119921342, + "max": 0.23260101010916948 + } + }, + { + "name": "optimizer.step", + "stage": "optimizer.step", + "gpu_ms": { + "median": 0.32220088743200204, + "min": 0.3140638375824148, + "max": 0.3266718380077921 + }, + "host_ms": { + "median": 0.5600515151427592, + "min": 0.5459878787846582, + "max": 0.5735040404125366 + } + }, + { + "name": "unattributed", + "stage": "unattributed", + "gpu_ms": { + "median": 0.7921133116734299, + "min": 0.7758306786512034, + "max": 0.7974453665730026 + }, + "host_ms": { + "median": 1.133164646574582, + "min": 1.1104252524854368, + "max": 1.1369656565696487 + } + } + ] +} diff --git a/benchmarks/history/v1.1.0/breakdown_rx9070xt.json b/benchmarks/history/v1.1.0/breakdown_rx9070xt.json new file mode 100644 index 0000000..5290645 --- /dev/null +++ b/benchmarks/history/v1.1.0/breakdown_rx9070xt.json @@ -0,0 +1,829 @@ +{ + "meta": { + "commit": "cbaa753a3abf612ef7d8a60eb1193a973e5f42fd", + "dirty": false, + "date": "2026-11-01T06:03:28", + "gpu": "AMD Radeon RX 9070 XT", + "cupy": "14.2.0", + "python": "3.14.4", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128, + "breakdown_steps": 100, + "repeats": 3, + "warmup_s": 1.0367416430001413 + }, + "paths": { + "00 Conv2d": "_forward_gpu compute=float16", + "01 BatchNorm": "_forward_gpu", + "02 ReLU": "forward", + "03 Conv2d": "_forward_gpu compute=float16", + "04 BatchNorm": "_forward_gpu", + "05 ReLU": "forward", + "06 MaxPool2d": "_forward_gpu", + "07 SpatialDropout": "_forward_gpu", + "08 Conv2d": "_forward_gpu compute=float16", + "09 BatchNorm": "_forward_gpu", + "10 ReLU": "forward", + "11 Conv2d": "_forward_gpu compute=float16", + "12 BatchNorm": "_forward_gpu", + "13 ReLU": "forward", + "14 MaxPool2d": "_forward_gpu", + "15 SpatialDropout": "_forward_gpu", + "16 Conv2d": "_forward_gpu compute=float16", + "17 BatchNorm": "_forward_gpu", + "18 ReLU": "forward", + "19 MaxPool2d": "_forward_gpu", + "20 SpatialDropout": "_forward_gpu", + "21 GlobalAvgPool": "_forward_gpu", + "22 Dense": "forward" + }, + "baseline_ms": { + "median": 10.244090578126475, + "min": 10.224559773437392, + "max": 10.25087264062563 + }, + "gpu_total_ms": { + "median": 10.786630360767095, + "min": 10.758876550077188, + "max": 10.791561531298088 + }, + "host_total_ms": { + "median": 2.1164478990025177, + "min": 2.1133971009875854, + "max": 2.1433353434333937 + }, + "rows": [ + { + "name": "preprocess ToTensor", + "stage": "preprocess", + "gpu_ms": { + "median": 0.01717273731048059, + "min": 0.016963474725307237, + "max": 0.017180444362262886 + }, + "host_ms": { + "median": 0.023536030305353273, + "min": 0.023485969686681537, + "max": 0.023584707058935145 + } + }, + { + "name": "preprocess Rescale", + "stage": "preprocess", + "gpu_ms": { + "median": 0.01325303026371532, + "min": 0.012946020172099875, + "max": 0.01326803026739696 + }, + "host_ms": { + "median": 0.011593585865895617, + "min": 0.01150551515544948, + "max": 0.011846181827551313 + } + }, + { + "name": "preprocess StandardScaler", + "stage": "preprocess", + "gpu_ms": { + "median": 0.0263420201348837, + "min": 0.026290373650915695, + "max": 0.02685769692514882 + }, + "host_ms": { + "median": 0.022184979788925367, + "min": 0.022113878804987625, + "max": 0.022378959604821197 + } + }, + { + "name": "forward 00 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.12685820166811798, + "min": 0.12626615121509088, + "max": 0.1275328283358102 + }, + "host_ms": { + "median": 0.034528919210667176, + "min": 0.03435888889597287, + "max": 0.035124818173290356 + } + }, + { + "name": "forward 01 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.30108056556094775, + "min": 0.30101919174194336, + "max": 0.30273966686894194 + }, + "host_ms": { + "median": 0.08317955557043141, + "min": 0.08258231313714468, + "max": 0.08367469698244193 + } + }, + { + "name": "forward 02 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.09426456554369493, + "min": 0.09366696945043525, + "max": 0.09470654467139582 + }, + "host_ms": { + "median": 0.0087488383886728, + "min": 0.008564434334733629, + "max": 0.008984505052696955 + } + }, + { + "name": "forward 03 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.6315265514633872, + "min": 0.6309213036238545, + "max": 0.6341168368705595 + }, + "host_ms": { + "median": 0.03089710101981708, + "min": 0.03052700000550587, + "max": 0.0309723131425871 + } + }, + { + "name": "forward 04 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.30634419033021637, + "min": 0.30599776693064756, + "max": 0.3066410786575741 + }, + "host_ms": { + "median": 0.07676397978565262, + "min": 0.07584088890059384, + "max": 0.07685391917928225 + } + }, + { + "name": "forward 05 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.09297276732295451, + "min": 0.09194209057875354, + "max": 0.09357197966539499 + }, + "host_ms": { + "median": 0.008427545467246531, + "min": 0.008342595951990465, + "max": 0.008513323231307717 + } + }, + { + "name": "forward 06 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.048280515002481865, + "min": 0.04818593932673185, + "max": 0.04921632293950428 + }, + "host_ms": { + "median": 0.018867797987493558, + "min": 0.018793676769167756, + "max": 0.01924030302367523 + } + }, + { + "name": "forward 07 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.11416580798951062, + "min": 0.11405022201514003, + "max": 0.11423849480019675 + }, + "host_ms": { + "median": 0.011086131326469559, + "min": 0.011017838382159802, + "max": 0.011190060592865609 + } + }, + { + "name": "forward 08 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.17758877650655883, + "min": 0.1775822824600971, + "max": 0.1785378179164848 + }, + "host_ms": { + "median": 0.03060130301712365, + "min": 0.030419717188693778, + "max": 0.031002414135226065 + } + }, + { + "name": "forward 09 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.15264050361483988, + "min": 0.15188088832479535, + "max": 0.15285144400114964 + }, + "host_ms": { + "median": 0.07405224244283806, + "min": 0.07368213131843676, + "max": 0.07417547474210322 + } + }, + { + "name": "forward 10 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.03412450485006727, + "min": 0.033995989710092545, + "max": 0.034294565593955495 + }, + "host_ms": { + "median": 0.007856343438216947, + "min": 0.007820636359280219, + "max": 0.008175909091402322 + } + }, + { + "name": "forward 11 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.32527654550292273, + "min": 0.3244671306826852, + "max": 0.32541864598640285 + }, + "host_ms": { + "median": 0.028296515160440724, + "min": 0.028125404042205475, + "max": 0.028441939401440176 + } + }, + { + "name": "forward 12 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.1519896357950538, + "min": 0.15183807999798746, + "max": 0.15209950416377097 + }, + "host_ms": { + "median": 0.07117407068852283, + "min": 0.07110309088786398, + "max": 0.07128347474095753 + } + }, + { + "name": "forward 13 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.03522513121968568, + "min": 0.03516649499987111, + "max": 0.0355134949539647 + }, + "host_ms": { + "median": 0.00782603030364209, + "min": 0.007811959598385705, + "max": 0.007895383835906008 + } + }, + { + "name": "forward 14 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.029263737319846345, + "min": 0.02921131302160446, + "max": 0.029472181761656144 + }, + "host_ms": { + "median": 0.015890707069929427, + "min": 0.015836868667497353, + "max": 0.01622554547720203 + } + }, + { + "name": "forward 15 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.06252481848603547, + "min": 0.06252430307895246, + "max": 0.06255991911195745 + }, + "host_ms": { + "median": 0.009998919189585117, + "min": 0.009951595973119527, + "max": 0.010033444453651412 + } + }, + { + "name": "forward 16 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.17322867566888983, + "min": 0.17313369657054092, + "max": 0.17407152493192693 + }, + "host_ms": { + "median": 0.0284243131414824, + "min": 0.028282181808732054, + "max": 0.028708101002517982 + } + }, + { + "name": "forward 17 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.10446989882473993, + "min": 0.10444076697934758, + "max": 0.10464160558250216 + }, + "host_ms": { + "median": 0.07246963635758091, + "min": 0.07216345454363557, + "max": 0.07283827272713649 + } + }, + { + "name": "forward 18 ReLU", + "stage": "forward", + "gpu_ms": { + "median": 0.020910191773013634, + "min": 0.020830202019876905, + "max": 0.021166737332488552 + }, + "host_ms": { + "median": 0.007834363617748199, + "min": 0.007830282827559829, + "max": 0.008208121230244145 + } + }, + { + "name": "forward 19 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.019702858583192633, + "min": 0.019662090944069805, + "max": 0.019733121167078163 + }, + "host_ms": { + "median": 0.015288747464284016, + "min": 0.015234808073489488, + "max": 0.015503090905596506 + } + }, + { + "name": "forward 20 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.03584690892485657, + "min": 0.03584557549670489, + "max": 0.03610060574731441 + }, + "host_ms": { + "median": 0.010082909105249477, + "min": 0.010071969682306217, + "max": 0.010409959596665542 + } + }, + { + "name": "forward 21 GlobalAvgPool", + "stage": "forward", + "gpu_ms": { + "median": 0.010195313162650124, + "min": 0.01016905050574228, + "max": 0.010231262637358723 + }, + "host_ms": { + "median": 0.006912282820720599, + "min": 0.006897303015795493, + "max": 0.00703783837597064 + } + }, + { + "name": "forward 22 Dense", + "stage": "forward", + "gpu_ms": { + "median": 0.027037908953398165, + "min": 0.027006717022470754, + "max": 0.02706053527542437 + }, + "host_ms": { + "median": 0.0704577070764296, + "min": 0.06984683839370766, + "max": 0.07181217170964925 + } + }, + { + "name": "loss.calculate", + "stage": "loss.calculate", + "gpu_ms": { + "median": 0.07453862622831807, + "min": 0.07452970645343414, + "max": 0.07458070650546238 + }, + "host_ms": { + "median": 0.16583522221820873, + "min": 0.16523294951441883, + "max": 0.16722793939473862 + } + }, + { + "name": "loss.regularization_loss", + "stage": "loss.regularization_loss", + "gpu_ms": { + "median": 0.035130484706976196, + "min": 0.03468488881157504, + "max": 0.035191484679928935 + }, + "host_ms": { + "median": 0.033561333320720206, + "min": 0.03351580808570857, + "max": 0.03409103030410026 + } + }, + { + "name": "accuracy.calculate", + "stage": "accuracy.calculate", + "gpu_ms": { + "median": 0.025673858557987695, + "min": 0.02566980794448443, + "max": 0.02567787863539927 + }, + "host_ms": { + "median": 0.04122144444620356, + "min": 0.04118660606332244, + "max": 0.04173403029731943 + } + }, + { + "name": "loss.backward", + "stage": "loss.backward", + "gpu_ms": { + "median": 0.013192878725628058, + "min": 0.013183222142885429, + "max": 0.013258333252084375 + }, + "host_ms": { + "median": 0.03286763638133187, + "min": 0.03262217171710529, + "max": 0.03308769694768037 + } + }, + { + "name": "backward 22 Dense", + "stage": "backward", + "gpu_ms": { + "median": 0.038042080635675275, + "min": 0.03801980802808145, + "max": 0.03812690899528638 + }, + "host_ms": { + "median": 0.09219832323385227, + "min": 0.09211175756718148, + "max": 0.09310187879053028 + } + }, + { + "name": "backward 21 GlobalAvgPool", + "stage": "backward", + "gpu_ms": { + "median": 0.009036494832899835, + "min": 0.009033212052526498, + "max": 0.009070777691750213 + }, + "host_ms": { + "median": 0.010150090918044714, + "min": 0.009963939414728425, + "max": 0.010498494934882278 + } + }, + { + "name": "backward 20 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.03612696943861066, + "min": 0.03610735341454997, + "max": 0.036338262410476954 + }, + "host_ms": { + "median": 0.01093208081760847, + "min": 0.010885222205946444, + "max": 0.010952868687852951 + } + }, + { + "name": "backward 19 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.025847979755413652, + "min": 0.025842383767318244, + "max": 0.02599012116064327 + }, + "host_ms": { + "median": 0.01795154543232815, + "min": 0.017775010101471745, + "max": 0.018320020198914773 + } + }, + { + "name": "backward 18 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.051830888885741284, + "min": 0.05173478783531622, + "max": 0.05194397963056661 + }, + "host_ms": { + "median": 0.01966507070530056, + "min": 0.019541404021260353, + "max": 0.019767959590142416 + } + }, + { + "name": "backward 17 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.09234691907962163, + "min": 0.09171664632029003, + "max": 0.09241844420180176 + }, + "host_ms": { + "median": 0.07694454544352167, + "min": 0.07669811110636054, + "max": 0.07729144445562784 + } + }, + { + "name": "backward 16 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.4999181294079983, + "min": 0.49966145103627985, + "max": 0.5035678374044823 + }, + "host_ms": { + "median": 0.04833549494585234, + "min": 0.04823579799467736, + "max": 0.04853865654254288 + } + }, + { + "name": "backward 15 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.06324596973982724, + "min": 0.06297603023774696, + "max": 0.06336845434976346 + }, + "host_ms": { + "median": 0.010344757556319537, + "min": 0.010313515145533143, + "max": 0.010603797981541837 + } + }, + { + "name": "backward 14 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.04720495964842613, + "min": 0.04694999980204033, + "max": 0.04723817174031277 + }, + "host_ms": { + "median": 0.014815545459942318, + "min": 0.014763343429105149, + "max": 0.01488098988981316 + } + }, + { + "name": "backward 13 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.10964082798572501, + "min": 0.10888934308531309, + "max": 0.11096861582211774 + }, + "host_ms": { + "median": 0.016668848484003303, + "min": 0.016579353546194858, + "max": 0.01717379799252611 + } + }, + { + "name": "backward 12 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.14441339385629903, + "min": 0.14297618212723973, + "max": 0.14480604998993152 + }, + "host_ms": { + "median": 0.06984055554689023, + "min": 0.06982938384221551, + "max": 0.07043896970792433 + } + }, + { + "name": "backward 11 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.9018576361916282, + "min": 0.8981562094254927, + "max": 0.9084113583420262 + }, + "host_ms": { + "median": 0.044480656568872165, + "min": 0.04412411112753034, + "max": 0.04505005049946759 + } + }, + { + "name": "backward 10 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.10783322193104812, + "min": 0.10677344427265302, + "max": 0.10988733277778433 + }, + "host_ms": { + "median": 0.015465272733281105, + "min": 0.015383303030426355, + "max": 0.015469545459005403 + } + }, + { + "name": "backward 09 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.1441567764438764, + "min": 0.1439948066918537, + "max": 0.14425535244171064 + }, + "host_ms": { + "median": 0.06731990908584255, + "min": 0.06691083837097841, + "max": 0.06797826262704297 + } + }, + { + "name": "backward 08 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.6279059196963455, + "min": 0.6272277976527358, + "max": 0.6291325507741986 + }, + "host_ms": { + "median": 0.04255059597005928, + "min": 0.04233420202241416, + "max": 0.04270855553997278 + } + }, + { + "name": "backward 07 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.11562269655140964, + "min": 0.11536542378892803, + "max": 0.11590833292163984 + }, + "host_ms": { + "median": 0.009758111110671172, + "min": 0.00972054546093774, + "max": 0.010362929306395036 + } + }, + { + "name": "backward 06 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.083614696757962, + "min": 0.0835848075873924, + "max": 0.0837096462797637 + }, + "host_ms": { + "median": 0.014636565648481448, + "min": 0.014573343426518753, + "max": 0.014803595955857996 + } + }, + { + "name": "backward 05 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.27182417174782414, + "min": 0.27170493867662215, + "max": 0.27227949313443117 + }, + "host_ms": { + "median": 0.015974131298262917, + "min": 0.01589335353874232, + "max": 0.016304121200763886 + } + }, + { + "name": "backward 04 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.35002425192582487, + "min": 0.35000207056902877, + "max": 0.35022467675835195 + }, + "host_ms": { + "median": 0.06698257573478975, + "min": 0.06667701010120815, + "max": 0.06721353535721566 + } + }, + { + "name": "backward 03 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 2.0218428987445254, + "min": 2.019169489542643, + "max": 2.022476350418245 + }, + "host_ms": { + "median": 0.04321156565669243, + "min": 0.04251389899571112, + "max": 0.0434234141381478 + } + }, + { + "name": "backward 02 ReLU", + "stage": "backward", + "gpu_ms": { + "median": 0.2706580511247269, + "min": 0.2693735454419647, + "max": 0.2709349285472523 + }, + "host_ms": { + "median": 0.014954262623076729, + "min": 0.014953969699178346, + "max": 0.01524040403821364 + } + }, + { + "name": "backward 01 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.33413355458866467, + "min": 0.3337263139811429, + "max": 0.33482083648142186 + }, + "host_ms": { + "median": 0.06515697980766429, + "min": 0.06497654545694447, + "max": 0.06582127275119046 + } + }, + { + "name": "backward 00 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.7793019383844703, + "min": 0.778065946969119, + "max": 0.7805487862741104 + }, + "host_ms": { + "median": 0.04186868687436873, + "min": 0.04168202019742478, + "max": 0.04226281819251428 + } + }, + { + "name": "optimizer.step", + "stage": "optimizer.step", + "gpu_ms": { + "median": 0.07879404016215392, + "min": 0.07876616121843608, + "max": 0.07920617181243318 + }, + "host_ms": { + "median": 0.09826254546454158, + "min": 0.09799083837339886, + "max": 0.09926931313955976 + } + }, + { + "name": "unattributed", + "stage": "unattributed", + "gpu_ms": { + "median": 0.28950182374830197, + "min": 0.28945863678712735, + "max": 0.2900067060113405 + }, + "host_ms": { + "median": 0.13036937372422158, + "min": 0.13034524245958523, + "max": 0.1365988989641894 + } + } + ] +} diff --git a/benchmarks/history/v1.2.0/breakdown_rtx3050.json b/benchmarks/history/v1.2.0/breakdown_rtx3050.json new file mode 100644 index 0000000..766987d --- /dev/null +++ b/benchmarks/history/v1.2.0/breakdown_rtx3050.json @@ -0,0 +1,689 @@ +{ + "meta": { + "commit": "089667c28f9d396f32794c64ae9a594ad7bf7c6d", + "dirty": false, + "date": "2026-11-01T01:56:33", + "gpu": "NVIDIA GeForce RTX 3050 6GB Laptop GPU", + "cupy": "14.2.0", + "python": "3.14.10", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128, + "breakdown_steps": 100, + "repeats": 3, + "warmup_s": 1.6355306000000382 + }, + "paths": { + "00 Conv2d": "_forward_gpu compute=float16", + "01 BatchNorm": "_forward_gpu", + "02 ReLU": "_forward_fallback", + "03 Conv2d": "_forward_gpu compute=float16", + "04 BatchNorm": "_forward_gpu", + "05 ReLU": "_forward_fallback", + "06 MaxPool2d": "_forward_gpu", + "07 SpatialDropout": "_forward_gpu", + "08 Conv2d": "_forward_gpu compute=float16", + "09 BatchNorm": "_forward_gpu", + "10 ReLU": "_forward_fallback", + "11 Conv2d": "_forward_gpu compute=float16", + "12 BatchNorm": "_forward_gpu", + "13 ReLU": "_forward_fallback", + "14 MaxPool2d": "_forward_gpu", + "15 SpatialDropout": "_forward_gpu", + "16 Conv2d": "_forward_gpu compute=float16", + "17 BatchNorm": "_forward_gpu", + "18 ReLU": "_forward_fallback", + "19 MaxPool2d": "_forward_gpu", + "20 SpatialDropout": "_forward_gpu", + "21 GlobalAvgPool": "_forward_gpu", + "22 Dense": "forward" + }, + "baseline_ms": { + "median": 21.712432031250728, + "min": 21.670398437500182, + "max": 21.753719531250937 + }, + "gpu_total_ms": { + "median": 22.633761531174784, + "min": 22.510250534674135, + "max": 22.636088920362067 + }, + "host_total_ms": { + "median": 6.102232323235402, + "min": 5.9557383838481, + "max": 6.998035353540723 + }, + "rows": [ + { + "name": "preprocess ToTensor", + "stage": "preprocess", + "gpu_ms": { + "median": 0.1947823835141731, + "min": 0.1769810106718179, + "max": 0.1982012117149854 + }, + "host_ms": { + "median": 0.17068989899084358, + "min": 0.1555535353565225, + "max": 0.1746030302846787 + } + }, + { + "name": "preprocess Rescale", + "stage": "preprocess", + "gpu_ms": { + "median": 0.11805996012808097, + "min": 0.1122763640049732, + "max": 0.12491636384617198 + }, + "host_ms": { + "median": 0.08379696970508241, + "min": 0.0801353535509884, + "max": 0.0893333333236785 + } + }, + { + "name": "preprocess StandardScaler", + "stage": "preprocess", + "gpu_ms": { + "median": 0.14316347595116105, + "min": 0.13149414264192485, + "max": 0.1446322424694745 + }, + "host_ms": { + "median": 0.12222121211958768, + "min": 0.10761515150446976, + "max": 0.12721414140002588 + } + }, + { + "name": "forward 00 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.507004120735207, + "min": 0.4924499410571474, + "max": 0.5219193534417585 + }, + "host_ms": { + "median": 0.21478080808641736, + "min": 0.20426565655532503, + "max": 0.23593333334605576 + } + }, + { + "name": "forward 01 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.825106103612919, + "min": 0.8249302675025632, + "max": 0.8258508347501659 + }, + "host_ms": { + "median": 0.14837676768237165, + "min": 0.14721111110610755, + "max": 0.17004646464685189 + } + }, + { + "name": "forward 03 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 1.1847783435474744, + "min": 1.180227229089448, + "max": 1.187829659442709 + }, + "host_ms": { + "median": 0.11979898988901175, + "min": 0.11830909090459693, + "max": 0.14914949496176064 + } + }, + { + "name": "forward 04 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.8245165299887609, + "min": 0.8244751565384142, + "max": 0.8246406515439352 + }, + "host_ms": { + "median": 0.10434141412671041, + "min": 0.09767575758557698, + "max": 0.11570909091828108 + } + }, + { + "name": "forward 06 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.4197158789393878, + "min": 0.4194883240593804, + "max": 0.41987103073283877 + }, + "host_ms": { + "median": 0.07506565655125767, + "min": 0.07394747476472262, + "max": 0.08289595959670099 + } + }, + { + "name": "forward 07 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.16185405970823885, + "min": 0.1615954726332366, + "max": 0.16196783714824253 + }, + "host_ms": { + "median": 0.10757979797781812, + "min": 0.1044929292950683, + "max": 0.11521515149998052 + } + }, + { + "name": "forward 08 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.5091659014273171, + "min": 0.5059284047044889, + "max": 0.509455516783878 + }, + "host_ms": { + "median": 0.10962929293396662, + "min": 0.10426464646024206, + "max": 0.11930404040105365 + } + }, + { + "name": "forward 09 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.42117430164356423, + "min": 0.42113292488184845, + "max": 0.42149494633530127 + }, + "host_ms": { + "median": 0.09383232323864032, + "min": 0.09198888887976404, + "max": 0.11399898987908061 + } + }, + { + "name": "forward 11 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.9305677576498552, + "min": 0.9221895722427753, + "max": 0.9330812102616436 + }, + "host_ms": { + "median": 0.08962525251061176, + "min": 0.08754343434932275, + "max": 0.10928686868861735 + } + }, + { + "name": "forward 12 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.42113292548391557, + "min": 0.42088468478183555, + "max": 0.421350137151853 + }, + "host_ms": { + "median": 0.0842717171682961, + "min": 0.08312727271858779, + "max": 0.10403939395611374 + } + }, + { + "name": "forward 14 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.2151537797968797, + "min": 0.21508137595773946, + "max": 0.21528824486515738 + }, + "host_ms": { + "median": 0.057403030319215356, + "min": 0.05166767677927955, + "max": 0.05820707071341696 + } + }, + { + "name": "forward 15 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.09047402102838863, + "min": 0.09003959663889625, + "max": 0.09049470790407875 + }, + "host_ms": { + "median": 0.07976464646410684, + "min": 0.0780828282733288, + "max": 0.09178484850219154 + } + }, + { + "name": "forward 16 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.4345276759128378, + "min": 0.4305868287279148, + "max": 0.43584129515320363 + }, + "host_ms": { + "median": 0.0903393939469507, + "min": 0.08958686868311935, + "max": 0.11052323232048145 + } + }, + { + "name": "forward 17 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.21930149393250245, + "min": 0.219280807959913, + "max": 0.2193428682859498 + }, + "host_ms": { + "median": 0.09031717170939667, + "min": 0.08588585858823801, + "max": 0.10345454546206218 + } + }, + { + "name": "forward 19 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.11707733518848515, + "min": 0.116684284550373, + "max": 0.11765656718099961 + }, + "host_ms": { + "median": 0.045989898982879085, + "min": 0.042743434349434055, + "max": 0.05077777778163668 + } + }, + { + "name": "forward 20 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.05613381855867126, + "min": 0.05603038418022069, + "max": 0.056154505434361374 + }, + "host_ms": { + "median": 0.0738717171702085, + "min": 0.07100404040368566, + "max": 0.08716060605218794 + } + }, + { + "name": "forward 21 GlobalAvgPool", + "stage": "forward", + "gpu_ms": { + "median": 0.026189576138300124, + "min": 0.02462771755050529, + "max": 0.026644687158892854 + }, + "host_ms": { + "median": 0.023995959596467942, + "min": 0.022111111106961758, + "max": 0.02699999999728761 + } + }, + { + "name": "forward 22 Dense", + "stage": "forward", + "gpu_ms": { + "median": 0.026448161689319995, + "min": 0.02434844437121141, + "max": 0.027523878679582565 + }, + "host_ms": { + "median": 0.24233636363211614, + "min": 0.23850000001163948, + "max": 0.26646262626579276 + } + }, + { + "name": "loss.calculate", + "stage": "loss.calculate", + "gpu_ms": { + "median": 0.20997559617866168, + "min": 0.20542125253364293, + "max": 0.2181126470818664 + }, + "host_ms": { + "median": 0.6400909090859727, + "min": 0.6076484848443839, + "max": 0.7071242424132439 + } + }, + { + "name": "loss.regularization_loss", + "stage": "loss.regularization_loss", + "gpu_ms": { + "median": 0.046473050640538485, + "min": 0.04279919179401012, + "max": 0.047892040582467814 + }, + "host_ms": { + "median": 0.10134949494762752, + "min": 0.09805555554053033, + "max": 0.1278555555518783 + } + }, + { + "name": "accuracy.calculate", + "stage": "accuracy.calculate", + "gpu_ms": { + "median": 0.07273697002668573, + "min": 0.0707484446842261, + "max": 0.07661834377983603 + }, + "host_ms": { + "median": 0.2737737373821011, + "min": 0.25985959596347136, + "max": 0.3001565656549329 + } + }, + { + "name": "loss.backward", + "stage": "loss.backward", + "gpu_ms": { + "median": 0.031988040609943744, + "min": 0.02971959597609862, + "max": 0.03457486873135121 + }, + "host_ms": { + "median": 0.11437272727369308, + "min": 0.11035656565021634, + "max": 0.13505454546358317 + } + }, + { + "name": "backward 22 Dense", + "stage": "backward", + "gpu_ms": { + "median": 0.06669834361533926, + "min": 0.06054981826155475, + "max": 0.07714133359717601 + }, + "host_ms": { + "median": 0.29683838383651606, + "min": 0.26638383840259167, + "max": 0.3383888888998222 + } + }, + { + "name": "backward 21 GlobalAvgPool", + "stage": "backward", + "gpu_ms": { + "median": 0.029940363245479988, + "min": 0.02761406022490877, + "max": 0.030093898865008593 + }, + "host_ms": { + "median": 0.04033737373176355, + "min": 0.0354525252601668, + "max": 0.043252525248971496 + } + }, + { + "name": "backward 20 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.04458957500379495, + "min": 0.043990625538910276, + "max": 0.046549655948624466 + }, + "host_ms": { + "median": 0.0329080808183569, + "min": 0.03242626263327662, + "max": 0.04053535353751163 + } + }, + { + "name": "backward 19 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.22113227979703384, + "min": 0.22060444123215145, + "max": 0.22215628021895284 + }, + "host_ms": { + "median": 0.07459494950217968, + "min": 0.07139393939512663, + "max": 0.08824242425160062 + } + }, + { + "name": "backward 17 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.3505389934236353, + "min": 0.3505389919184675, + "max": 0.35081826526709275 + }, + "host_ms": { + "median": 0.13612828281405412, + "min": 0.13537171717972443, + "max": 0.151479797984917 + } + }, + { + "name": "backward 16 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.9800714362751354, + "min": 0.9714346678570064, + "max": 0.9839605670986753 + }, + "host_ms": { + "median": 0.13768282830061812, + "min": 0.13106969698477622, + "max": 0.1547090908967551 + } + }, + { + "name": "backward 15 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.07724476876583966, + "min": 0.07718270776247738, + "max": 0.07733785902911966 + }, + "host_ms": { + "median": 0.039056565638571614, + "min": 0.028635353550675466, + "max": 0.03962222223036048 + } + }, + { + "name": "backward 14 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.42642876535955104, + "min": 0.423294705272925, + "max": 0.4294076745558267 + }, + "host_ms": { + "median": 0.05509797980000322, + "min": 0.053436363631672895, + "max": 0.06401515149649861 + } + }, + { + "name": "backward 12 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.6837320442151542, + "min": 0.6834010542041123, + "max": 0.6837630729482631 + }, + "host_ms": { + "median": 0.10246767675942631, + "min": 0.0963313131264978, + "max": 0.127233333329849 + } + }, + { + "name": "backward 11 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 2.0694936622272837, + "min": 2.0490446861344154, + "max": 2.077075379063385 + }, + "host_ms": { + "median": 0.1069989899116825, + "min": 0.10162323229965893, + "max": 0.12427777778611926 + } + }, + { + "name": "backward 09 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.6815702644261447, + "min": 0.6815702626199434, + "max": 0.6818081646254568 + }, + "host_ms": { + "median": 0.08994646465696567, + "min": 0.08812828282959409, + "max": 0.10297575756633917 + } + }, + { + "name": "backward 08 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 1.1109365670367926, + "min": 1.1023618572890157, + "max": 1.1144636821265173 + }, + "host_ms": { + "median": 0.09324343433734318, + "min": 0.09187474746235091, + "max": 0.11995050505966635 + } + }, + { + "name": "backward 07 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.14886270975223695, + "min": 0.1486454965791317, + "max": 0.14906957820810454 + }, + "host_ms": { + "median": 0.029644444440337894, + "min": 0.0285555555695861, + "max": 0.036516161620084225 + } + }, + { + "name": "backward 06 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.8471789913948136, + "min": 0.8429898980892065, + "max": 0.8523610517232105 + }, + "host_ms": { + "median": 0.05226161616666665, + "min": 0.04879090909875084, + "max": 0.062132323228988845 + } + }, + { + "name": "backward 04 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 1.3493527475029532, + "min": 1.3493320628850147, + "max": 1.3495285860215775 + }, + "host_ms": { + "median": 0.09534747477293348, + "min": 0.09284545454059785, + "max": 0.10210101010275736 + } + }, + { + "name": "backward 03 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 2.4480012980374424, + "min": 2.4279970809666795, + "max": 2.4547555542955495 + }, + "host_ms": { + "median": 0.09692525252766511, + "min": 0.09468484848763513, + "max": 0.11068585857201745 + } + }, + { + "name": "backward 01 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 1.3471806085470952, + "min": 1.34717026744226, + "max": 1.347253016751222 + }, + "host_ms": { + "median": 0.09375454545658034, + "min": 0.09083030303225663, + "max": 0.10079292927443792 + } + }, + { + "name": "backward 00 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.6853663012234852, + "min": 0.6821908722020159, + "max": 0.6863282413193674 + }, + "host_ms": { + "median": 0.07593838384107382, + "min": 0.07045959595452954, + "max": 0.08434545454770388 + } + }, + { + "name": "optimizer.step", + "stage": "optimizer.step", + "gpu_ms": { + "median": 0.2492890494941461, + "min": 0.23764040433999264, + "max": 0.2502080009441183 + }, + "host_ms": { + "median": 0.3983616161739786, + "min": 0.38461111112477314, + "max": 0.4965323232337334 + } + }, + { + "name": "unattributed", + "stage": "unattributed", + "gpu_ms": { + "median": 0.5985440488415535, + "min": 0.556439725795027, + "max": 0.6275962865770315 + }, + "host_ms": { + "median": 0.7512232323393857, + "min": 0.7470606059762235, + "max": 0.837955555591014 + } + } + ] +} diff --git a/benchmarks/history/v1.2.0/breakdown_rx9070xt.json b/benchmarks/history/v1.2.0/breakdown_rx9070xt.json new file mode 100644 index 0000000..7a8237e --- /dev/null +++ b/benchmarks/history/v1.2.0/breakdown_rx9070xt.json @@ -0,0 +1,689 @@ +{ + "meta": { + "commit": "089667c28f9d396f32794c64ae9a594ad7bf7c6d", + "dirty": false, + "date": "2026-11-01T06:03:34", + "gpu": "AMD Radeon RX 9070 XT", + "cupy": "14.2.0", + "python": "3.14.4", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128, + "breakdown_steps": 100, + "repeats": 3, + "warmup_s": 0.7245134039999357 + }, + "paths": { + "00 Conv2d": "_forward_gpu compute=float16", + "01 BatchNorm": "_forward_gpu", + "02 ReLU": "_forward_fallback", + "03 Conv2d": "_forward_gpu compute=float16", + "04 BatchNorm": "_forward_gpu", + "05 ReLU": "_forward_fallback", + "06 MaxPool2d": "_forward_gpu", + "07 SpatialDropout": "_forward_gpu", + "08 Conv2d": "_forward_gpu compute=float16", + "09 BatchNorm": "_forward_gpu", + "10 ReLU": "_forward_fallback", + "11 Conv2d": "_forward_gpu compute=float16", + "12 BatchNorm": "_forward_gpu", + "13 ReLU": "_forward_fallback", + "14 MaxPool2d": "_forward_gpu", + "15 SpatialDropout": "_forward_gpu", + "16 Conv2d": "_forward_gpu compute=float16", + "17 BatchNorm": "_forward_gpu", + "18 ReLU": "_forward_fallback", + "19 MaxPool2d": "_forward_gpu", + "20 SpatialDropout": "_forward_gpu", + "21 GlobalAvgPool": "_forward_gpu", + "22 Dense": "forward" + }, + "baseline_ms": { + "median": 4.855191773437184, + "min": 4.850693906249148, + "max": 4.887264125001067 + }, + "gpu_total_ms": { + "median": 5.291655699412028, + "min": 5.286670636649084, + "max": 5.29354668145228 + }, + "host_total_ms": { + "median": 1.4948654949572908, + "min": 1.4946524141426683, + "max": 1.545806222221665 + }, + "rows": [ + { + "name": "preprocess ToTensor", + "stage": "preprocess", + "gpu_ms": { + "median": 0.015366010085651369, + "min": 0.015168757544773998, + "max": 0.0166508989747275 + }, + "host_ms": { + "median": 0.025390030321597642, + "min": 0.025239595965967504, + "max": 0.026197888877908956 + } + }, + { + "name": "preprocess Rescale", + "stage": "preprocess", + "gpu_ms": { + "median": 0.013362989817379099, + "min": 0.013243010070062044, + "max": 0.013609828192251499 + }, + "host_ms": { + "median": 0.014692151516367566, + "min": 0.014679353547891346, + "max": 0.015048666661889663 + } + }, + { + "name": "preprocess StandardScaler", + "stage": "preprocess", + "gpu_ms": { + "median": 0.026282343354941617, + "min": 0.026245565708689016, + "max": 0.02659840396407879 + }, + "host_ms": { + "median": 0.023911191922734637, + "min": 0.023891333326404987, + "max": 0.025044000003250366 + } + }, + { + "name": "forward 00 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.1166721305461845, + "min": 0.1158070404722233, + "max": 0.11707984799086446 + }, + "host_ms": { + "median": 0.031759272727904686, + "min": 0.031691141418717726, + "max": 0.03217202020371615 + } + }, + { + "name": "forward 01 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.11455087860425313, + "min": 0.11436381793082362, + "max": 0.11483209060900139 + }, + "host_ms": { + "median": 0.03523306061266119, + "min": 0.03523076768005646, + "max": 0.03614459596790748 + } + }, + { + "name": "forward 03 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.30216480565793585, + "min": 0.30074419036056055, + "max": 0.3021854640859546 + }, + "host_ms": { + "median": 0.032996868679155605, + "min": 0.032848939393472686, + "max": 0.033718676765390844 + } + }, + { + "name": "forward 04 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.11453113087801019, + "min": 0.11438399989797611, + "max": 0.11472059537967046 + }, + "host_ms": { + "median": 0.030658878790516132, + "min": 0.03059601011624512, + "max": 0.03268359595072392 + } + }, + { + "name": "forward 06 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.05844273726747493, + "min": 0.058165484719505216, + "max": 0.05895551504811855 + }, + "host_ms": { + "median": 0.017450343427822878, + "min": 0.017329292909885732, + "max": 0.01808519192364298 + } + }, + { + "name": "forward 07 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.03164443403783471, + "min": 0.03162293930095856, + "max": 0.03177782785937642 + }, + "host_ms": { + "median": 0.028002151522760144, + "min": 0.027996848484231077, + "max": 0.028761525258728812 + } + }, + { + "name": "forward 08 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.12453524188862906, + "min": 0.12439100000292364, + "max": 0.12476872750604996 + }, + "host_ms": { + "median": 0.03139761615141587, + "min": 0.03112484849069263, + "max": 0.03197065655862242 + } + }, + { + "name": "forward 09 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.06449207061468953, + "min": 0.06420967652641162, + "max": 0.0649384946534128 + }, + "host_ms": { + "median": 0.030376393923036475, + "min": 0.030356151501569048, + "max": 0.0319447878651276 + } + }, + { + "name": "forward 11 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.21609150475323802, + "min": 0.215610302307389, + "max": 0.2163476858175162 + }, + "host_ms": { + "median": 0.029167252510299963, + "min": 0.029033323235033996, + "max": 0.02995276768941572 + } + }, + { + "name": "forward 12 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.06336998984669194, + "min": 0.06332360591852304, + "max": 0.0636892221642263 + }, + "host_ms": { + "median": 0.028242171724328233, + "min": 0.02822948486119467, + "max": 0.029084505053969673 + } + }, + { + "name": "forward 14 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.02683189883828163, + "min": 0.026766828157835536, + "max": 0.027039080764157602 + }, + "host_ms": { + "median": 0.01467912121333571, + "min": 0.014597151515074368, + "max": 0.015279727284161543 + } + }, + { + "name": "forward 15 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.023894030318567246, + "min": 0.02384223226420205, + "max": 0.024089555585324163 + }, + "host_ms": { + "median": 0.02435148484657861, + "min": 0.024298777763687388, + "max": 0.02583370707560045 + } + }, + { + "name": "forward 16 Conv2d", + "stage": "forward", + "gpu_ms": { + "median": 0.11879907074299725, + "min": 0.11866688856271783, + "max": 0.11908226211865743 + }, + "host_ms": { + "median": 0.028721363635512063, + "min": 0.028612373738485415, + "max": 0.029661525251518616 + } + }, + { + "name": "forward 17 BatchNorm", + "stage": "forward", + "gpu_ms": { + "median": 0.04743326252157038, + "min": 0.04740455550978882, + "max": 0.04754929267095797 + }, + "host_ms": { + "median": 0.028482131306696426, + "min": 0.028028212128823855, + "max": 0.029119808078401622 + } + }, + { + "name": "forward 19 MaxPool2d", + "stage": "forward", + "gpu_ms": { + "median": 0.019427727230570534, + "min": 0.019409858424103622, + "max": 0.01959102028849149 + }, + "host_ms": { + "median": 0.013632878806380875, + "min": 0.013567565660692211, + "max": 0.014312676774242198 + } + }, + { + "name": "forward 20 SpatialDropout", + "stage": "forward", + "gpu_ms": { + "median": 0.01942932323524446, + "min": 0.019422474721766482, + "max": 0.019816353653717523 + }, + "host_ms": { + "median": 0.023149343433308815, + "min": 0.023057909094510397, + "max": 0.02439784848182612 + } + }, + { + "name": "forward 21 GlobalAvgPool", + "stage": "forward", + "gpu_ms": { + "median": 0.010160080862767769, + "min": 0.01014112127060541, + "max": 0.010192444491567034 + }, + "host_ms": { + "median": 0.006043515167671057, + "min": 0.005942323234580037, + "max": 0.006217878795135708 + } + }, + { + "name": "forward 22 Dense", + "stage": "forward", + "gpu_ms": { + "median": 0.026704919232864572, + "min": 0.026659373607900407, + "max": 0.026797474672396977 + }, + "host_ms": { + "median": 0.06805861616010937, + "min": 0.06785073739106016, + "max": 0.07044529291242081 + } + }, + { + "name": "loss.calculate", + "stage": "loss.calculate", + "gpu_ms": { + "median": 0.07502641415957248, + "min": 0.07499404055903656, + "max": 0.07558719211756582 + }, + "host_ms": { + "median": 0.16531691918693597, + "min": 0.1646808585712389, + "max": 0.16870803031479503 + } + }, + { + "name": "loss.regularization_loss", + "stage": "loss.regularization_loss", + "gpu_ms": { + "median": 0.03403034345293888, + "min": 0.033827504933331955, + "max": 0.035015454271225016 + }, + "host_ms": { + "median": 0.02659868686111962, + "min": 0.02654364645119737, + "max": 0.028128262641470742 + } + }, + { + "name": "accuracy.calculate", + "stage": "accuracy.calculate", + "gpu_ms": { + "median": 0.0256709594919224, + "min": 0.025634626190018172, + "max": 0.025873444391169932 + }, + "host_ms": { + "median": 0.03982814141498679, + "min": 0.039637929302618356, + "max": 0.0405387373883408 + } + }, + { + "name": "loss.backward", + "stage": "loss.backward", + "gpu_ms": { + "median": 0.013232030228457668, + "min": 0.01322651504905838, + "max": 0.013334323153501809 + }, + "host_ms": { + "median": 0.031919868697881645, + "min": 0.03190133336351889, + "max": 0.03275288888905168 + } + }, + { + "name": "backward 22 Dense", + "stage": "backward", + "gpu_ms": { + "median": 0.037938252467699725, + "min": 0.03791769691789993, + "max": 0.038181878663975787 + }, + "host_ms": { + "median": 0.09135189898326203, + "min": 0.09124469697137946, + "max": 0.09366247475857865 + } + }, + { + "name": "backward 21 GlobalAvgPool", + "stage": "backward", + "gpu_ms": { + "median": 0.009152040280627481, + "min": 0.009151636199517683, + "max": 0.009214262515682765 + }, + "host_ms": { + "median": 0.009939727269371323, + "min": 0.009883161625350682, + "max": 0.011055252518344051 + } + }, + { + "name": "backward 20 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.011821969609820482, + "min": 0.011810666580468115, + "max": 0.01185222217493286 + }, + "host_ms": { + "median": 0.009256909094682384, + "min": 0.009097656572401944, + "max": 0.009414323220412232 + } + }, + { + "name": "backward 19 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.025791060442876334, + "min": 0.02567995952987912, + "max": 0.025823747344089275 + }, + "host_ms": { + "median": 0.017175030291242604, + "min": 0.017077050511998648, + "max": 0.017556000006097194 + } + }, + { + "name": "backward 17 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.05346810076423366, + "min": 0.053455969375191315, + "max": 0.05354522194976759 + }, + "host_ms": { + "median": 0.035504848497249906, + "min": 0.035289585864552234, + "max": 0.03634679797878654 + } + }, + { + "name": "backward 16 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.3089092117969436, + "min": 0.3087519575851132, + "max": 0.30959360707889905 + }, + "host_ms": { + "median": 0.033858898981487424, + "min": 0.033667787870353544, + "max": 0.035056505061624596 + } + }, + { + "name": "backward 15 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.017098747476032287, + "min": 0.01709316157256112, + "max": 0.017108040382013176 + }, + "host_ms": { + "median": 0.008495787870292634, + "min": 0.008449474762528553, + "max": 0.008702282822482768 + } + }, + { + "name": "backward 14 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.047684605901289465, + "min": 0.0474797473336109, + "max": 0.048085899064035126 + }, + "host_ms": { + "median": 0.013374191931146962, + "min": 0.013326535357312727, + "max": 0.013587020214000597 + } + }, + { + "name": "backward 12 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.10548811052182708, + "min": 0.10516126213049648, + "max": 0.10562307101608527 + }, + "host_ms": { + "median": 0.02986805050182518, + "min": 0.029695494956717763, + "max": 0.0304413030417808 + } + }, + { + "name": "backward 11 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.5388938152428829, + "min": 0.5377418266402351, + "max": 0.5404909794980829 + }, + "host_ms": { + "median": 0.029323353557647536, + "min": 0.029201828285772766, + "max": 0.030499585850842354 + } + }, + { + "name": "backward 09 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.09934669650263256, + "min": 0.09914136301688474, + "max": 0.09951386818982134 + }, + "host_ms": { + "median": 0.027958707071527235, + "min": 0.02794924243558357, + "max": 0.02889602020754543 + } + }, + { + "name": "backward 08 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.38367896971076426, + "min": 0.3832066670210675, + "max": 0.3839900294939677 + }, + "host_ms": { + "median": 0.0274445151515263, + "min": 0.027381262613073547, + "max": 0.027678909076473586 + } + }, + { + "name": "backward 07 SpatialDropout", + "stage": "backward", + "gpu_ms": { + "median": 0.029255262566636308, + "min": 0.029177767704382088, + "max": 0.029285171586606238 + }, + "host_ms": { + "median": 0.00826247473051266, + "min": 0.00812993939138268, + "max": 0.008456727259313497 + } + }, + { + "name": "backward 06 MaxPool2d", + "stage": "backward", + "gpu_ms": { + "median": 0.08189796942352044, + "min": 0.08160869694418377, + "max": 0.08189997980088899 + }, + "host_ms": { + "median": 0.013069838380695066, + "min": 0.013042161616326528, + "max": 0.01426350503046559 + } + }, + { + "name": "backward 04 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.23493371648017805, + "min": 0.23441172669632265, + "max": 0.23541777799225816 + }, + "host_ms": { + "median": 0.02819802019838229, + "min": 0.027701474751629207, + "max": 0.02886400000194283 + } + }, + { + "name": "backward 03 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.7487156023882856, + "min": 0.7484829088654181, + "max": 0.7498449544713954 + }, + "host_ms": { + "median": 0.02766893939306928, + "min": 0.027650555553882072, + "max": 0.02834198990012278 + } + }, + { + "name": "backward 01 BatchNorm", + "stage": "backward", + "gpu_ms": { + "median": 0.2728182616257908, + "min": 0.27256486963744114, + "max": 0.2729794139211828 + }, + "host_ms": { + "median": 0.02680962627276342, + "min": 0.026638101003713513, + "max": 0.027711636370569444 + } + }, + { + "name": "backward 00 Conv2d", + "stage": "backward", + "gpu_ms": { + "median": 0.24899006958561715, + "min": 0.24777912055001114, + "max": 0.25022569554622726 + }, + "host_ms": { + "median": 0.020895868691227268, + "min": 0.02076090910315653, + "max": 0.021967121201118007 + } + }, + { + "name": "optimizer.step", + "stage": "optimizer.step", + "gpu_ms": { + "median": 0.0787257875757988, + "min": 0.07856741397067754, + "max": 0.07920623254595381 + }, + "host_ms": { + "median": 0.09842412120835636, + "min": 0.09788246464065903, + "max": 0.10017051513294964 + } + }, + { + "name": "unattributed", + "stage": "unattributed", + "gpu_ms": { + "median": 0.2514014671170952, + "min": 0.25095333196599073, + "max": 0.25429925845578466 + }, + "host_ms": { + "median": 0.11096538381282542, + "min": 0.11057696961512248, + "max": 0.11692898990095457 + } + } + ] +} diff --git a/benchmarks/history/v1.2.0/paths.json b/benchmarks/history/v1.2.0/paths.json new file mode 100644 index 0000000..d59ca53 --- /dev/null +++ b/benchmarks/history/v1.2.0/paths.json @@ -0,0 +1,1303 @@ +{ + "meta": { + "commit": "089667c28f9d396f32794c64ae9a594ad7bf7c6d", + "dirty": false, + "date": "2026-11-01T06:18:08", + "gpu": "AMD Radeon RX 9070 XT", + "cupy": "14.2.0", + "python": "3.14.4", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128 + }, + "results": [ + { + "key": "numpy-cpu", + "label": "NumPy CPU", + "repeats": 1, + "ms_per_step": { + "median": 1864.879145125002, + "min": 1864.879145125002, + "max": 1864.879145125002 + }, + "loss": [ + 2.0791234970092773, + 2.1204352378845215, + 1.9607759714126587, + 2.0435619354248047, + 2.0713768005371094, + 2.037095546722412, + 2.095552682876587, + 2.021148204803467, + 1.9837052822113037, + 1.9734175205230713, + 2.041973114013672, + 2.039243221282959, + 2.0170488357543945, + 2.116513967514038, + 1.9259860515594482, + 1.9581693410873413, + 2.066929340362549, + 1.8671185970306396, + 1.9567198753356934, + 1.9294501543045044, + 1.9942266941070557, + 1.9271364212036133, + 2.0355687141418457, + 2.0402727127075195, + 1.8340587615966797, + 1.9882564544677734, + 1.901668667793274, + 1.9138545989990234, + 1.9704139232635498, + 1.8409671783447266, + 1.813981533050537, + 1.9416565895080566, + 1.941821813583374, + 1.928127408027649, + 1.883833885192871, + 1.9765796661376953, + 1.9943314790725708, + 1.899674892425537, + 1.916198492050171, + 1.928789496421814, + 1.8743716478347778, + 1.7559040784835815, + 1.8534669876098633, + 1.8951078653335571, + 1.798410415649414, + 1.8066133260726929, + 1.8757410049438477, + 1.8129987716674805, + 1.9070096015930176, + 1.8797470331192017, + 1.9259464740753174, + 1.824710726737976, + 1.8113830089569092, + 1.874244213104248, + 1.7649495601654053, + 1.8684518337249756, + 1.8378466367721558, + 1.8215986490249634, + 1.8962804079055786, + 1.8311480283737183, + 1.8144934177398682, + 1.8386142253875732, + 1.8001902103424072, + 1.8106772899627686, + 1.824654459953308, + 1.8877463340759277, + 1.7548255920410156, + 1.7998223304748535, + 1.7992620468139648, + 1.8008710145950317, + 1.7930411100387573, + 1.795708179473877, + 1.6782174110412598, + 1.7613928318023682, + 1.7696983814239502, + 1.8821675777435303, + 1.8363044261932373, + 1.7888398170471191, + 1.705260992050171, + 1.7461280822753906, + 1.7554951906204224, + 1.6823387145996094, + 1.6935639381408691, + 1.8532562255859375, + 1.705009937286377, + 1.7658536434173584, + 1.702729344367981, + 1.7277617454528809, + 1.7182602882385254, + 1.7201430797576904, + 1.7512156963348389, + 1.733093500137329, + 1.6640846729278564, + 1.795751929283142, + 1.7613760232925415, + 1.7927360534667969, + 1.7691571712493896, + 1.7088494300842285, + 1.6305524110794067, + 1.7121655941009521, + 1.7470346689224243, + 1.7565792798995972, + 1.7739824056625366, + 1.797484040260315, + 1.804025650024414, + 1.759737253189087, + 1.7241430282592773, + 1.6585729122161865, + 1.7364972829818726, + 1.6928026676177979, + 1.7578916549682617, + 1.7627092599868774, + 1.7360352277755737, + 1.651181936264038, + 1.7984654903411865, + 1.8190546035766602, + 1.7575874328613281, + 1.642033338546753, + 1.7083163261413574, + 1.7290922403335571, + 1.6048269271850586, + 1.7535103559494019, + 1.7353885173797607, + 1.7438924312591553, + 1.6532859802246094, + 1.6955783367156982, + 1.7544498443603516, + 1.66658616065979, + 1.7770192623138428, + 1.679250717163086, + 1.7292330265045166, + 1.5676822662353516, + 1.697801113128662, + 1.6611995697021484, + 1.716270089149475, + 1.6759263277053833, + 1.7309002876281738, + 1.624119758605957, + 1.6672643423080444, + 1.8094968795776367, + 1.668798804283142, + 1.6570839881896973, + 1.7465977668762207, + 1.70977783203125, + 1.6171870231628418, + 1.5880881547927856, + 1.7385530471801758, + 1.6310052871704102, + 1.6891758441925049, + 1.6656358242034912, + 1.6954612731933594, + 1.703312873840332, + 1.667173147201538, + 1.6587313413619995, + 1.6523313522338867, + 1.678445816040039, + 1.6593310832977295, + 1.7193293571472168, + 1.6987330913543701, + 1.6459376811981201, + 1.6476598978042603, + 1.695275068283081, + 1.6088767051696777, + 1.7676945924758911, + 1.6968809366226196, + 1.63032865524292, + 1.6114075183868408, + 1.5637303590774536, + 1.6193305253982544, + 1.6661876440048218, + 1.644110918045044, + 1.7304478883743286, + 1.6240438222885132, + 1.6382673978805542, + 1.6219370365142822, + 1.5746934413909912, + 1.6267046928405762, + 1.577682614326477, + 1.6462494134902954, + 1.6117165088653564, + 1.6581870317459106, + 1.5545706748962402, + 1.608431339263916, + 1.585486888885498, + 1.5679125785827637, + 1.6366214752197266, + 1.5967662334442139, + 1.6198058128356934, + 1.6043498516082764, + 1.5370299816131592, + 1.585712194442749, + 1.6290847063064575, + 1.5727028846740723, + 1.5931075811386108, + 1.5847282409667969 + ], + "elapsed_s": [ + 1.8958040979998714, + 3.7085601459998543, + 5.550835204999657, + 7.365839247999702, + 9.189696228999537, + 11.012317524999617, + 12.841704647999677, + 14.692720797999755, + 16.531073736999588, + 18.37673823399973, + 20.196632534999708, + 22.010096909999675, + 23.889242268999624, + 25.710181442999556, + 27.516948339999544, + 29.356042382999476, + 31.189079387999527, + 33.14141224599962, + 35.00589504599975, + 36.87411922099977, + 38.71569893599985, + 40.570328693999954, + 42.40581498100005, + 44.22505282399993, + 46.146571029999905, + 47.98401284099964, + 49.83636635199946, + 51.65189036799984, + 53.5433589649997, + 55.37347515099964, + 57.2046423529996, + 59.02224536099948, + 60.858090912999614, + 62.683987368999624, + 64.61375154699954, + 66.47862721199954, + 68.33418385499954, + 70.17072368699974, + 71.99258946099962, + 73.82351653599972, + 75.65545331899966, + 77.57957464599986, + 79.40441496699987, + 81.22259463299997, + 83.05120419899959, + 84.91344036199962, + 86.73987624899928, + 88.57301844099902, + 90.38952249199883, + 92.21697857699883, + 94.03037623799855, + 95.83832332599877, + 97.68607027899861, + 99.50291904599862, + 101.33366048699872, + 103.1661314719986, + 104.9933989649985, + 106.82613623699854, + 108.63270343599834, + 110.47961248099818, + 112.33426195199809, + 114.14680083899793, + 115.98156267499826, + 117.8645097009985, + 119.7688389889986, + 121.59625666499869, + 123.41075120299865, + 125.24509691999856, + 127.0935473569989, + 128.94829213399908, + 130.77901135799902, + 132.5737502839993, + 134.43038641899966, + 136.26653015899956, + 138.16203575999953, + 140.0123754469996, + 141.84824593099984, + 143.67900251799983, + 145.54372066499946, + 147.37310940899943, + 149.22978148999937, + 151.2175190829994, + 153.20674095099957, + 155.11861571499958, + 157.06209107699942, + 158.92895847799946, + 160.83059443999946, + 162.74950552599967, + 164.6194464289997, + 166.62176444999932, + 168.66528126199955, + 170.63144280599954, + 172.5728767519995, + 174.45251141599965, + 176.39602935499988, + 178.3836500299999, + 180.24197816800006, + 182.175162601, + 184.10406252500024, + 186.03717258500046, + 187.92154646000063, + 189.83680037400063, + 191.6619327580006, + 193.57560279600057, + 195.4262758550003, + 197.27100875900032, + 199.16055023300032, + 201.03381517900016, + 202.88534141100035, + 204.71710715700056, + 206.55825950100052, + 208.41805195400093, + 210.24527942500117, + 212.06400856100117, + 213.88152423300107, + 215.70745091400113, + 217.59890663000124, + 219.41541078200135, + 221.25666737100119, + 223.0772582360014, + 224.8633942240017, + 226.75793600500174, + 228.58896265400153, + 230.40415699400137, + 232.30770975600126, + 234.13194384600138, + 235.94446288700146, + 237.77371800200171, + 239.59881382400135, + 241.43388187100163, + 243.26380500300183, + 245.0853973490016, + 246.89388289100134, + 248.70521517000134, + 250.53019037400145, + 252.35132295600124, + 254.16799652800137, + 256.01590953400137, + 257.8408827270014, + 259.75922675200127, + 261.5824558790014, + 263.41046493300155, + 265.26586472400186, + 267.0971575230019, + 268.942354995002, + 270.7990017540019, + 272.6344680990019, + 274.4598827130021, + 276.2856795290022, + 278.1231601530021, + 279.93743732500207, + 281.79125983500217, + 283.5804458600023, + 285.4036829530023, + 287.2485311860023, + 289.07271973600245, + 290.91444822000267, + 292.7568109140029, + 294.5847325300033, + 296.4175468330036, + 298.2419021040032, + 300.08563976700316, + 301.94331892600303, + 303.76554190200295, + 305.5930965130028, + 307.39917946200285, + 309.2278781240029, + 311.0754859330027, + 312.911281588003, + 314.7364768170032, + 316.53051289900304, + 318.32086178100326, + 320.1335058630034, + 321.97694087000355, + 323.78743253300354, + 325.6223483370036, + 327.42513123600384, + 329.2340995830036, + 331.0580685000036, + 332.87889647400334, + 334.69277138100324, + 336.4997996160032, + 338.3087665670032, + 340.1316180420031, + 341.9329449130032, + 343.7358903940035, + 345.6315116100036, + 347.4309599600033, + 349.2425187230033, + 351.097583554003, + 352.9224142980029, + 354.712241978003, + 356.522717777003, + 358.34869248000314, + 360.19048085500344 + ], + "paths": { + "00 Conv2d": "_forward_fallback compute=float32", + "01 BatchNorm": "_forward_fallback", + "02 ReLU": "_forward_fallback", + "03 Conv2d": "_forward_fallback compute=float32", + "04 BatchNorm": "_forward_fallback", + "05 ReLU": "_forward_fallback", + "06 MaxPool2d": "_forward_fallback", + "07 SpatialDropout": "_forward_fallback", + "08 Conv2d": "_forward_fallback compute=float32", + "09 BatchNorm": "_forward_fallback", + "10 ReLU": "_forward_fallback", + "11 Conv2d": "_forward_fallback compute=float32", + "12 BatchNorm": "_forward_fallback", + "13 ReLU": "_forward_fallback", + "14 MaxPool2d": "_forward_fallback", + "15 SpatialDropout": "_forward_fallback", + "16 Conv2d": "_forward_fallback compute=float32", + "17 BatchNorm": "_forward_fallback", + "18 ReLU": "_forward_fallback", + "19 MaxPool2d": "_forward_fallback", + "20 SpatialDropout": "_forward_fallback", + "21 GlobalAvgPool": "_forward_fallback", + "22 Dense": "forward" + } + }, + { + "key": "cupy-fallback", + "label": "CuPy, no custom kernels", + "repeats": 3, + "ms_per_step": { + "median": 201.58676107812568, + "min": 201.09927703125052, + "max": 204.37771341406474 + }, + "loss": [ + 2.050244092941284, + 2.169557571411133, + 1.9149501323699951, + 2.0718536376953125, + 2.002157688140869, + 2.0375638008117676, + 2.052699089050293, + 1.982043743133545, + 2.059297561645508, + 1.9943273067474365, + 1.9805047512054443, + 1.9937212467193604, + 2.092176914215088, + 1.9922572374343872, + 1.9403393268585205, + 1.8806297779083252, + 2.0285677909851074, + 1.8158382177352905, + 1.9551903009414673, + 1.9003483057022095, + 1.9895870685577393, + 1.922872543334961, + 2.0252060890197754, + 2.0267457962036133, + 1.8638137578964233, + 1.9063284397125244, + 1.870794653892517, + 1.933653712272644, + 1.9907587766647339, + 1.8504135608673096, + 1.7913663387298584, + 1.9636610746383667, + 1.908558964729309, + 1.8384184837341309, + 1.9268795251846313, + 1.9819581508636475, + 2.0320911407470703, + 1.8301842212677002, + 1.8362927436828613, + 1.8785736560821533, + 1.928529977798462, + 1.8277077674865723, + 1.8038420677185059, + 1.8482439517974854, + 1.86944580078125, + 1.8838157653808594, + 1.9258801937103271, + 1.9347947835922241, + 1.8447542190551758, + 1.860733985900879, + 1.954971432685852, + 1.8832216262817383, + 1.783861517906189, + 1.8848577737808228, + 1.8553893566131592, + 1.8947219848632812, + 1.752833366394043, + 1.9399815797805786, + 1.8365615606307983, + 1.8447226285934448, + 1.809327244758606, + 1.878655195236206, + 1.8331681489944458, + 1.8694372177124023, + 1.832263469696045, + 1.8125402927398682, + 1.8776212930679321, + 1.8305275440216064, + 1.8055369853973389, + 1.8240002393722534, + 1.820615291595459, + 1.7312304973602295, + 1.752691388130188, + 1.7842376232147217, + 1.8814702033996582, + 1.927754282951355, + 1.8640025854110718, + 1.811927318572998, + 1.7807466983795166, + 1.802025556564331, + 1.713542103767395, + 1.6942030191421509, + 1.6885327100753784, + 1.839198112487793, + 1.788475751876831, + 1.8419432640075684, + 1.8297669887542725, + 1.7505438327789307, + 1.738976240158081, + 1.692451000213623, + 1.7840774059295654, + 1.7670233249664307, + 1.683346152305603, + 1.8201241493225098, + 1.7999591827392578, + 1.8015508651733398, + 1.728576421737671, + 1.7484736442565918, + 1.6750972270965576, + 1.7520562410354614, + 1.734535574913025, + 1.7470780611038208, + 1.7771127223968506, + 1.824328899383545, + 1.72517728805542, + 1.7823797464370728, + 1.6905739307403564, + 1.7611205577850342, + 1.714794397354126, + 1.7161951065063477, + 1.762512445449829, + 1.77056884765625, + 1.7549519538879395, + 1.739363670349121, + 1.8037829399108887, + 1.827418565750122, + 1.7756507396697998, + 1.6127355098724365, + 1.712945580482483, + 1.7912685871124268, + 1.6841257810592651, + 1.7061219215393066, + 1.6816555261611938, + 1.734632134437561, + 1.6633644104003906, + 1.736448049545288, + 1.7324703931808472, + 1.669004201889038, + 1.799729824066162, + 1.7590692043304443, + 1.7452812194824219, + 1.6124508380889893, + 1.7553813457489014, + 1.6486122608184814, + 1.7154546976089478, + 1.643859624862671, + 1.7271931171417236, + 1.726668119430542, + 1.6676337718963623, + 1.824512004852295, + 1.6814123392105103, + 1.709265112876892, + 1.7508752346038818, + 1.695322036743164, + 1.5848252773284912, + 1.6300818920135498, + 1.7727594375610352, + 1.6657283306121826, + 1.740891695022583, + 1.7133915424346924, + 1.684468388557434, + 1.7227206230163574, + 1.6783078908920288, + 1.586474895477295, + 1.71865713596344, + 1.665873408317566, + 1.5904326438903809, + 1.7484369277954102, + 1.7031651735305786, + 1.6337049007415771, + 1.7226192951202393, + 1.7140307426452637, + 1.6723031997680664, + 1.7362068891525269, + 1.721008062362671, + 1.6450533866882324, + 1.6746370792388916, + 1.5607542991638184, + 1.6630122661590576, + 1.6104576587677002, + 1.6628860235214233, + 1.632922887802124, + 1.6349077224731445, + 1.6574883460998535, + 1.6628937721252441, + 1.662520408630371, + 1.6993842124938965, + 1.584362268447876, + 1.65537428855896, + 1.623317003250122, + 1.6577507257461548, + 1.5733284950256348, + 1.6498746871948242, + 1.62353515625, + 1.597511649131775, + 1.5296579599380493, + 1.6236283779144287, + 1.6854090690612793, + 1.5913312435150146, + 1.5898863077163696, + 1.6512720584869385, + 1.584123134613037, + 1.5591955184936523, + 1.6615731716156006, + 1.6399985551834106 + ], + "elapsed_s": [ + 0.21375163700031408, + 0.41355823000048986, + 0.6144736900005228, + 0.8149403690003965, + 1.0142828410002949, + 1.2141657420006595, + 1.414361564000501, + 1.6173844950003513, + 1.820487814000444, + 2.0307678700005454, + 2.245078291000482, + 2.4574995130005846, + 2.6670375660005448, + 2.8678806070001883, + 3.0824337770000056, + 3.282280532000186, + 3.483377505000135, + 3.6828491650003343, + 3.8821230990001823, + 4.081600664000234, + 4.281762580000304, + 4.483860612000171, + 4.684059617000003, + 4.886873074999585, + 5.087348178999946, + 5.287791950000155, + 5.488035878000119, + 5.687735497000176, + 5.888123324000389, + 6.088603784000497, + 6.288370692000171, + 6.487444022000091, + 6.688617980000345, + 6.88956401500036, + 7.090023645000656, + 7.2891813150004054, + 7.488288375000593, + 7.690161017000264, + 7.897255303000293, + 8.096945644000243, + 8.299153558000398, + 8.499243416000354, + 8.699084054000195, + 8.9000144620004, + 9.113663301000543, + 9.313535886000864, + 9.52909762300078, + 9.729414344000816, + 9.929830507000588, + 10.129772319000494, + 10.329580537000766, + 10.534271978001016, + 10.739546381001219, + 10.943540627001312, + 11.142882446001295, + 11.344162231001519, + 11.545638562001386, + 11.746754768001665, + 11.94654436700148, + 12.148117024001294, + 12.356403290001253, + 12.558925205001287, + 12.759469977001572, + 12.961313151001832, + 13.161246743001811, + 13.361026506001508, + 13.562541430001602, + 13.762427869001385, + 13.962128792001295, + 14.16141918500125, + 14.360478888001126, + 14.559437155000978, + 14.77203003500108, + 14.971749010001076, + 15.171904901001199, + 15.371363121001195, + 15.57948907500122, + 15.796589319001214, + 16.005012568001348, + 16.218470944001638, + 16.422341691001293, + 16.641244252001343, + 16.84652094700141, + 17.046394319001138, + 17.25015782300079, + 17.455005555000753, + 17.658946097000808, + 17.860814620000838, + 18.062598929000615, + 18.276727109000603, + 18.49191768000037, + 18.697468795000532, + 18.907379241000854, + 19.112113263000538, + 19.313940755000658, + 19.526775625000937, + 19.727374717001112, + 19.93090840500099, + 20.132138533001125, + 20.3338959910011, + 20.536744418001035, + 20.752665765000984, + 20.970741887001168, + 21.186404046000916, + 21.387914920001094, + 21.60168000100066, + 21.82762367400028, + 22.05029716600029, + 22.27584489500032, + 22.502624201000344, + 22.72470322800018, + 22.945479101000274, + 23.164267163000204, + 23.371282080000128, + 23.579629193000073, + 23.783095095000135, + 23.986351697999908, + 24.197851846999583, + 24.414366689999497, + 24.628684294999857, + 24.834960269999556, + 25.037458302999312, + 25.241120280998985, + 25.44614392399899, + 25.652833591998842, + 25.87174385799881, + 26.079104344998996, + 26.28328239399889, + 26.494978816999264, + 26.706546798999625, + 26.908050964999802, + 27.109306321000076, + 27.312307547999808, + 27.514257240000006, + 27.71525342599989, + 27.91791590699995, + 28.12004299199998, + 28.326306374999604, + 28.52748393499951, + 28.729482520999227, + 28.932071223999174, + 29.145327354999154, + 29.35654021899927, + 29.55943778299934, + 29.76011894599924, + 29.963118017999022, + 30.165480235998984, + 30.368061877999025, + 30.582063308999295, + 30.79601042899958, + 31.000324219999584, + 31.20345828299969, + 31.407872895999844, + 31.625827648999802, + 31.841584513999805, + 32.04253184500021, + 32.24488768900028, + 32.44993102100034, + 32.65245983800014, + 32.85582707799995, + 33.05815958999983, + 33.26274464700009, + 33.468509625000024, + 33.67306931600024, + 33.87792057000024, + 34.082232489000035, + 34.286144869000054, + 34.50007074400037, + 34.717019430000164, + 34.924521066000125, + 35.12888240200027, + 35.34977236900022, + 35.56656965900038, + 35.76964168600034, + 35.97886241500055, + 36.19536698900038, + 36.40591050400053, + 36.608114306000516, + 36.80911407400072, + 37.01300962100095, + 37.21438547800108, + 37.42356542200105, + 37.64109919800103, + 37.85063423900101, + 38.0517966740008, + 38.2531069430006, + 38.45454638700039, + 38.6553481130004, + 38.857932018000156, + 39.05974483900036, + 39.260932744000456, + 39.46316630900037, + 39.66537407700025, + 39.867738003999875, + 40.072214712999994 + ], + "paths": { + "00 Conv2d": "_forward_fallback compute=float32", + "01 BatchNorm": "_forward_fallback", + "02 ReLU": "_forward_fallback", + "03 Conv2d": "_forward_fallback compute=float32", + "04 BatchNorm": "_forward_fallback", + "05 ReLU": "_forward_fallback", + "06 MaxPool2d": "_forward_fallback", + "07 SpatialDropout": "_forward_fallback", + "08 Conv2d": "_forward_fallback compute=float32", + "09 BatchNorm": "_forward_fallback", + "10 ReLU": "_forward_fallback", + "11 Conv2d": "_forward_fallback compute=float32", + "12 BatchNorm": "_forward_fallback", + "13 ReLU": "_forward_fallback", + "14 MaxPool2d": "_forward_fallback", + "15 SpatialDropout": "_forward_fallback", + "16 Conv2d": "_forward_fallback compute=float32", + "17 BatchNorm": "_forward_fallback", + "18 ReLU": "_forward_fallback", + "19 MaxPool2d": "_forward_fallback", + "20 SpatialDropout": "_forward_fallback", + "21 GlobalAvgPool": "_forward_fallback", + "22 Dense": "forward" + } + }, + { + "key": "cupy-fused", + "label": "CuPy, fused kernels", + "repeats": 3, + "ms_per_step": { + "median": 4.848292476562932, + "min": 4.843995906252729, + "max": 4.852403820315487 + }, + "loss": [ + 2.076085329055786, + 2.137629508972168, + 2.023575782775879, + 2.074655771255493, + 1.9516918659210205, + 2.085057258605957, + 2.0852837562561035, + 2.0160460472106934, + 2.0333032608032227, + 1.9665690660476685, + 2.036428689956665, + 2.009012460708618, + 2.0495615005493164, + 2.054725170135498, + 1.9475899934768677, + 1.932157278060913, + 1.9582693576812744, + 1.886281132698059, + 1.9540777206420898, + 1.933868646621704, + 1.9512799978256226, + 1.8926736116409302, + 2.0745792388916016, + 2.0560545921325684, + 1.7937196493148804, + 1.9730780124664307, + 1.9302711486816406, + 1.8917977809906006, + 2.0422117710113525, + 1.826196312904358, + 1.8384482860565186, + 1.846996784210205, + 1.9184064865112305, + 1.883291244506836, + 1.9080047607421875, + 1.9285774230957031, + 1.9853109121322632, + 2.004190444946289, + 1.8713338375091553, + 1.8244980573654175, + 1.8501002788543701, + 1.7421956062316895, + 1.8582911491394043, + 1.8884919881820679, + 1.7921348810195923, + 1.8148894309997559, + 1.9989341497421265, + 1.9104959964752197, + 1.8618805408477783, + 1.8455299139022827, + 1.920788049697876, + 1.8288335800170898, + 1.7034564018249512, + 1.8609447479248047, + 1.842788815498352, + 1.8286421298980713, + 1.850432276725769, + 1.8384077548980713, + 1.8720341920852661, + 1.8752222061157227, + 1.8524038791656494, + 1.8524396419525146, + 1.8458142280578613, + 1.843698501586914, + 1.823370337486267, + 1.7801573276519775, + 1.8233530521392822, + 1.8679194450378418, + 1.8162965774536133, + 1.7761602401733398, + 1.7964214086532593, + 1.7697720527648926, + 1.6941136121749878, + 1.8463006019592285, + 1.8061902523040771, + 1.9100863933563232, + 1.8307561874389648, + 1.8324172496795654, + 1.7553765773773193, + 1.7581908702850342, + 1.737762689590454, + 1.746254324913025, + 1.704662799835205, + 1.8085064888000488, + 1.7745211124420166, + 1.8265576362609863, + 1.7738960981369019, + 1.6988966464996338, + 1.719081163406372, + 1.7417521476745605, + 1.726519227027893, + 1.7914282083511353, + 1.717917561531067, + 1.7833521366119385, + 1.771710753440857, + 1.803269863128662, + 1.8274664878845215, + 1.717463731765747, + 1.7068440914154053, + 1.7816181182861328, + 1.823359727859497, + 1.758439302444458, + 1.6846251487731934, + 1.7928895950317383, + 1.7852060794830322, + 1.7891416549682617, + 1.720090627670288, + 1.6778934001922607, + 1.7210732698440552, + 1.7005434036254883, + 1.6515140533447266, + 1.7573087215423584, + 1.7318850755691528, + 1.752415657043457, + 1.7464392185211182, + 1.8816468715667725, + 1.786121129989624, + 1.6727650165557861, + 1.6977784633636475, + 1.9208194017410278, + 1.654852271080017, + 1.7265963554382324, + 1.7461271286010742, + 1.741877794265747, + 1.712090253829956, + 1.6741275787353516, + 1.793966293334961, + 1.683109164237976, + 1.7847589254379272, + 1.7133004665374756, + 1.8563545942306519, + 1.5646464824676514, + 1.6646482944488525, + 1.655566692352295, + 1.78272545337677, + 1.72202467918396, + 1.7802565097808838, + 1.643505573272705, + 1.6909863948822021, + 1.7643235921859741, + 1.724428415298462, + 1.6701302528381348, + 1.7683027982711792, + 1.645246148109436, + 1.6152939796447754, + 1.6425524950027466, + 1.7612714767456055, + 1.7748974561691284, + 1.7686967849731445, + 1.6814863681793213, + 1.7529995441436768, + 1.6525046825408936, + 1.705291986465454, + 1.639960527420044, + 1.6739590167999268, + 1.6815297603607178, + 1.6852803230285645, + 1.6986627578735352, + 1.6188870668411255, + 1.654383659362793, + 1.712929606437683, + 1.7006359100341797, + 1.6683077812194824, + 1.810966968536377, + 1.7334010601043701, + 1.692800760269165, + 1.6778521537780762, + 1.600088119506836, + 1.7091256380081177, + 1.6341663599014282, + 1.7260279655456543, + 1.7465977668762207, + 1.660423994064331, + 1.5941492319107056, + 1.6810617446899414, + 1.6313996315002441, + 1.6822810173034668, + 1.6349079608917236, + 1.644239068031311, + 1.5739290714263916, + 1.6502810716629028, + 1.6191176176071167, + 1.6465002298355103, + 1.6464755535125732, + 1.5962207317352295, + 1.5936721563339233, + 1.658672571182251, + 1.581221103668213, + 1.631819486618042, + 1.52236008644104, + 1.726784586906433, + 1.5952311754226685, + 1.6389493942260742, + 1.5890445709228516, + 1.6557414531707764 + ], + "elapsed_s": [ + 0.005345847000171489, + 0.01024844200037478, + 0.015143657000407984, + 0.020101539000279445, + 0.024966210000457068, + 0.029836048000561277, + 0.03465705400049046, + 0.03956797500086395, + 0.04448932000104833, + 0.049364175000846444, + 0.05424650100076178, + 0.05910689300071681, + 0.06402160900051967, + 0.06887722800047413, + 0.07373752300054548, + 0.07863668000027246, + 0.08353843800023242, + 0.08841077100032635, + 0.09337944300068557, + 0.09830677100035246, + 0.1032361860002311, + 0.10809834200017576, + 0.11297895000006974, + 0.11787310500039894, + 0.1228037170008065, + 0.12773962100072822, + 0.13267684800075585, + 0.13777784700096163, + 0.14266256900100416, + 0.14762286400127778, + 0.15251128900126787, + 0.15741754000146102, + 0.16229789500175684, + 0.1672086390021832, + 0.17211855500181628, + 0.17709140700208081, + 0.1820502620021216, + 0.18700876100228925, + 0.19194833000210565, + 0.1968484740018539, + 0.20174697900165484, + 0.20665081200195345, + 0.21157926000159932, + 0.2165093180014992, + 0.2214443350012516, + 0.22644409100121266, + 0.23138282300124047, + 0.23626047500101777, + 0.24121582100087835, + 0.24610554300079457, + 0.25104393900073774, + 0.2559749760007435, + 0.26088231700077813, + 0.2658332180008074, + 0.27075918500077023, + 0.2756845470007647, + 0.28063950200066756, + 0.2855305580010281, + 0.2904069020009956, + 0.2952599730010661, + 0.30018078800139847, + 0.3050468230012484, + 0.3099950380010341, + 0.3149297780009874, + 0.31984405500134017, + 0.3247252840010333, + 0.32965170200122884, + 0.33456850000129634, + 0.33951488900129334, + 0.3444310980012233, + 0.34930995100103246, + 0.3541778560011153, + 0.3590144560007502, + 0.363959132000673, + 0.36887267700058146, + 0.37375641700054985, + 0.37867183700063833, + 0.38357374500083097, + 0.3885009080004238, + 0.3933903540000756, + 0.39827962000026673, + 0.40319736400033435, + 0.4080891140001768, + 0.41299738100042305, + 0.4178820850002012, + 0.4227332440000282, + 0.42756850199975815, + 0.4325248769996506, + 0.4374070179997034, + 0.4423036789994512, + 0.44719097999950463, + 0.4520983649995287, + 0.45698957699960374, + 0.46188864399982776, + 0.46681878799972765, + 0.471667308999713, + 0.4765964029998031, + 0.48147409300008803, + 0.4863747469998998, + 0.49130557399985264, + 0.4962035229996218, + 0.5011065379994761, + 0.5060143889991195, + 0.5108886439988964, + 0.5158304389992736, + 0.5207452189993091, + 0.5256479719996605, + 0.5305533189994094, + 0.5354384789993674, + 0.5403813219995754, + 0.5453243389997624, + 0.5502967449997413, + 0.5552261389998421, + 0.560130702999686, + 0.5649811229995976, + 0.5698937709998972, + 0.5747867749996658, + 0.5796990429998914, + 0.5846273829997699, + 0.5894851449997986, + 0.5943988479998552, + 0.5993205249997118, + 0.6042467229995054, + 0.6091270619995157, + 0.6140083769996636, + 0.6189221479999105, + 0.6238296080000509, + 0.6287089430002197, + 0.6336279800002558, + 0.6386374450003132, + 0.6435312990001876, + 0.6484174500001245, + 0.6532516249999389, + 0.6581758789998275, + 0.6630572819999543, + 0.6679461270000502, + 0.672857593000117, + 0.6777917770004933, + 0.6826998360002108, + 0.6876314680002906, + 0.6925034240002788, + 0.6974169190002613, + 0.7023386770006255, + 0.7072570510008518, + 0.7121345290006502, + 0.716990839000573, + 0.7218872990006275, + 0.7268121470006008, + 0.7316862030006632, + 0.7366143490007744, + 0.7414535420007269, + 0.7463585440004863, + 0.751250185000572, + 0.7561267680002857, + 0.7610548500001642, + 0.7659719610005595, + 0.7708256250007253, + 0.7757775620007124, + 0.7806578700005957, + 0.785570232000282, + 0.7904491070003132, + 0.7953832260004674, + 0.8002838600004907, + 0.805171858000449, + 0.810006135000549, + 0.8149007830006667, + 0.8197862430006353, + 0.8246952470008182, + 0.8295364910009084, + 0.8344877570007156, + 0.8393695830004617, + 0.8443204920004064, + 0.8492113450001852, + 0.8541148700001031, + 0.8590182670004651, + 0.8639211040003829, + 0.868838957000662, + 0.8737841450006272, + 0.8787188990004324, + 0.8836491670003852, + 0.888554627000758, + 0.8933938780010067, + 0.8982646570007091, + 0.9031937280005877, + 0.9081027620004534, + 0.9129824740002732, + 0.9179435660003037, + 0.9228914030004489, + 0.9278201440001794, + 0.9326807150000604, + 0.9375839690001158, + 0.9424654970002848, + 0.9473304840003038, + 0.9522387890001482, + 0.9571312020002551 + ], + "paths": { + "00 Conv2d": "_forward_gpu compute=float16", + "01 BatchNorm": "_forward_gpu", + "02 ReLU": "_forward_fallback", + "03 Conv2d": "_forward_gpu compute=float16", + "04 BatchNorm": "_forward_gpu", + "05 ReLU": "_forward_fallback", + "06 MaxPool2d": "_forward_gpu", + "07 SpatialDropout": "_forward_gpu", + "08 Conv2d": "_forward_gpu compute=float16", + "09 BatchNorm": "_forward_gpu", + "10 ReLU": "_forward_fallback", + "11 Conv2d": "_forward_gpu compute=float16", + "12 BatchNorm": "_forward_gpu", + "13 ReLU": "_forward_fallback", + "14 MaxPool2d": "_forward_gpu", + "15 SpatialDropout": "_forward_gpu", + "16 Conv2d": "_forward_gpu compute=float16", + "17 BatchNorm": "_forward_gpu", + "18 ReLU": "_forward_fallback", + "19 MaxPool2d": "_forward_gpu", + "20 SpatialDropout": "_forward_gpu", + "21 GlobalAvgPool": "_forward_gpu", + "22 Dense": "forward" + } + } + ] +} diff --git a/benchmarks/history/v1.2.0/paths_fused_rtx3050.json b/benchmarks/history/v1.2.0/paths_fused_rtx3050.json new file mode 100644 index 0000000..0415191 --- /dev/null +++ b/benchmarks/history/v1.2.0/paths_fused_rtx3050.json @@ -0,0 +1,445 @@ +{ + "meta": { + "commit": "089667c28f9d396f32794c64ae9a594ad7bf7c6d", + "dirty": false, + "date": "2026-11-01T01:57:01", + "gpu": "NVIDIA GeForce RTX 3050 6GB Laptop GPU", + "cupy": "14.2.0", + "python": "3.14.10", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128 + }, + "results": [ + { + "key": "cupy-fused", + "label": "CuPy, fused kernels", + "repeats": 3, + "ms_per_step": { + "median": 21.99635078125084, + "min": 21.973965624999536, + "max": 22.114739843750897 + }, + "loss": [ + 2.0510988235473633, + 2.0803062915802, + 1.9955954551696777, + 2.052794933319092, + 2.1077706813812256, + 1.9987077713012695, + 2.0332517623901367, + 1.9413410425186157, + 1.9111073017120361, + 2.0481183528900146, + 1.9867525100708008, + 1.9869334697723389, + 2.0386366844177246, + 1.9867671728134155, + 1.880357027053833, + 1.9133176803588867, + 1.9640971422195435, + 1.8432940244674683, + 1.9328652620315552, + 1.9345953464508057, + 1.9528793096542358, + 1.8460173606872559, + 2.032604932785034, + 1.9836137294769287, + 1.8665893077850342, + 1.9502754211425781, + 1.8706142902374268, + 1.7898801565170288, + 1.9710627794265747, + 1.8476316928863525, + 1.853530764579773, + 1.8038136959075928, + 1.8552732467651367, + 1.9201691150665283, + 1.880555272102356, + 1.8940613269805908, + 1.9533995389938354, + 1.9050416946411133, + 1.8182966709136963, + 1.837931513786316, + 1.91290283203125, + 1.8110600709915161, + 1.886517882347107, + 1.8734792470932007, + 1.8419595956802368, + 1.850744366645813, + 1.90146803855896, + 1.839348554611206, + 1.8932545185089111, + 1.8307098150253296, + 1.8634250164031982, + 1.8557004928588867, + 1.8153128623962402, + 1.8044379949569702, + 1.7722467184066772, + 1.8176445960998535, + 1.8731396198272705, + 1.7996076345443726, + 1.8352277278900146, + 1.8524410724639893, + 1.8139907121658325, + 1.8084169626235962, + 1.7758666276931763, + 1.8119093179702759, + 1.8382668495178223, + 1.7771962881088257, + 1.8168871402740479, + 1.7029848098754883, + 1.8228214979171753, + 1.88217294216156, + 1.804142951965332, + 1.6915762424468994, + 1.751138687133789, + 1.7917473316192627, + 1.764882206916809, + 1.875752568244934, + 1.7914824485778809, + 1.7597646713256836, + 1.74778413772583, + 1.7305618524551392, + 1.793686866760254, + 1.6868637800216675, + 1.756786584854126, + 1.7929620742797852, + 1.7509750127792358, + 1.7688418626785278, + 1.7893537282943726, + 1.7447080612182617, + 1.72490656375885, + 1.7100661993026733, + 1.7796247005462646, + 1.6819143295288086, + 1.706071138381958, + 1.7627760171890259, + 1.7718136310577393, + 1.7953039407730103, + 1.7211374044418335, + 1.7174475193023682, + 1.6530368328094482, + 1.7392734289169312, + 1.700063705444336, + 1.7275402545928955, + 1.7131900787353516, + 1.7503565549850464, + 1.7516621351242065, + 1.828689694404602, + 1.748015284538269, + 1.7090773582458496, + 1.723413348197937, + 1.6651839017868042, + 1.7229852676391602, + 1.7714132070541382, + 1.7390841245651245, + 1.7797060012817383, + 1.738761067390442, + 1.8825913667678833, + 1.7323095798492432, + 1.667951226234436, + 1.6344307661056519, + 1.821832299232483, + 1.6253900527954102, + 1.7110257148742676, + 1.6744059324264526, + 1.7578908205032349, + 1.5920004844665527, + 1.6252367496490479, + 1.6833739280700684, + 1.6444921493530273, + 1.790570855140686, + 1.7601821422576904, + 1.7623875141143799, + 1.5921008586883545, + 1.7045081853866577, + 1.6181892156600952, + 1.7282168865203857, + 1.662742257118225, + 1.6225699186325073, + 1.732344388961792, + 1.6643882989883423, + 1.7520253658294678, + 1.6425979137420654, + 1.6426860094070435, + 1.8247851133346558, + 1.6780716180801392, + 1.6285512447357178, + 1.619199275970459, + 1.721043586730957, + 1.7895406484603882, + 1.719529151916504, + 1.6697832345962524, + 1.683603286743164, + 1.6806330680847168, + 1.714181661605835, + 1.5881810188293457, + 1.7029470205307007, + 1.6993943452835083, + 1.548094391822815, + 1.6517138481140137, + 1.6700429916381836, + 1.6400939226150513, + 1.6354295015335083, + 1.6587693691253662, + 1.7431620359420776, + 1.7770154476165771, + 1.7654131650924683, + 1.667027235031128, + 1.6387628316879272, + 1.5735595226287842, + 1.6691124439239502, + 1.5826337337493896, + 1.7048929929733276, + 1.6637192964553833, + 1.5956891775131226, + 1.61094069480896, + 1.6849169731140137, + 1.6065168380737305, + 1.6827387809753418, + 1.6073009967803955, + 1.6418511867523193, + 1.5539785623550415, + 1.6887847185134888, + 1.5482276678085327, + 1.649659276008606, + 1.631777286529541, + 1.645896077156067, + 1.5651921033859253, + 1.6433173418045044, + 1.6159318685531616, + 1.5905858278274536, + 1.509329080581665, + 1.6997723579406738, + 1.6136391162872314, + 1.6378604173660278, + 1.6783792972564697, + 1.636680006980896 + ], + "elapsed_s": [ + 0.023834599999872808, + 0.04620909999994183, + 0.06900709999990795, + 0.09141289999979563, + 0.1138232999999218, + 0.13601949999997487, + 0.15814100000011422, + 0.18046440000011899, + 0.20337660000018332, + 0.2258798000002571, + 0.24834000000009837, + 0.2711116000000402, + 0.29455110000003515, + 0.3176948000000266, + 0.34112370000002556, + 0.3646086999999625, + 0.38860569999997097, + 0.4132865999999922, + 0.43705439999985174, + 0.4608581999998478, + 0.48355389999983345, + 0.5062907999997606, + 0.5289568999996845, + 0.551667699999598, + 0.5740029999994931, + 0.5961635999994996, + 0.6187471999994614, + 0.6410847999993621, + 0.6632930999992368, + 0.6853805999992346, + 0.7076684999990448, + 0.7303646999989724, + 0.7527188999988539, + 0.7747640999989471, + 0.7968551999990723, + 0.8188760999989881, + 0.841584899999134, + 0.864278699999204, + 0.8868229999991399, + 0.9090776999992158, + 0.9314223999992919, + 0.9538829999994505, + 0.9764770999995562, + 0.998585799999546, + 1.0206376999994973, + 1.0426742999995895, + 1.065005699999574, + 1.0872945999994954, + 1.109256899999309, + 1.1313387999991846, + 1.1535434999991594, + 1.176242599999341, + 1.198558899999398, + 1.2209356999994725, + 1.243687899999486, + 1.2660666999995556, + 1.2888457999995353, + 1.3119466999996803, + 1.3352427999996053, + 1.3583514999995714, + 1.3812606999995296, + 1.4043040999995355, + 1.4269835999996303, + 1.449987599999531, + 1.47277479999957, + 1.495553999999629, + 1.517823999999564, + 1.5406680999994933, + 1.5627393999993728, + 1.585148399999298, + 1.6073275999995076, + 1.6293215999994572, + 1.6513997999995809, + 1.6741834999995717, + 1.696149999999534, + 1.7185666999996556, + 1.7406187999995382, + 1.7626144999996995, + 1.7848557999996046, + 1.8071302999996988, + 1.8293286999999054, + 1.8514917999998488, + 1.8737163999996937, + 1.8957891999996264, + 1.9178660999996282, + 1.940268299999616, + 1.962724199999684, + 1.9848883999995905, + 2.007189999999582, + 2.029154199999539, + 2.0513314999996055, + 2.0739045999994232, + 2.0959553999994114, + 2.117945299999292, + 2.1400842999992165, + 2.162464199999249, + 2.1849315999991177, + 2.2073800999992272, + 2.2302342999992106, + 2.2526809999992565, + 2.2752971999991587, + 2.2982718999992358, + 2.321203399999149, + 2.3439067999993313, + 2.366563499999529, + 2.3891419999993104, + 2.4120122999993328, + 2.434725599999183, + 2.457370899999205, + 2.479954599999246, + 2.50240519999943, + 2.524813199999471, + 2.547205599999643, + 2.5694133999995756, + 2.5923810999995567, + 2.615005299999666, + 2.6375851999996485, + 2.6597906999995757, + 2.6829304999996566, + 2.7054136999995535, + 2.727327199999536, + 2.7496227999995426, + 2.7719799999995303, + 2.794389299999466, + 2.816445599999497, + 2.838628599999538, + 2.860874799999465, + 2.883240899999464, + 2.905600599999616, + 2.9276832999996714, + 2.9498729999995703, + 2.972201299999597, + 2.9942460999996, + 3.0164591999996446, + 3.0385738999996192, + 3.0606423999995513, + 3.0829335999997056, + 3.105137499999728, + 3.1272739999997157, + 3.1494739999996, + 3.17185629999949, + 3.1944719999994504, + 3.2175236999994468, + 3.2399580999995123, + 3.2628438999995524, + 3.2851464999994278, + 3.3075577999993584, + 3.3327736999992794, + 3.355153199999222, + 3.377467999999226, + 3.3997709999994186, + 3.4220480999993015, + 3.4444647999991957, + 3.4670057999992423, + 3.4893886999991537, + 3.5116659999991953, + 3.5338487999990775, + 3.5567152999990412, + 3.579175699999041, + 3.6015352999991137, + 3.6239242999990893, + 3.6462497999991683, + 3.6686143999991145, + 3.690912299999127, + 3.713218099999267, + 3.7353553999994347, + 3.7575766999996176, + 3.7797795999995287, + 3.801949199999626, + 3.824442799999815, + 3.8465526999998474, + 3.869126499999993, + 3.8914472999999816, + 3.913803299999927, + 3.936364899999944, + 3.958321500000011, + 3.980393599999843, + 4.002461099999664, + 4.024517699999706, + 4.0472614999996495, + 4.069702199999483, + 4.092366699999502, + 4.114970099999482, + 4.13735629999951, + 4.160064499999407, + 4.182744499999444, + 4.205494999999473, + 4.228029899999456, + 4.250558499999443, + 4.273044699999446, + 4.295662899999343, + 4.318186199999218, + 4.3408822999992935, + 4.363282199999276, + 4.38608809999937 + ], + "paths": { + "00 Conv2d": "_forward_gpu compute=float16", + "01 BatchNorm": "_forward_gpu", + "02 ReLU": "_forward_fallback", + "03 Conv2d": "_forward_gpu compute=float16", + "04 BatchNorm": "_forward_gpu", + "05 ReLU": "_forward_fallback", + "06 MaxPool2d": "_forward_gpu", + "07 SpatialDropout": "_forward_gpu", + "08 Conv2d": "_forward_gpu compute=float16", + "09 BatchNorm": "_forward_gpu", + "10 ReLU": "_forward_fallback", + "11 Conv2d": "_forward_gpu compute=float16", + "12 BatchNorm": "_forward_gpu", + "13 ReLU": "_forward_fallback", + "14 MaxPool2d": "_forward_gpu", + "15 SpatialDropout": "_forward_gpu", + "16 Conv2d": "_forward_gpu compute=float16", + "17 BatchNorm": "_forward_gpu", + "18 ReLU": "_forward_fallback", + "19 MaxPool2d": "_forward_gpu", + "20 SpatialDropout": "_forward_gpu", + "21 GlobalAvgPool": "_forward_gpu", + "22 Dense": "forward" + } + } + ] +} diff --git a/benchmarks/history/v1.2.0/vram.json b/benchmarks/history/v1.2.0/vram.json new file mode 100644 index 0000000..858b39f --- /dev/null +++ b/benchmarks/history/v1.2.0/vram.json @@ -0,0 +1,7188 @@ +{ + "meta": { + "commit": "089667c28f9d396f32794c64ae9a594ad7bf7c6d", + "dirty": false, + "date": "2026-11-01T06:03:53", + "gpu": "AMD Radeon RX 9070 XT", + "cupy": "14.2.0", + "python": "3.14.4", + "batch_size": 256, + "compute_dtype": "float16", + "warmup_steps": 20, + "baseline_steps": 128, + "recorded_steps": 2, + "repeats": 3 + }, + "phase_widths": { + "forward": 0.4, + "loss": 0.1, + "backward": 0.4, + "optimizer": 0.1 + }, + "results": [ + { + "key": "cupy-fused", + "label": "CuPy, fused kernels", + "baseline_bytes": 472883200, + "peak_transient_bytes": 83887616, + "peak_x": 0.08440366972477065, + "peak_phase": "forward", + "x": [ + 0.0, + 0.003669724770642202, + 0.007339449541284404, + 0.011009174311926608, + 0.014678899082568808, + 0.01834862385321101, + 0.022018348623853216, + 0.025688073394495414, + 0.029357798165137616, + 0.03302752293577982, + 0.03669724770642202, + 0.04036697247706422, + 0.04403669724770643, + 0.047706422018348627, + 0.05137614678899083, + 0.05504587155963303, + 0.05871559633027523, + 0.062385321100917435, + 0.06605504587155964, + 0.06972477064220184, + 0.07339449541284404, + 0.07706422018348624, + 0.08073394495412844, + 0.08440366972477065, + 0.08807339449541286, + 0.09174311926605505, + 0.09541284403669725, + 0.09908256880733946, + 0.10275229357798166, + 0.10642201834862387, + 0.11009174311926606, + 0.11376146788990826, + 0.11743119266055047, + 0.12110091743119267, + 0.12477064220183487, + 0.12844036697247707, + 0.13211009174311927, + 0.13577981651376148, + 0.13944954128440368, + 0.14311926605504588, + 0.14678899082568808, + 0.15045871559633028, + 0.15412844036697249, + 0.15779816513761466, + 0.1614678899082569, + 0.1651376146788991, + 0.1688073394495413, + 0.1724770642201835, + 0.17614678899082573, + 0.1798165137614679, + 0.1834862385321101, + 0.1871559633027523, + 0.1908256880733945, + 0.19449541284403674, + 0.1981651376146789, + 0.2018348623853211, + 0.20550458715596331, + 0.20917431192660552, + 0.21284403669724775, + 0.21651376146788992, + 0.22018348623853212, + 0.22385321100917432, + 0.22752293577981653, + 0.23119266055045873, + 0.23486238532110093, + 0.23853211009174313, + 0.24220183486238533, + 0.24587155963302754, + 0.24954128440366974, + 0.25321100917431194, + 0.25688073394495414, + 0.26055045871559634, + 0.26422018348623855, + 0.26788990825688075, + 0.27155963302752295, + 0.27522935779816515, + 0.27889908256880735, + 0.28256880733944956, + 0.28623853211009176, + 0.28990825688073396, + 0.29357798165137616, + 0.2972477064220183, + 0.30091743119266057, + 0.30458715596330277, + 0.30825688073394497, + 0.3119266055045872, + 0.3155963302752293, + 0.3192660550458716, + 0.3229357798165138, + 0.326605504587156, + 0.3302752293577982, + 0.33394495412844033, + 0.3376146788990826, + 0.3412844036697248, + 0.344954128440367, + 0.3486238532110092, + 0.35229357798165145, + 0.3559633027522936, + 0.3596330275229358, + 0.363302752293578, + 0.3669724770642202, + 0.37064220183486246, + 0.3743119266055046, + 0.3779816513761468, + 0.381651376146789, + 0.3853211009174312, + 0.38899082568807347, + 0.3926605504587156, + 0.3963302752293578, + 0.4, + 0.4017543859649123, + 0.4035087719298246, + 0.4052631578947369, + 0.40701754385964917, + 0.40877192982456145, + 0.4105263157894737, + 0.41228070175438597, + 0.41403508771929826, + 0.41578947368421054, + 0.41754385964912283, + 0.4192982456140351, + 0.4210526315789474, + 0.4228070175438597, + 0.424561403508772, + 0.4263157894736842, + 0.4280701754385965, + 0.4298245614035088, + 0.43157894736842106, + 0.43333333333333335, + 0.43508771929824563, + 0.4368421052631579, + 0.4385964912280702, + 0.4403508771929825, + 0.4421052631578948, + 0.44385964912280707, + 0.44561403508771935, + 0.4473684210526316, + 0.44912280701754387, + 0.45087719298245615, + 0.45263157894736844, + 0.4543859649122807, + 0.456140350877193, + 0.4578947368421053, + 0.4596491228070176, + 0.4614035087719298, + 0.4631578947368421, + 0.4649122807017544, + 0.4666666666666667, + 0.46842105263157896, + 0.47017543859649125, + 0.47192982456140353, + 0.4736842105263158, + 0.4754385964912281, + 0.4771929824561404, + 0.4789473684210527, + 0.48070175438596496, + 0.48245614035087725, + 0.48421052631578954, + 0.48596491228070177, + 0.48771929824561405, + 0.48947368421052634, + 0.4912280701754386, + 0.4929824561403509, + 0.4947368421052632, + 0.4964912280701754, + 0.49824561403508777, + 0.5, + 0.5038095238095238, + 0.5076190476190476, + 0.5114285714285715, + 0.5152380952380953, + 0.5190476190476191, + 0.5228571428571429, + 0.5266666666666666, + 0.5304761904761904, + 0.5342857142857143, + 0.5380952380952381, + 0.5419047619047619, + 0.5457142857142857, + 0.5495238095238095, + 0.5533333333333333, + 0.5571428571428572, + 0.560952380952381, + 0.5647619047619048, + 0.5685714285714286, + 0.5723809523809524, + 0.5761904761904761, + 0.58, + 0.5838095238095238, + 0.5876190476190476, + 0.5914285714285714, + 0.5952380952380952, + 0.599047619047619, + 0.6028571428571429, + 0.6066666666666667, + 0.6104761904761905, + 0.6142857142857143, + 0.6180952380952381, + 0.621904761904762, + 0.6257142857142857, + 0.6295238095238096, + 0.6333333333333333, + 0.6371428571428571, + 0.6409523809523809, + 0.6447619047619048, + 0.6485714285714286, + 0.6523809523809524, + 0.6561904761904762, + 0.66, + 0.6638095238095238, + 0.6676190476190477, + 0.6714285714285715, + 0.6752380952380952, + 0.6790476190476191, + 0.6828571428571428, + 0.6866666666666666, + 0.6904761904761905, + 0.6942857142857143, + 0.6980952380952381, + 0.7019047619047619, + 0.7057142857142857, + 0.7095238095238096, + 0.7133333333333334, + 0.7171428571428571, + 0.720952380952381, + 0.7247619047619047, + 0.7285714285714285, + 0.7323809523809524, + 0.7361904761904762, + 0.74, + 0.7438095238095238, + 0.7476190476190476, + 0.7514285714285714, + 0.7552380952380953, + 0.7590476190476191, + 0.7628571428571429, + 0.7666666666666666, + 0.7704761904761905, + 0.7742857142857142, + 0.7780952380952382, + 0.7819047619047619, + 0.7857142857142857, + 0.7895238095238095, + 0.7933333333333333, + 0.7971428571428572, + 0.800952380952381, + 0.8047619047619048, + 0.8085714285714285, + 0.8123809523809524, + 0.8161904761904762, + 0.8200000000000001, + 0.8238095238095238, + 0.8276190476190476, + 0.8314285714285714, + 0.8352380952380953, + 0.839047619047619, + 0.8428571428571429, + 0.8466666666666667, + 0.8504761904761905, + 0.8542857142857143, + 0.8580952380952381, + 0.861904761904762, + 0.8657142857142858, + 0.8695238095238096, + 0.8733333333333333, + 0.8771428571428572, + 0.8809523809523809, + 0.8847619047619049, + 0.8885714285714286, + 0.8923809523809524, + 0.8961904761904762, + 1.0, + 1.0035714285714286, + 1.0071428571428571, + 1.0107142857142857, + 1.0142857142857142, + 1.0178571428571428, + 1.0214285714285714, + 1.025, + 1.0285714285714285, + 1.0321428571428573, + 1.0357142857142858, + 1.0392857142857144, + 1.042857142857143, + 1.0464285714285715, + 1.05, + 1.0535714285714286, + 1.0571428571428572, + 1.0607142857142857, + 1.0642857142857143, + 1.0678571428571428, + 1.0714285714285714, + 1.075, + 1.0785714285714285, + 1.082142857142857, + 1.0857142857142856, + 1.0892857142857142, + 1.092857142857143, + 1.0964285714285715, + 1.1, + 1.1035714285714286, + 1.1071428571428572, + 1.1107142857142858, + 1.1142857142857143, + 1.1178571428571429, + 1.1214285714285714, + 1.125, + 1.1285714285714286, + 1.1321428571428571, + 1.1357142857142857, + 1.1392857142857142, + 1.1428571428571428, + 1.1464285714285714, + 1.15, + 1.1535714285714285, + 1.1571428571428573, + 1.1607142857142858, + 1.1642857142857144, + 1.167857142857143, + 1.1714285714285715, + 1.175, + 1.1785714285714286, + 1.1821428571428572, + 1.1857142857142857, + 1.1892857142857143, + 1.1928571428571428, + 1.1964285714285714, + 1.2, + 1.2035714285714285, + 1.207142857142857, + 1.2107142857142856, + 1.2142857142857142, + 1.217857142857143, + 1.2214285714285715, + 1.225, + 1.2285714285714286, + 1.2321428571428572, + 1.2357142857142858, + 1.2392857142857143, + 1.2428571428571429, + 1.2464285714285714, + 1.25, + 1.2535714285714286, + 1.2571428571428571, + 1.2607142857142857, + 1.2642857142857142, + 1.2678571428571428, + 1.2714285714285714, + 1.275, + 1.2785714285714285, + 1.282142857142857, + 1.2857142857142856, + 1.2892857142857141, + 1.292857142857143, + 1.2964285714285715, + 1.3, + 1.3035714285714286, + 1.3071428571428572, + 1.3107142857142857, + 1.3142857142857143, + 1.3178571428571428, + 1.3214285714285714, + 1.325, + 1.3285714285714287, + 1.3321428571428573, + 1.3357142857142859, + 1.3392857142857144, + 1.342857142857143, + 1.3464285714285715, + 1.35, + 1.3535714285714286, + 1.3571428571428572, + 1.3607142857142858, + 1.3642857142857143, + 1.3678571428571429, + 1.3714285714285714, + 1.375, + 1.3785714285714286, + 1.3821428571428571, + 1.3857142857142857, + 1.3892857142857142, + 1.3928571428571428, + 1.3964285714285714, + 1.4, + 1.4017543859649122, + 1.4035087719298245, + 1.4052631578947368, + 1.407017543859649, + 1.4087719298245613, + 1.4105263157894736, + 1.412280701754386, + 1.4140350877192982, + 1.4157894736842105, + 1.4175438596491228, + 1.419298245614035, + 1.4210526315789473, + 1.4228070175438596, + 1.424561403508772, + 1.4263157894736842, + 1.4280701754385965, + 1.4298245614035088, + 1.431578947368421, + 1.4333333333333333, + 1.4350877192982456, + 1.4368421052631577, + 1.43859649122807, + 1.4403508771929823, + 1.4421052631578946, + 1.4438596491228068, + 1.4456140350877191, + 1.4473684210526314, + 1.4491228070175437, + 1.450877192982456, + 1.4526315789473683, + 1.4543859649122806, + 1.4561403508771928, + 1.4578947368421051, + 1.4596491228070174, + 1.4614035087719297, + 1.463157894736842, + 1.4649122807017543, + 1.4666666666666666, + 1.4684210526315788, + 1.4701754385964911, + 1.4719298245614034, + 1.4736842105263157, + 1.475438596491228, + 1.4771929824561403, + 1.4789473684210526, + 1.4807017543859649, + 1.4824561403508771, + 1.4842105263157894, + 1.4859649122807017, + 1.487719298245614, + 1.4894736842105263, + 1.4912280701754386, + 1.4929824561403509, + 1.4947368421052631, + 1.4964912280701754, + 1.4982456140350877, + 1.5, + 1.5038095238095237, + 1.5076190476190476, + 1.5114285714285713, + 1.5152380952380953, + 1.519047619047619, + 1.522857142857143, + 1.5266666666666666, + 1.5304761904761905, + 1.5342857142857143, + 1.5380952380952382, + 1.541904761904762, + 1.5457142857142858, + 1.5495238095238095, + 1.5533333333333332, + 1.5571428571428572, + 1.5609523809523809, + 1.5647619047619048, + 1.5685714285714285, + 1.5723809523809524, + 1.5761904761904761, + 1.58, + 1.5838095238095238, + 1.5876190476190477, + 1.5914285714285714, + 1.5952380952380953, + 1.599047619047619, + 1.6028571428571428, + 1.6066666666666667, + 1.6104761904761904, + 1.6142857142857143, + 1.618095238095238, + 1.621904761904762, + 1.6257142857142857, + 1.6295238095238096, + 1.6333333333333333, + 1.6371428571428572, + 1.640952380952381, + 1.6447619047619049, + 1.6485714285714286, + 1.6523809523809523, + 1.6561904761904762, + 1.66, + 1.6638095238095238, + 1.6676190476190476, + 1.6714285714285715, + 1.6752380952380952, + 1.6790476190476191, + 1.6828571428571428, + 1.6866666666666668, + 1.6904761904761905, + 1.6942857142857144, + 1.698095238095238, + 1.7019047619047618, + 1.7057142857142857, + 1.7095238095238094, + 1.7133333333333334, + 1.717142857142857, + 1.720952380952381, + 1.7247619047619047, + 1.7285714285714286, + 1.7323809523809524, + 1.7361904761904763, + 1.74, + 1.743809523809524, + 1.7476190476190476, + 1.7514285714285713, + 1.7552380952380953, + 1.7590476190476192, + 1.762857142857143, + 1.7666666666666666, + 1.7704761904761905, + 1.7742857142857142, + 1.7780952380952382, + 1.7819047619047619, + 1.7857142857142856, + 1.7895238095238095, + 1.7933333333333334, + 1.7971428571428572, + 1.8009523809523809, + 1.8047619047619048, + 1.8085714285714285, + 1.8123809523809524, + 1.8161904761904761, + 1.82, + 1.8238095238095238, + 1.8276190476190477, + 1.8314285714285714, + 1.8352380952380953, + 1.839047619047619, + 1.842857142857143, + 1.8466666666666667, + 1.8504761904761904, + 1.8542857142857143, + 1.8580952380952382, + 1.861904761904762, + 1.8657142857142857, + 1.8695238095238096, + 1.8733333333333333, + 1.8771428571428572, + 1.880952380952381, + 1.8847619047619049, + 1.8885714285714286, + 1.8923809523809525, + 1.8961904761904762 + ], + "transient_bytes": [ + 1536, + 3147264, + 6292992, + 3147264, + 6292992, + 6293504, + 9439232, + 6293504, + 6292992, + 3147264, + 3149312, + 3147264, + 36701696, + 33555968, + 67110400, + 67175936, + 67176448, + 33622016, + 33621504, + 33555968, + 50333184, + 50351616, + 50333184, + 83887616, + 50333184, + 33555968, + 67110400, + 67175936, + 67176448, + 33622016, + 33621504, + 33555968, + 41944576, + 33555968, + 41944576, + 33555968, + 1536, + 34304, + 67072, + 34304, + 8422912, + 8390144, + 12584448, + 12621312, + 12584448, + 29361664, + 20973056, + 16778752, + 33555968, + 33621504, + 33622528, + 16845312, + 16844288, + 16778752, + 25167360, + 25241088, + 25167360, + 41944576, + 25167360, + 16778752, + 33555968, + 33621504, + 33622528, + 16845312, + 16844288, + 16778752, + 20973056, + 16778752, + 20973056, + 16778752, + 1536, + 67072, + 132608, + 67072, + 4261376, + 4195840, + 6292992, + 6440448, + 6292992, + 14681600, + 10487296, + 8390144, + 16778752, + 16844288, + 16846336, + 8457728, + 8455680, + 8390144, + 10487296, + 8390144, + 10487296, + 8390144, + 1536, + 132608, + 263680, + 132608, + 2229760, + 132608, + 1536, + 132608, + 198144, + 200704, + 201216, + 198656, + 198144, + 67072, + 72192, + 77312, + 72192, + 82432, + 83456, + 93696, + 92672, + 102912, + 92672, + 93696, + 103936, + 102912, + 92672, + 82432, + 72192, + 82432, + 84480, + 86528, + 87552, + 85504, + 95744, + 96768, + 86528, + 87552, + 88576, + 87552, + 88576, + 89600, + 88576, + 87552, + 86528, + 85504, + 83456, + 73216, + 73728, + 74240, + 73728, + 72704, + 146432, + 146944, + 73216, + 73728, + 73216, + 73728, + 73216, + 75264, + 75776, + 73728, + 74240, + 74752, + 74240, + 74752, + 74240, + 73728, + 74240, + 84480, + 89600, + 84480, + 74240, + 73728, + 76288, + 76800, + 81920, + 82432, + 77312, + 76800, + 142336, + 273408, + 142336, + 76800, + 76288, + 73728, + 8192, + 2105344, + 8192, + 2105344, + 8192, + 8396800, + 8192, + 73728, + 75776, + 73728, + 8462336, + 73728, + 8192, + 4202496, + 6299648, + 6300160, + 6299648, + 6594560, + 6299648, + 10493952, + 6299648, + 4202496, + 8192, + 4202496, + 8192, + 16785408, + 8192, + 73728, + 74752, + 73728, + 16850944, + 73728, + 8192, + 8396800, + 16785408, + 16785920, + 16785408, + 16932864, + 16785408, + 33562624, + 16785408, + 8396800, + 8192, + 73728, + 74752, + 73728, + 16850944, + 73728, + 8192, + 8396800, + 12591104, + 12591616, + 12591104, + 12664832, + 12591104, + 20979712, + 12591104, + 8396800, + 8192, + 8396800, + 8192, + 33562624, + 8192, + 73728, + 74240, + 73728, + 33628160, + 73728, + 8192, + 16785408, + 33562624, + 33563136, + 33562624, + 33599488, + 33562624, + 67117056, + 33562624, + 16785408, + 8192, + 73728, + 74240, + 73728, + 33628160, + 73728, + 8192, + 16785408, + 18358272, + 18358784, + 18358272, + 18361856, + 18358272, + 16785408, + 8192, + 3072, + 2560, + 2048, + 1536, + 3147264, + 6292992, + 3147264, + 6292992, + 6293504, + 9439232, + 6293504, + 6292992, + 3147264, + 3149312, + 3147264, + 36701696, + 33555968, + 67110400, + 67175936, + 67176448, + 33622016, + 33621504, + 33555968, + 50333184, + 50351616, + 50333184, + 83887616, + 50333184, + 33555968, + 67110400, + 67175936, + 67176448, + 33622016, + 33621504, + 33555968, + 41944576, + 33555968, + 41944576, + 33555968, + 1536, + 34304, + 67072, + 34304, + 8422912, + 8390144, + 12584448, + 12621312, + 12584448, + 29361664, + 20973056, + 16778752, + 33555968, + 33621504, + 33622528, + 16845312, + 16844288, + 16778752, + 25167360, + 25241088, + 25167360, + 41944576, + 25167360, + 16778752, + 33555968, + 33621504, + 33622528, + 16845312, + 16844288, + 16778752, + 20973056, + 16778752, + 20973056, + 16778752, + 1536, + 67072, + 132608, + 67072, + 4261376, + 4195840, + 6292992, + 6440448, + 6292992, + 14681600, + 10487296, + 8390144, + 16778752, + 16844288, + 16846336, + 8457728, + 8455680, + 8390144, + 10487296, + 8390144, + 10487296, + 8390144, + 1536, + 132608, + 263680, + 132608, + 2229760, + 132608, + 1536, + 132608, + 198144, + 200704, + 201216, + 198656, + 198144, + 67072, + 72192, + 77312, + 72192, + 82432, + 83456, + 93696, + 92672, + 102912, + 92672, + 93696, + 103936, + 102912, + 92672, + 82432, + 72192, + 82432, + 84480, + 86528, + 87552, + 85504, + 95744, + 96768, + 86528, + 87552, + 88576, + 87552, + 88576, + 89600, + 88576, + 87552, + 86528, + 85504, + 83456, + 73216, + 73728, + 74240, + 73728, + 72704, + 146432, + 146944, + 73216, + 73728, + 73216, + 73728, + 73216, + 75264, + 75776, + 73728, + 74240, + 74752, + 74240, + 74752, + 74240, + 73728, + 74240, + 84480, + 89600, + 84480, + 74240, + 73728, + 76288, + 76800, + 81920, + 82432, + 77312, + 76800, + 142336, + 273408, + 142336, + 76800, + 76288, + 73728, + 8192, + 2105344, + 8192, + 2105344, + 8192, + 8396800, + 8192, + 73728, + 75776, + 73728, + 8462336, + 73728, + 8192, + 4202496, + 6299648, + 6300160, + 6299648, + 6594560, + 6299648, + 10493952, + 6299648, + 4202496, + 8192, + 4202496, + 8192, + 16785408, + 8192, + 73728, + 74752, + 73728, + 16850944, + 73728, + 8192, + 8396800, + 16785408, + 16785920, + 16785408, + 16932864, + 16785408, + 33562624, + 16785408, + 8396800, + 8192, + 73728, + 74752, + 73728, + 16850944, + 73728, + 8192, + 8396800, + 12591104, + 12591616, + 12591104, + 12664832, + 12591104, + 20979712, + 12591104, + 8396800, + 8192, + 8396800, + 8192, + 33562624, + 8192, + 73728, + 74240, + 73728, + 33628160, + 73728, + 8192, + 16785408, + 33562624, + 33563136, + 33562624, + 33599488, + 33562624, + 67117056, + 33562624, + 16785408, + 8192, + 73728, + 74240, + 73728, + 33628160, + 73728, + 8192, + 16785408, + 18358272, + 18358784, + 18358272, + 18361856, + 18358272, + 16785408, + 8192 + ] + }, + { + "key": "cupy-fallback", + "label": "CuPy, no custom kernels", + "baseline_bytes": 829266432, + "peak_transient_bytes": 440571392, + "peak_x": 0.053452115812917596, + "peak_phase": "forward", + "x": [ + 0.0, + 0.0008908685968819599, + 0.0017817371937639199, + 0.00267260579064588, + 0.0035634743875278397, + 0.004454342984409799, + 0.00534521158129176, + 0.00623608017817372, + 0.0071269487750556795, + 0.008017817371937639, + 0.008908685968819599, + 0.00979955456570156, + 0.01069042316258352, + 0.011581291759465479, + 0.01247216035634744, + 0.013363028953229399, + 0.014253897550111359, + 0.01514476614699332, + 0.016035634743875277, + 0.01692650334075724, + 0.017817371937639197, + 0.018708240534521157, + 0.01959910913140312, + 0.02048997772828508, + 0.02138084632516704, + 0.022271714922048998, + 0.023162583518930958, + 0.024053452115812918, + 0.02494432071269488, + 0.02583518930957684, + 0.026726057906458798, + 0.027616926503340758, + 0.028507795100222718, + 0.029398663697104678, + 0.03028953229398664, + 0.031180400890868598, + 0.032071269487750555, + 0.03296213808463252, + 0.03385300668151448, + 0.03474387527839644, + 0.035634743875278395, + 0.03652561247216036, + 0.037416481069042315, + 0.03830734966592427, + 0.03919821826280624, + 0.0400890868596882, + 0.04097995545657016, + 0.04187082405345212, + 0.04276169265033408, + 0.04365256124721604, + 0.044543429844097995, + 0.04543429844097996, + 0.046325167037861915, + 0.04721603563474388, + 0.048106904231625836, + 0.04899777282850779, + 0.04988864142538976, + 0.05077951002227172, + 0.05167037861915368, + 0.05256124721603564, + 0.053452115812917596, + 0.05434298440979956, + 0.055233853006681516, + 0.05612472160356348, + 0.057015590200445436, + 0.05790645879732739, + 0.058797327394209356, + 0.05968819599109131, + 0.06057906458797328, + 0.06146993318485524, + 0.062360801781737196, + 0.06325167037861916, + 0.06414253897550111, + 0.06503340757238307, + 0.06592427616926504, + 0.066815144766147, + 0.06770601336302896, + 0.06859688195991091, + 0.06948775055679288, + 0.07037861915367484, + 0.07126948775055679, + 0.07216035634743875, + 0.07305122494432072, + 0.07394209354120268, + 0.07483296213808463, + 0.0757238307349666, + 0.07661469933184854, + 0.07750556792873052, + 0.07839643652561248, + 0.07928730512249443, + 0.0801781737193764, + 0.08106904231625835, + 0.08195991091314032, + 0.08285077951002227, + 0.08374164810690424, + 0.08463251670378619, + 0.08552338530066816, + 0.08641425389755011, + 0.08730512249443208, + 0.08819599109131404, + 0.08908685968819599, + 0.08997772828507797, + 0.09086859688195992, + 0.09175946547884188, + 0.09265033407572383, + 0.0935412026726058, + 0.09443207126948776, + 0.09532293986636972, + 0.09621380846325167, + 0.09710467706013363, + 0.09799554565701558, + 0.09888641425389756, + 0.09977728285077953, + 0.10066815144766147, + 0.10155902004454344, + 0.10244988864142539, + 0.10334075723830737, + 0.10423162583518931, + 0.10512249443207128, + 0.10601336302895323, + 0.10690423162583519, + 0.10779510022271716, + 0.10868596881959912, + 0.10957683741648108, + 0.11046770601336303, + 0.111358574610245, + 0.11224944320712696, + 0.11314031180400892, + 0.11403118040089087, + 0.11492204899777284, + 0.11581291759465479, + 0.11670378619153676, + 0.11759465478841871, + 0.11848552338530068, + 0.11937639198218263, + 0.12026726057906459, + 0.12115812917594657, + 0.12204899777282852, + 0.12293986636971048, + 0.12383073496659243, + 0.12472160356347439, + 0.12561247216035637, + 0.12650334075723832, + 0.12739420935412027, + 0.12828507795100222, + 0.1291759465478842, + 0.13006681514476615, + 0.13095768374164812, + 0.13184855233853007, + 0.13273942093541202, + 0.133630289532294, + 0.13452115812917595, + 0.13541202672605793, + 0.13630289532293988, + 0.13719376391982183, + 0.13808463251670378, + 0.13897550111358575, + 0.1398663697104677, + 0.14075723830734968, + 0.14164810690423163, + 0.14253897550111358, + 0.14342984409799556, + 0.1443207126948775, + 0.14521158129175948, + 0.14610244988864143, + 0.14699331848552338, + 0.14788418708240536, + 0.1487750556792873, + 0.14966592427616926, + 0.15055679287305124, + 0.1514476614699332, + 0.15233853006681516, + 0.1532293986636971, + 0.15412026726057906, + 0.15501113585746104, + 0.155902004454343, + 0.15679287305122497, + 0.1576837416481069, + 0.15857461024498887, + 0.15946547884187084, + 0.1603563474387528, + 0.16124721603563474, + 0.1621380846325167, + 0.16302895322939867, + 0.16391982182628065, + 0.16481069042316257, + 0.16570155902004455, + 0.1665924276169265, + 0.16748329621380847, + 0.16837416481069045, + 0.16926503340757237, + 0.17015590200445435, + 0.17104677060133633, + 0.17193763919821828, + 0.17282850779510023, + 0.17371937639198218, + 0.17461024498886416, + 0.17550111358574613, + 0.17639198218262808, + 0.17728285077951003, + 0.17817371937639198, + 0.17906458797327396, + 0.17995545657015594, + 0.18084632516703786, + 0.18173719376391984, + 0.18262806236080179, + 0.18351893095768376, + 0.18440979955456574, + 0.18530066815144766, + 0.18619153674832964, + 0.1870824053452116, + 0.18797327394209357, + 0.18886414253897552, + 0.18975501113585747, + 0.19064587973273944, + 0.1915367483296214, + 0.19242761692650334, + 0.19331848552338532, + 0.19420935412026727, + 0.19510022271714925, + 0.19599109131403117, + 0.19688195991091315, + 0.19777282850779512, + 0.19866369710467707, + 0.19955456570155905, + 0.20044543429844097, + 0.20133630289532295, + 0.20222717149220493, + 0.20311804008908688, + 0.20400890868596883, + 0.20489977728285078, + 0.20579064587973275, + 0.20668151447661473, + 0.20757238307349665, + 0.20846325167037863, + 0.20935412026726058, + 0.21024498886414256, + 0.21113585746102453, + 0.21202672605790646, + 0.21291759465478843, + 0.21380846325167038, + 0.21469933184855236, + 0.2155902004454343, + 0.21648106904231626, + 0.21737193763919824, + 0.2182628062360802, + 0.21915367483296216, + 0.2200445434298441, + 0.22093541202672606, + 0.22182628062360804, + 0.22271714922049, + 0.22360801781737194, + 0.22449888641425392, + 0.22538975501113587, + 0.22628062360801784, + 0.22717149220489977, + 0.22806236080178174, + 0.22895322939866372, + 0.22984409799554567, + 0.23073496659242765, + 0.23162583518930957, + 0.23251670378619155, + 0.23340757238307352, + 0.23429844097995547, + 0.23518930957683742, + 0.23608017817371937, + 0.23697104677060135, + 0.23786191536748333, + 0.23875278396436525, + 0.23964365256124723, + 0.24053452115812918, + 0.24142538975501115, + 0.24231625835189313, + 0.24320712694877505, + 0.24409799554565703, + 0.24498886414253898, + 0.24587973273942096, + 0.2467706013363029, + 0.24766146993318486, + 0.24855233853006684, + 0.24944320712694878, + 0.25033407572383076, + 0.25122494432071274, + 0.25211581291759466, + 0.25300668151447664, + 0.25389755011135856, + 0.25478841870824054, + 0.2556792873051225, + 0.25657015590200444, + 0.2574610244988864, + 0.2583518930957684, + 0.25924276169265037, + 0.2601336302895323, + 0.26102449888641427, + 0.26191536748329625, + 0.26280623608017817, + 0.26369710467706015, + 0.2645879732739421, + 0.26547884187082404, + 0.266369710467706, + 0.267260579064588, + 0.2681514476614699, + 0.2690423162583519, + 0.2699331848552339, + 0.27082405345211585, + 0.2717149220489978, + 0.27260579064587975, + 0.27349665924276173, + 0.27438752783964365, + 0.27527839643652563, + 0.27616926503340755, + 0.27706013363028953, + 0.2779510022271715, + 0.2788418708240535, + 0.2797327394209354, + 0.2806236080178174, + 0.28151447661469936, + 0.28240534521158134, + 0.28329621380846326, + 0.28418708240534524, + 0.28507795100222716, + 0.28596881959910914, + 0.2868596881959911, + 0.2877505567928731, + 0.288641425389755, + 0.289532293986637, + 0.29042316258351897, + 0.2913140311804009, + 0.29220489977728287, + 0.2930957683741648, + 0.29398663697104677, + 0.29487750556792874, + 0.2957683741648107, + 0.2966592427616927, + 0.2975501113585746, + 0.2984409799554566, + 0.2993318485523385, + 0.3002227171492205, + 0.3011135857461025, + 0.3020044543429844, + 0.3028953229398664, + 0.30378619153674835, + 0.30467706013363033, + 0.3055679287305123, + 0.3064587973273942, + 0.30734966592427615, + 0.3082405345211581, + 0.3091314031180401, + 0.3100222717149221, + 0.310913140311804, + 0.311804008908686, + 0.31269487750556796, + 0.31358574610244994, + 0.3144766146993319, + 0.3153674832962138, + 0.31625835189309576, + 0.31714922048997773, + 0.3180400890868597, + 0.3189309576837417, + 0.3198218262806236, + 0.3207126948775056, + 0.32160356347438757, + 0.3224944320712695, + 0.32338530066815147, + 0.3242761692650334, + 0.32516703786191536, + 0.32605790645879734, + 0.3269487750556793, + 0.3278396436525613, + 0.3287305122494432, + 0.32962138084632514, + 0.3305122494432071, + 0.3314031180400891, + 0.3322939866369711, + 0.333184855233853, + 0.33407572383073497, + 0.33496659242761695, + 0.3358574610244989, + 0.3367483296213809, + 0.33763919821826277, + 0.33853006681514475, + 0.3394209354120267, + 0.3403118040089087, + 0.3412026726057907, + 0.34209354120267266, + 0.3429844097995546, + 0.34387527839643656, + 0.34476614699331853, + 0.34565701559020046, + 0.34654788418708243, + 0.34743875278396436, + 0.34832962138084633, + 0.3492204899777283, + 0.3501113585746103, + 0.35100222717149226, + 0.3518930957683742, + 0.35278396436525616, + 0.3536748329621381, + 0.35456570155902006, + 0.35545657015590204, + 0.35634743875278396, + 0.35723830734966594, + 0.3581291759465479, + 0.3590200445434299, + 0.35991091314031187, + 0.36080178173719374, + 0.3616926503340757, + 0.3625835189309577, + 0.36347438752783967, + 0.36436525612472165, + 0.36525612472160357, + 0.36614699331848555, + 0.3670378619153675, + 0.3679287305122495, + 0.3688195991091315, + 0.36971046770601335, + 0.3706013363028953, + 0.3714922048997773, + 0.3723830734966593, + 0.37327394209354126, + 0.3741648106904232, + 0.37505567928730515, + 0.37594654788418713, + 0.37683741648106905, + 0.37772828507795103, + 0.37861915367483295, + 0.37951002227171493, + 0.3804008908685969, + 0.3812917594654789, + 0.38218262806236086, + 0.3830734966592428, + 0.3839643652561247, + 0.3848552338530067, + 0.38574610244988866, + 0.38663697104677064, + 0.38752783964365256, + 0.38841870824053454, + 0.3893095768374165, + 0.3902004454342985, + 0.39109131403118047, + 0.39198218262806234, + 0.3928730512249443, + 0.3937639198218263, + 0.39465478841870827, + 0.39554565701559025, + 0.39643652561247217, + 0.39732739420935415, + 0.3982182628062361, + 0.3991091314031181, + 0.4, + 0.4013333333333334, + 0.4026666666666667, + 0.404, + 0.4053333333333334, + 0.4066666666666667, + 0.40800000000000003, + 0.4093333333333334, + 0.4106666666666667, + 0.41200000000000003, + 0.41333333333333333, + 0.4146666666666667, + 0.41600000000000004, + 0.41733333333333333, + 0.4186666666666667, + 0.42000000000000004, + 0.42133333333333334, + 0.4226666666666667, + 0.42400000000000004, + 0.42533333333333334, + 0.4266666666666667, + 0.42800000000000005, + 0.42933333333333334, + 0.4306666666666667, + 0.43200000000000005, + 0.43333333333333335, + 0.4346666666666667, + 0.43600000000000005, + 0.43733333333333335, + 0.4386666666666667, + 0.44, + 0.44133333333333336, + 0.4426666666666667, + 0.444, + 0.44533333333333336, + 0.4466666666666667, + 0.448, + 0.44933333333333336, + 0.4506666666666667, + 0.452, + 0.45333333333333337, + 0.4546666666666667, + 0.456, + 0.45733333333333337, + 0.45866666666666667, + 0.46, + 0.4613333333333334, + 0.46266666666666667, + 0.464, + 0.4653333333333334, + 0.4666666666666667, + 0.468, + 0.4693333333333334, + 0.4706666666666667, + 0.47200000000000003, + 0.4733333333333334, + 0.4746666666666667, + 0.47600000000000003, + 0.4773333333333334, + 0.4786666666666667, + 0.48000000000000004, + 0.4813333333333334, + 0.4826666666666667, + 0.48400000000000004, + 0.4853333333333334, + 0.4866666666666667, + 0.48800000000000004, + 0.48933333333333334, + 0.4906666666666667, + 0.492, + 0.49333333333333335, + 0.4946666666666667, + 0.496, + 0.49733333333333335, + 0.4986666666666667, + 0.5, + 0.5010025062656641, + 0.5020050125313283, + 0.5030075187969925, + 0.5040100250626567, + 0.5050125313283208, + 0.5060150375939849, + 0.5070175438596491, + 0.5080200501253133, + 0.5090225563909775, + 0.5100250626566416, + 0.5110275689223057, + 0.51203007518797, + 0.5130325814536341, + 0.5140350877192983, + 0.5150375939849624, + 0.5160401002506265, + 0.5170426065162907, + 0.5180451127819549, + 0.5190476190476191, + 0.5200501253132832, + 0.5210526315789473, + 0.5220551378446115, + 0.5230576441102757, + 0.5240601503759399, + 0.525062656641604, + 0.5260651629072681, + 0.5270676691729324, + 0.5280701754385965, + 0.5290726817042607, + 0.5300751879699248, + 0.5310776942355889, + 0.5320802005012532, + 0.5330827067669173, + 0.5340852130325815, + 0.5350877192982456, + 0.5360902255639097, + 0.537092731829574, + 0.5380952380952381, + 0.5390977443609023, + 0.5401002506265664, + 0.5411027568922305, + 0.5421052631578948, + 0.5431077694235589, + 0.5441102756892231, + 0.5451127819548872, + 0.5461152882205513, + 0.5471177944862156, + 0.5481203007518797, + 0.5491228070175439, + 0.550125313283208, + 0.5511278195488722, + 0.5521303258145364, + 0.5531328320802005, + 0.5541353383458647, + 0.5551378446115288, + 0.5561403508771929, + 0.5571428571428572, + 0.5581453634085213, + 0.5591478696741855, + 0.5601503759398496, + 0.5611528822055138, + 0.562155388471178, + 0.5631578947368421, + 0.5641604010025063, + 0.5651629072681704, + 0.5661654135338345, + 0.5671679197994988, + 0.5681704260651629, + 0.5691729323308271, + 0.5701754385964912, + 0.5711779448621555, + 0.5721804511278196, + 0.5731829573934837, + 0.5741854636591479, + 0.575187969924812, + 0.5761904761904761, + 0.5771929824561404, + 0.5781954887218045, + 0.5791979949874687, + 0.5802005012531328, + 0.5812030075187969, + 0.5822055137844612, + 0.5832080200501253, + 0.5842105263157895, + 0.5852130325814536, + 0.5862155388471177, + 0.587218045112782, + 0.5882205513784461, + 0.5892230576441103, + 0.5902255639097744, + 0.5912280701754385, + 0.5922305764411028, + 0.5932330827067669, + 0.5942355889724311, + 0.5952380952380952, + 0.5962406015037595, + 0.5972431077694236, + 0.5982456140350877, + 0.5992481203007519, + 0.600250626566416, + 0.6012531328320803, + 0.6022556390977444, + 0.6032581453634085, + 0.6042606516290727, + 0.6052631578947368, + 0.6062656641604011, + 0.6072681704260652, + 0.6082706766917293, + 0.6092731829573935, + 0.6102756892230576, + 0.6112781954887219, + 0.612280701754386, + 0.6132832080200501, + 0.6142857142857143, + 0.6152882205513784, + 0.6162907268170427, + 0.6172932330827068, + 0.6182957393483709, + 0.6192982456140351, + 0.6203007518796992, + 0.6213032581453635, + 0.6223057644110276, + 0.6233082706766917, + 0.6243107769423559, + 0.62531328320802, + 0.6263157894736842, + 0.6273182957393484, + 0.6283208020050125, + 0.6293233082706767, + 0.6303258145363408, + 0.6313283208020051, + 0.6323308270676692, + 0.6333333333333333, + 0.6343358395989975, + 0.6353383458646616, + 0.6363408521303258, + 0.63734335839599, + 0.6383458646616541, + 0.6393483709273183, + 0.6403508771929824, + 0.6413533834586467, + 0.6423558897243108, + 0.6433583959899749, + 0.6443609022556391, + 0.6453634085213033, + 0.6463659147869674, + 0.6473684210526316, + 0.6483709273182957, + 0.6493734335839599, + 0.650375939849624, + 0.6513784461152883, + 0.6523809523809524, + 0.6533834586466165, + 0.6543859649122807, + 0.6553884711779449, + 0.656390977443609, + 0.6573934837092732, + 0.6583959899749373, + 0.6593984962406015, + 0.6604010025062657, + 0.6614035087719299, + 0.662406015037594, + 0.6634085213032581, + 0.6644110275689223, + 0.6654135338345865, + 0.6664160401002506, + 0.6674185463659148, + 0.6684210526315789, + 0.6694235588972431, + 0.6704260651629073, + 0.6714285714285715, + 0.6724310776942356, + 0.6734335839598997, + 0.674436090225564, + 0.6754385964912281, + 0.6764411027568922, + 0.6774436090225564, + 0.6784461152882205, + 0.6794486215538847, + 0.6804511278195489, + 0.6814536340852131, + 0.6824561403508772, + 0.6834586466165413, + 0.6844611528822055, + 0.6854636591478697, + 0.6864661654135338, + 0.687468671679198, + 0.6884711779448621, + 0.6894736842105263, + 0.6904761904761905, + 0.6914786967418547, + 0.6924812030075188, + 0.6934837092731829, + 0.6944862155388472, + 0.6954887218045113, + 0.6964912280701755, + 0.6974937343358396, + 0.6984962406015037, + 0.699498746867168, + 0.7005012531328321, + 0.7015037593984963, + 0.7025062656641604, + 0.7035087719298245, + 0.7045112781954888, + 0.7055137844611529, + 0.7065162907268171, + 0.7075187969924812, + 0.7085213032581453, + 0.7095238095238096, + 0.7105263157894737, + 0.7115288220551379, + 0.712531328320802, + 0.7135338345864661, + 0.7145363408521304, + 0.7155388471177945, + 0.7165413533834587, + 0.7175438596491228, + 0.7185463659147869, + 0.7195488721804512, + 0.7205513784461153, + 0.7215538847117795, + 0.7225563909774436, + 0.7235588972431077, + 0.724561403508772, + 0.7255639097744361, + 0.7265664160401003, + 0.7275689223057644, + 0.7285714285714286, + 0.7295739348370928, + 0.7305764411027569, + 0.7315789473684211, + 0.7325814536340852, + 0.7335839598997493, + 0.7345864661654136, + 0.7355889724310777, + 0.7365914786967419, + 0.737593984962406, + 0.7385964912280703, + 0.7395989974937344, + 0.7406015037593985, + 0.7416040100250627, + 0.7426065162907268, + 0.7436090225563909, + 0.7446115288220552, + 0.7456140350877193, + 0.7466165413533835, + 0.7476190476190476, + 0.7486215538847119, + 0.749624060150376, + 0.7506265664160401, + 0.7516290726817043, + 0.7526315789473684, + 0.7536340852130325, + 0.7546365914786968, + 0.7556390977443609, + 0.756641604010025, + 0.7576441102756892, + 0.7586466165413535, + 0.7596491228070176, + 0.7606516290726817, + 0.7616541353383459, + 0.76265664160401, + 0.7636591478696741, + 0.7646616541353384, + 0.7656641604010025, + 0.7666666666666666, + 0.7676691729323308, + 0.7686716791979951, + 0.7696741854636592, + 0.7706766917293233, + 0.7716791979949875, + 0.7726817042606516, + 0.7736842105263158, + 0.77468671679198, + 0.7756892230576441, + 0.7766917293233082, + 0.7776942355889724, + 0.7786967418546367, + 0.7796992481203008, + 0.7807017543859649, + 0.7817042606516291, + 0.7827067669172932, + 0.7837092731829574, + 0.7847117794486216, + 0.7857142857142857, + 0.7867167919799498, + 0.787719298245614, + 0.7887218045112783, + 0.7897243107769424, + 0.7907268170426065, + 0.7917293233082707, + 0.7927318295739348, + 0.793734335839599, + 0.7947368421052632, + 0.7957393483709273, + 0.7967418546365914, + 0.7977443609022556, + 0.7987468671679199, + 0.799749373433584, + 0.8007518796992481, + 0.8017543859649123, + 0.8027568922305764, + 0.8037593984962406, + 0.8047619047619048, + 0.8057644110275689, + 0.806766917293233, + 0.8077694235588972, + 0.8087719298245615, + 0.8097744360902256, + 0.8107769423558897, + 0.8117794486215539, + 0.812781954887218, + 0.8137844611528822, + 0.8147869674185464, + 0.8157894736842105, + 0.8167919799498746, + 0.8177944862155389, + 0.8187969924812031, + 0.8197994987468672, + 0.8208020050125313, + 0.8218045112781955, + 0.8228070175438597, + 0.8238095238095238, + 0.824812030075188, + 0.8258145363408521, + 0.8268170426065162, + 0.8278195488721805, + 0.8288220551378447, + 0.8298245614035087, + 0.8308270676691729, + 0.8318295739348371, + 0.8328320802005013, + 0.8338345864661655, + 0.8348370927318296, + 0.8358395989974937, + 0.8368421052631578, + 0.8378446115288221, + 0.8388471177944863, + 0.8398496240601503, + 0.8408521303258145, + 0.8418546365914787, + 0.8428571428571429, + 0.8438596491228071, + 0.8448621553884712, + 0.8458646616541353, + 0.8468671679197994, + 0.8478696741854637, + 0.8488721804511279, + 0.8498746867167919, + 0.8508771929824561, + 0.8518796992481203, + 0.8528822055137845, + 0.8538847117794487, + 0.8548872180451128, + 0.8558897243107769, + 0.856892230576441, + 0.8578947368421053, + 0.8588972431077695, + 0.8598997493734335, + 0.8609022556390977, + 0.861904761904762, + 0.8629072681704261, + 0.8639097744360903, + 0.8649122807017544, + 0.8659147869674185, + 0.8669172932330826, + 0.8679197994987469, + 0.8689223057644111, + 0.8699248120300751, + 0.8709273182957393, + 0.8719298245614036, + 0.8729323308270677, + 0.8739348370927319, + 0.874937343358396, + 0.8759398496240601, + 0.8769423558897242, + 0.8779448621553885, + 0.8789473684210527, + 0.8799498746867167, + 0.8809523809523809, + 0.8819548872180452, + 0.8829573934837093, + 0.8839598997493735, + 0.8849624060150376, + 0.8859649122807017, + 0.8869674185463658, + 0.8879699248120301, + 0.8889724310776943, + 0.8899749373433585, + 0.8909774436090225, + 0.8919799498746868, + 0.892982456140351, + 0.8939849624060151, + 0.8949874686716792, + 0.8959899749373433, + 0.8969924812030075, + 0.8979949874686717, + 0.8989974937343359, + 0.9, + 0.9001700680272109, + 0.9003401360544218, + 0.9005102040816326, + 0.9006802721088436, + 0.9008503401360545, + 0.9010204081632653, + 0.9011904761904762, + 0.9013605442176871, + 0.9015306122448979, + 0.9017006802721088, + 0.9018707482993198, + 0.9020408163265307, + 0.9022108843537415, + 0.9023809523809524, + 0.9025510204081633, + 0.9027210884353741, + 0.9028911564625851, + 0.903061224489796, + 0.9032312925170068, + 0.9034013605442177, + 0.9035714285714286, + 0.9037414965986394, + 0.9039115646258503, + 0.9040816326530613, + 0.9042517006802722, + 0.904421768707483, + 0.9045918367346939, + 0.9047619047619048, + 0.9049319727891156, + 0.9051020408163265, + 0.9052721088435375, + 0.9054421768707483, + 0.9056122448979592, + 0.9057823129251701, + 0.905952380952381, + 0.9061224489795918, + 0.9062925170068027, + 0.9064625850340137, + 0.9066326530612245, + 0.9068027210884354, + 0.9069727891156463, + 0.9071428571428571, + 0.907312925170068, + 0.907482993197279, + 0.9076530612244899, + 0.9078231292517007, + 0.9079931972789116, + 0.9081632653061225, + 0.9083333333333333, + 0.9085034013605442, + 0.9086734693877552, + 0.908843537414966, + 0.9090136054421769, + 0.9091836734693878, + 0.9093537414965986, + 0.9095238095238095, + 0.9096938775510204, + 0.9098639455782314, + 0.9100340136054422, + 0.9102040816326531, + 0.910374149659864, + 0.9105442176870748, + 0.9107142857142857, + 0.9108843537414966, + 0.9110544217687075, + 0.9112244897959184, + 0.9113945578231293, + 0.9115646258503401, + 0.911734693877551, + 0.9119047619047619, + 0.9120748299319729, + 0.9122448979591837, + 0.9124149659863946, + 0.9125850340136055, + 0.9127551020408163, + 0.9129251700680272, + 0.9130952380952381, + 0.913265306122449, + 0.9134353741496599, + 0.9136054421768708, + 0.9137755102040817, + 0.9139455782312925, + 0.9141156462585034, + 0.9142857142857143, + 0.9144557823129252, + 0.9146258503401361, + 0.914795918367347, + 0.9149659863945578, + 0.9151360544217687, + 0.9153061224489796, + 0.9154761904761904, + 0.9156462585034014, + 0.9158163265306123, + 0.9159863945578232, + 0.916156462585034, + 0.9163265306122449, + 0.9164965986394558, + 0.9166666666666667, + 0.9168367346938776, + 0.9170068027210885, + 0.9171768707482993, + 0.9173469387755102, + 0.9175170068027211, + 0.917687074829932, + 0.9178571428571429, + 0.9180272108843538, + 0.9181972789115647, + 0.9183673469387755, + 0.9185374149659864, + 0.9187074829931973, + 0.9188775510204081, + 0.9190476190476191, + 0.91921768707483, + 0.9193877551020408, + 0.9195578231292517, + 0.9197278911564626, + 0.9198979591836735, + 0.9200680272108843, + 0.9202380952380953, + 0.9204081632653062, + 0.920578231292517, + 0.9207482993197279, + 0.9209183673469388, + 0.9210884353741496, + 0.9212585034013606, + 0.9214285714285715, + 0.9215986394557824, + 0.9217687074829932, + 0.9219387755102041, + 0.922108843537415, + 0.9222789115646258, + 0.9224489795918368, + 0.9226190476190477, + 0.9227891156462585, + 0.9229591836734694, + 0.9231292517006803, + 0.9232993197278911, + 0.923469387755102, + 0.923639455782313, + 0.9238095238095239, + 0.9239795918367347, + 0.9241496598639456, + 0.9243197278911565, + 0.9244897959183673, + 0.9246598639455783, + 0.9248299319727892, + 0.925, + 0.9251700680272109, + 0.9253401360544218, + 0.9255102040816326, + 0.9256802721088435, + 0.9258503401360545, + 0.9260204081632654, + 0.9261904761904762, + 0.9263605442176871, + 0.926530612244898, + 0.9267006802721088, + 0.9268707482993197, + 0.9270408163265307, + 0.9272108843537415, + 0.9273809523809524, + 0.9275510204081633, + 0.9277210884353742, + 0.927891156462585, + 0.9280612244897959, + 0.9282312925170069, + 0.9284013605442177, + 0.9285714285714286, + 0.9287414965986395, + 0.9289115646258503, + 0.9290816326530612, + 0.9292517006802721, + 0.929421768707483, + 0.9295918367346939, + 0.9297619047619048, + 0.9299319727891157, + 0.9301020408163265, + 0.9302721088435374, + 0.9304421768707484, + 0.9306122448979592, + 0.9307823129251701, + 0.930952380952381, + 0.9311224489795918, + 0.9312925170068027, + 0.9314625850340136, + 0.9316326530612246, + 0.9318027210884354, + 0.9319727891156463, + 0.9321428571428572, + 0.932312925170068, + 0.9324829931972789, + 0.9326530612244899, + 0.9328231292517007, + 0.9329931972789116, + 0.9331632653061225, + 0.9333333333333333, + 0.9335034013605442, + 0.9336734693877551, + 0.9338435374149661, + 0.9340136054421769, + 0.9341836734693878, + 0.9343537414965987, + 0.9345238095238095, + 0.9346938775510204, + 0.9348639455782313, + 0.9350340136054422, + 0.9352040816326531, + 0.935374149659864, + 0.9355442176870749, + 0.9357142857142857, + 0.9358843537414966, + 0.9360544217687075, + 0.9362244897959184, + 0.9363945578231293, + 0.9365646258503402, + 0.936734693877551, + 0.9369047619047619, + 0.9370748299319728, + 0.9372448979591836, + 0.9374149659863946, + 0.9375850340136055, + 0.9377551020408164, + 0.9379251700680272, + 0.9380952380952381, + 0.938265306122449, + 0.9384353741496599, + 0.9386054421768708, + 0.9387755102040817, + 0.9389455782312925, + 0.9391156462585034, + 0.9392857142857143, + 0.9394557823129251, + 0.9396258503401361, + 0.939795918367347, + 0.9399659863945579, + 0.9401360544217687, + 0.9403061224489796, + 0.9404761904761905, + 0.9406462585034014, + 0.9408163265306123, + 0.9409863945578232, + 0.941156462585034, + 0.9413265306122449, + 0.9414965986394558, + 0.9416666666666667, + 0.9418367346938776, + 0.9420068027210885, + 0.9421768707482994, + 0.9423469387755102, + 0.9425170068027211, + 0.942687074829932, + 0.9428571428571428, + 0.9430272108843538, + 0.9431972789115647, + 0.9433673469387756, + 0.9435374149659864, + 0.9437074829931973, + 0.9438775510204082, + 0.944047619047619, + 0.94421768707483, + 0.9443877551020409, + 0.9445578231292517, + 0.9447278911564626, + 0.9448979591836735, + 0.9450680272108843, + 0.9452380952380952, + 0.9454081632653062, + 0.9455782312925171, + 0.9457482993197279, + 0.9459183673469388, + 0.9460884353741497, + 0.9462585034013605, + 0.9464285714285714, + 0.9465986394557824, + 0.9467687074829932, + 0.9469387755102041, + 0.947108843537415, + 0.9472789115646258, + 0.9474489795918367, + 0.9476190476190476, + 0.9477891156462586, + 0.9479591836734694, + 0.9481292517006803, + 0.9482993197278912, + 0.948469387755102, + 0.9486394557823129, + 0.9488095238095239, + 0.9489795918367347, + 0.9491496598639456, + 0.9493197278911565, + 0.9494897959183674, + 0.9496598639455782, + 0.9498299319727891, + 0.9500000000000001, + 0.9501700680272109, + 0.9503401360544218, + 0.9505102040816327, + 0.9506802721088435, + 0.9508503401360544, + 0.9510204081632654, + 0.9511904761904763, + 0.9513605442176871, + 0.951530612244898, + 0.9517006802721089, + 0.9518707482993197, + 0.9520408163265306, + 0.9522108843537416, + 0.9523809523809524, + 0.9525510204081633, + 0.9527210884353742, + 0.952891156462585, + 0.9530612244897959, + 0.9532312925170068, + 0.9534013605442178, + 0.9535714285714286, + 0.9537414965986395, + 0.9539115646258504, + 0.9540816326530612, + 0.9542517006802721, + 0.954421768707483, + 0.9545918367346939, + 0.9547619047619048, + 0.9549319727891157, + 0.9551020408163265, + 0.9552721088435374, + 0.9554421768707483, + 0.9556122448979593, + 0.9557823129251701, + 0.955952380952381, + 0.9561224489795919, + 0.9562925170068027, + 0.9564625850340136, + 0.9566326530612245, + 0.9568027210884353, + 0.9569727891156463, + 0.9571428571428572, + 0.957312925170068, + 0.9574829931972789, + 0.9576530612244898, + 0.9578231292517007, + 0.9579931972789116, + 0.9581632653061225, + 0.9583333333333334, + 0.9585034013605442, + 0.9586734693877551, + 0.958843537414966, + 0.959013605442177, + 0.9591836734693878, + 0.9593537414965987, + 0.9595238095238096, + 0.9596938775510204, + 0.9598639455782313, + 0.9600340136054422, + 0.9602040816326531, + 0.960374149659864, + 0.9605442176870749, + 0.9607142857142857, + 0.9608843537414966, + 0.9610544217687075, + 0.9612244897959183, + 0.9613945578231293, + 0.9615646258503402, + 0.9617346938775511, + 0.9619047619047619, + 0.9620748299319728, + 0.9622448979591837, + 0.9624149659863945, + 0.9625850340136055, + 0.9627551020408164, + 0.9629251700680272, + 0.9630952380952381, + 0.963265306122449, + 0.9634353741496599, + 0.9636054421768707, + 0.9637755102040817, + 0.9639455782312926, + 0.9641156462585034, + 0.9642857142857143, + 0.9644557823129252, + 0.964625850340136, + 0.9647959183673469, + 0.9649659863945579, + 0.9651360544217688, + 0.9653061224489796, + 0.9654761904761905, + 0.9656462585034014, + 0.9658163265306122, + 0.9659863945578232, + 0.9661564625850341, + 0.9663265306122449, + 0.9664965986394558, + 0.9666666666666667, + 0.9668367346938775, + 0.9670068027210885, + 0.9671768707482994, + 0.9673469387755103, + 0.9675170068027211, + 0.967687074829932, + 0.9678571428571429, + 0.9680272108843537, + 0.9681972789115647, + 0.9683673469387756, + 0.9685374149659864, + 0.9687074829931973, + 0.9688775510204082, + 0.969047619047619, + 0.9692176870748299, + 0.9693877551020409, + 0.9695578231292518, + 0.9697278911564626, + 0.9698979591836735, + 0.9700680272108844, + 0.9702380952380952, + 0.9704081632653061, + 0.9705782312925171, + 0.970748299319728, + 0.9709183673469388, + 0.9710884353741497, + 0.9712585034013606, + 0.9714285714285714, + 0.9715986394557823, + 0.9717687074829933, + 0.9719387755102041, + 0.972108843537415, + 0.9722789115646259, + 0.9724489795918367, + 0.9726190476190476, + 0.9727891156462585, + 0.9729591836734695, + 0.9731292517006803, + 0.9732993197278912, + 0.9734693877551021, + 0.9736394557823129, + 0.9738095238095239, + 0.9739795918367347, + 0.9741496598639456, + 0.9743197278911565, + 0.9744897959183674, + 0.9746598639455782, + 0.9748299319727891, + 0.975, + 0.975170068027211, + 0.9753401360544218, + 0.9755102040816327, + 0.9756802721088436, + 0.9758503401360544, + 0.9760204081632653, + 0.9761904761904763, + 0.9763605442176871, + 0.976530612244898, + 0.9767006802721089, + 0.9768707482993197, + 0.9770408163265306, + 0.9772108843537415, + 0.9773809523809525, + 0.9775510204081633, + 0.9777210884353742, + 0.9778911564625851, + 0.9780612244897959, + 0.9782312925170068, + 0.9784013605442177, + 0.9785714285714286, + 0.9787414965986395, + 0.9789115646258504, + 0.9790816326530613, + 0.9792517006802721, + 0.979421768707483, + 0.9795918367346939, + 0.9797619047619048, + 0.9799319727891157, + 0.9801020408163266, + 0.9802721088435374, + 0.9804421768707483, + 0.9806122448979592, + 0.98078231292517, + 0.980952380952381, + 0.9811224489795919, + 0.9812925170068028, + 0.9814625850340136, + 0.9816326530612245, + 0.9818027210884354, + 0.9819727891156462, + 0.9821428571428572, + 0.9823129251700681, + 0.9824829931972789, + 0.9826530612244898, + 0.9828231292517007, + 0.9829931972789115, + 0.9831632653061225, + 0.9833333333333334, + 0.9835034013605443, + 0.9836734693877551, + 0.983843537414966, + 0.9840136054421769, + 0.9841836734693877, + 0.9843537414965987, + 0.9845238095238096, + 0.9846938775510204, + 0.9848639455782313, + 0.9850340136054422, + 0.985204081632653, + 0.985374149659864, + 0.9855442176870749, + 0.9857142857142858, + 0.9858843537414966, + 0.9860544217687075, + 0.9862244897959184, + 0.9863945578231292, + 0.9865646258503402, + 0.9867346938775511, + 0.986904761904762, + 0.9870748299319728, + 0.9872448979591837, + 0.9874149659863946, + 0.9875850340136054, + 0.9877551020408164, + 0.9879251700680273, + 0.9880952380952381, + 0.988265306122449, + 0.9884353741496599, + 0.9886054421768707, + 0.9887755102040816, + 0.9889455782312926, + 0.9891156462585035, + 0.9892857142857143, + 0.9894557823129252, + 0.9896258503401361, + 0.9897959183673469, + 0.9899659863945578, + 0.9901360544217688, + 0.9903061224489796, + 0.9904761904761905, + 0.9906462585034014, + 0.9908163265306122, + 0.9909863945578231, + 0.991156462585034, + 0.991326530612245, + 0.9914965986394558, + 0.9916666666666667, + 0.9918367346938776, + 0.9920068027210884, + 0.9921768707482993, + 0.9923469387755103, + 0.9925170068027211, + 0.992687074829932, + 0.9928571428571429, + 0.9930272108843538, + 0.9931972789115646, + 0.9933673469387756, + 0.9935374149659865, + 0.9937074829931973, + 0.9938775510204082, + 0.9940476190476191, + 0.9942176870748299, + 0.9943877551020408, + 0.9945578231292518, + 0.9947278911564627, + 0.9948979591836735, + 0.9950680272108844, + 0.9952380952380953, + 0.9954081632653061, + 0.995578231292517, + 0.995748299319728, + 0.9959183673469388, + 0.9960884353741497, + 0.9962585034013606, + 0.9964285714285714, + 0.9965986394557823, + 0.9967687074829932, + 0.9969387755102042, + 0.997108843537415, + 0.9972789115646259, + 0.9974489795918368, + 0.9976190476190476, + 0.9977891156462585, + 0.9979591836734694, + 0.9981292517006803, + 0.9982993197278912, + 0.9984693877551021, + 0.998639455782313, + 0.9988095238095238, + 0.9989795918367347, + 0.9991496598639455, + 0.9993197278911565, + 0.9994897959183674, + 0.9996598639455783, + 0.9998299319727891, + 1.0, + 1.0008849557522124, + 1.0017699115044247, + 1.002654867256637, + 1.0035398230088495, + 1.0044247787610618, + 1.0053097345132744, + 1.0061946902654868, + 1.0070796460176992, + 1.0079646017699115, + 1.008849557522124, + 1.0097345132743363, + 1.0106194690265486, + 1.011504424778761, + 1.0123893805309734, + 1.0132743362831858, + 1.0141592920353983, + 1.0150442477876107, + 1.015929203539823, + 1.0168141592920354, + 1.0176991150442478, + 1.0185840707964602, + 1.0194690265486726, + 1.020353982300885, + 1.0212389380530973, + 1.0221238938053097, + 1.023008849557522, + 1.0238938053097346, + 1.024778761061947, + 1.0256637168141594, + 1.0265486725663717, + 1.027433628318584, + 1.0283185840707965, + 1.0292035398230088, + 1.0300884955752212, + 1.0309734513274336, + 1.031858407079646, + 1.0327433628318583, + 1.033628318584071, + 1.0345132743362833, + 1.0353982300884956, + 1.036283185840708, + 1.0371681415929204, + 1.0380530973451327, + 1.038938053097345, + 1.0398230088495575, + 1.0407079646017698, + 1.0415929203539822, + 1.0424778761061948, + 1.0433628318584072, + 1.0442477876106195, + 1.045132743362832, + 1.0460176991150443, + 1.0469026548672566, + 1.047787610619469, + 1.0486725663716814, + 1.0495575221238937, + 1.0504424778761061, + 1.0513274336283185, + 1.052212389380531, + 1.0530973451327434, + 1.0539823008849558, + 1.0548672566371682, + 1.0557522123893806, + 1.056637168141593, + 1.0575221238938053, + 1.0584070796460177, + 1.05929203539823, + 1.0601769911504424, + 1.061061946902655, + 1.0619469026548674, + 1.0628318584070797, + 1.063716814159292, + 1.0646017699115045, + 1.0654867256637168, + 1.0663716814159292, + 1.0672566371681416, + 1.068141592920354, + 1.0690265486725663, + 1.0699115044247787, + 1.0707964601769913, + 1.0716814159292036, + 1.072566371681416, + 1.0734513274336284, + 1.0743362831858407, + 1.075221238938053, + 1.0761061946902655, + 1.0769911504424778, + 1.0778761061946902, + 1.0787610619469026, + 1.079646017699115, + 1.0805309734513275, + 1.08141592920354, + 1.0823008849557523, + 1.0831858407079646, + 1.084070796460177, + 1.0849557522123894, + 1.0858407079646017, + 1.0867256637168141, + 1.0876106194690265, + 1.0884955752212389, + 1.0893805309734512, + 1.0902654867256638, + 1.0911504424778762, + 1.0920353982300885, + 1.092920353982301, + 1.0938053097345133, + 1.0946902654867257, + 1.095575221238938, + 1.0964601769911504, + 1.0973451327433628, + 1.0982300884955754, + 1.0991150442477877, + 1.1, + 1.1008849557522125, + 1.1017699115044248, + 1.1026548672566372, + 1.1035398230088496, + 1.104424778761062, + 1.1053097345132743, + 1.1061946902654867, + 1.107079646017699, + 1.1079646017699116, + 1.108849557522124, + 1.1097345132743364, + 1.1106194690265487, + 1.111504424778761, + 1.1123893805309735, + 1.1132743362831858, + 1.1141592920353982, + 1.1150442477876106, + 1.115929203539823, + 1.1168141592920353, + 1.117699115044248, + 1.1185840707964603, + 1.1194690265486726, + 1.120353982300885, + 1.1212389380530974, + 1.1221238938053097, + 1.1230088495575221, + 1.1238938053097345, + 1.1247787610619469, + 1.1256637168141592, + 1.1265486725663716, + 1.1274336283185842, + 1.1283185840707965, + 1.129203539823009, + 1.1300884955752213, + 1.1309734513274337, + 1.131858407079646, + 1.1327433628318584, + 1.1336283185840708, + 1.1345132743362831, + 1.1353982300884955, + 1.1362831858407079, + 1.1371681415929205, + 1.1380530973451328, + 1.1389380530973452, + 1.1398230088495576, + 1.14070796460177, + 1.1415929203539823, + 1.1424778761061947, + 1.143362831858407, + 1.1442477876106194, + 1.145132743362832, + 1.1460176991150441, + 1.1469026548672567, + 1.147787610619469, + 1.1486725663716815, + 1.1495575221238938, + 1.1504424778761062, + 1.1513274336283186, + 1.152212389380531, + 1.1530973451327433, + 1.1539823008849557, + 1.154867256637168, + 1.1557522123893806, + 1.156637168141593, + 1.1575221238938054, + 1.1584070796460177, + 1.1592920353982301, + 1.1601769911504425, + 1.1610619469026549, + 1.1619469026548672, + 1.1628318584070796, + 1.163716814159292, + 1.1646017699115045, + 1.1654867256637167, + 1.1663716814159293, + 1.1672566371681417, + 1.168141592920354, + 1.1690265486725664, + 1.1699115044247788, + 1.1707964601769911, + 1.1716814159292035, + 1.1725663716814159, + 1.1734513274336282, + 1.1743362831858408, + 1.1752212389380532, + 1.1761061946902656, + 1.176991150442478, + 1.1778761061946903, + 1.1787610619469027, + 1.179646017699115, + 1.1805309734513274, + 1.1814159292035398, + 1.1823008849557521, + 1.1831858407079645, + 1.184070796460177, + 1.1849557522123895, + 1.1858407079646018, + 1.1867256637168142, + 1.1876106194690266, + 1.188495575221239, + 1.1893805309734513, + 1.1902654867256637, + 1.191150442477876, + 1.1920353982300886, + 1.1929203539823008, + 1.1938053097345134, + 1.1946902654867257, + 1.1955752212389381, + 1.1964601769911505, + 1.1973451327433628, + 1.1982300884955752, + 1.1991150442477876, + 1.2, + 1.2008849557522123, + 1.2017699115044247, + 1.2026548672566373, + 1.2035398230088497, + 1.204424778761062, + 1.2053097345132744, + 1.2061946902654868, + 1.2070796460176991, + 1.2079646017699115, + 1.2088495575221239, + 1.2097345132743362, + 1.2106194690265486, + 1.2115044247787612, + 1.2123893805309733, + 1.213274336283186, + 1.2141592920353983, + 1.2150442477876107, + 1.215929203539823, + 1.2168141592920354, + 1.2176991150442478, + 1.2185840707964601, + 1.2194690265486725, + 1.2203539823008849, + 1.2212389380530975, + 1.2221238938053098, + 1.2230088495575222, + 1.2238938053097346, + 1.224778761061947, + 1.2256637168141593, + 1.2265486725663717, + 1.227433628318584, + 1.2283185840707964, + 1.2292035398230088, + 1.2300884955752212, + 1.2309734513274337, + 1.2318584070796461, + 1.2327433628318585, + 1.2336283185840708, + 1.2345132743362832, + 1.2353982300884956, + 1.236283185840708, + 1.2371681415929203, + 1.2380530973451327, + 1.238938053097345, + 1.2398230088495574, + 1.24070796460177, + 1.2415929203539824, + 1.2424778761061948, + 1.2433628318584071, + 1.2442477876106195, + 1.2451327433628319, + 1.2460176991150442, + 1.2469026548672566, + 1.247787610619469, + 1.2486725663716813, + 1.249557522123894, + 1.2504424778761063, + 1.2513274336283187, + 1.252212389380531, + 1.2530973451327434, + 1.2539823008849558, + 1.2548672566371681, + 1.2557522123893805, + 1.2566371681415929, + 1.2575221238938052, + 1.2584070796460178, + 1.25929203539823, + 1.2601769911504426, + 1.261061946902655, + 1.2619469026548673, + 1.2628318584070797, + 1.263716814159292, + 1.2646017699115044, + 1.2654867256637168, + 1.2663716814159292, + 1.2672566371681415, + 1.268141592920354, + 1.2690265486725663, + 1.2699115044247788, + 1.2707964601769912, + 1.2716814159292036, + 1.272566371681416, + 1.2734513274336283, + 1.2743362831858407, + 1.275221238938053, + 1.2761061946902656, + 1.2769911504424778, + 1.2778761061946904, + 1.2787610619469025, + 1.2796460176991151, + 1.2805309734513275, + 1.2814159292035399, + 1.2823008849557522, + 1.2831858407079646, + 1.284070796460177, + 1.2849557522123893, + 1.285840707964602, + 1.286725663716814, + 1.2876106194690267, + 1.288495575221239, + 1.2893805309734514, + 1.2902654867256638, + 1.2911504424778761, + 1.2920353982300885, + 1.2929203539823009, + 1.2938053097345132, + 1.2946902654867256, + 1.295575221238938, + 1.2964601769911503, + 1.297345132743363, + 1.2982300884955753, + 1.2991150442477877, + 1.3, + 1.3008849557522124, + 1.3017699115044248, + 1.3026548672566372, + 1.3035398230088495, + 1.3044247787610619, + 1.3053097345132743, + 1.3061946902654866, + 1.3070796460176992, + 1.3079646017699116, + 1.308849557522124, + 1.3097345132743363, + 1.3106194690265487, + 1.311504424778761, + 1.3123893805309734, + 1.3132743362831858, + 1.3141592920353982, + 1.3150442477876108, + 1.315929203539823, + 1.3168141592920355, + 1.3176991150442476, + 1.3185840707964602, + 1.3194690265486726, + 1.320353982300885, + 1.3212389380530973, + 1.3221238938053097, + 1.323008849557522, + 1.3238938053097344, + 1.324778761061947, + 1.3256637168141594, + 1.3265486725663718, + 1.3274336283185841, + 1.3283185840707965, + 1.3292035398230089, + 1.3300884955752212, + 1.3309734513274336, + 1.331858407079646, + 1.3327433628318583, + 1.3336283185840707, + 1.3345132743362833, + 1.3353982300884955, + 1.336283185840708, + 1.3371681415929204, + 1.3380530973451328, + 1.3389380530973451, + 1.3398230088495575, + 1.3407079646017699, + 1.3415929203539823, + 1.3424778761061948, + 1.343362831858407, + 1.3442477876106196, + 1.3451327433628317, + 1.3460176991150443, + 1.3469026548672567, + 1.347787610619469, + 1.3486725663716814, + 1.3495575221238938, + 1.3504424778761062, + 1.3513274336283185, + 1.3522123893805311, + 1.3530973451327435, + 1.3539823008849559, + 1.3548672566371682, + 1.3557522123893806, + 1.356637168141593, + 1.3575221238938053, + 1.3584070796460177, + 1.35929203539823, + 1.3601769911504424, + 1.3610619469026548, + 1.3619469026548674, + 1.3628318584070795, + 1.3637168141592921, + 1.3646017699115045, + 1.3654867256637169, + 1.3663716814159292, + 1.3672566371681416, + 1.368141592920354, + 1.3690265486725663, + 1.369911504424779, + 1.370796460176991, + 1.3716814159292037, + 1.3725663716814158, + 1.3734513274336284, + 1.3743362831858408, + 1.3752212389380531, + 1.3761061946902655, + 1.3769911504424779, + 1.3778761061946903, + 1.3787610619469026, + 1.3796460176991152, + 1.3805309734513274, + 1.38141592920354, + 1.3823008849557523, + 1.3831858407079647, + 1.384070796460177, + 1.3849557522123894, + 1.3858407079646018, + 1.3867256637168142, + 1.3876106194690265, + 1.388495575221239, + 1.3893805309734513, + 1.3902654867256636, + 1.3911504424778762, + 1.3920353982300886, + 1.392920353982301, + 1.3938053097345133, + 1.3946902654867257, + 1.395575221238938, + 1.3964601769911504, + 1.3973451327433628, + 1.3982300884955752, + 1.3991150442477875, + 1.4, + 1.4013333333333333, + 1.4026666666666665, + 1.404, + 1.4053333333333333, + 1.4066666666666665, + 1.408, + 1.4093333333333333, + 1.4106666666666665, + 1.412, + 1.4133333333333333, + 1.4146666666666665, + 1.416, + 1.4173333333333333, + 1.4186666666666665, + 1.42, + 1.4213333333333333, + 1.4226666666666665, + 1.424, + 1.4253333333333333, + 1.4266666666666665, + 1.428, + 1.4293333333333333, + 1.4306666666666665, + 1.432, + 1.4333333333333333, + 1.4346666666666665, + 1.436, + 1.4373333333333334, + 1.4386666666666665, + 1.44, + 1.4413333333333331, + 1.4426666666666665, + 1.444, + 1.4453333333333331, + 1.4466666666666665, + 1.448, + 1.4493333333333331, + 1.4506666666666665, + 1.452, + 1.4533333333333331, + 1.4546666666666666, + 1.456, + 1.4573333333333331, + 1.4586666666666666, + 1.46, + 1.4613333333333332, + 1.4626666666666666, + 1.464, + 1.4653333333333332, + 1.4666666666666666, + 1.468, + 1.4693333333333332, + 1.4706666666666666, + 1.472, + 1.4733333333333332, + 1.4746666666666666, + 1.476, + 1.4773333333333332, + 1.4786666666666666, + 1.48, + 1.4813333333333332, + 1.4826666666666666, + 1.484, + 1.4853333333333332, + 1.4866666666666666, + 1.488, + 1.4893333333333332, + 1.4906666666666666, + 1.492, + 1.4933333333333332, + 1.4946666666666666, + 1.496, + 1.4973333333333332, + 1.4986666666666666, + 1.5, + 1.5010025062656642, + 1.5020050125313282, + 1.5030075187969925, + 1.5040100250626567, + 1.5050125313283207, + 1.506015037593985, + 1.5070175438596491, + 1.5080200501253134, + 1.5090225563909774, + 1.5100250626566416, + 1.5110275689223058, + 1.5120300751879698, + 1.513032581453634, + 1.5140350877192983, + 1.5150375939849625, + 1.5160401002506265, + 1.5170426065162907, + 1.518045112781955, + 1.519047619047619, + 1.5200501253132832, + 1.5210526315789474, + 1.5220551378446114, + 1.5230576441102757, + 1.52406015037594, + 1.525062656641604, + 1.5260651629072681, + 1.5270676691729324, + 1.5280701754385966, + 1.5290726817042606, + 1.5300751879699248, + 1.531077694235589, + 1.532080200501253, + 1.5330827067669173, + 1.5340852130325815, + 1.5350877192982457, + 1.5360902255639097, + 1.537092731829574, + 1.5380952380952382, + 1.5390977443609022, + 1.5401002506265664, + 1.5411027568922306, + 1.5421052631578946, + 1.5431077694235589, + 1.544110275689223, + 1.545112781954887, + 1.5461152882205513, + 1.5471177944862156, + 1.5481203007518798, + 1.5491228070175438, + 1.550125313283208, + 1.5511278195488722, + 1.5521303258145362, + 1.5531328320802005, + 1.5541353383458647, + 1.555137844611529, + 1.556140350877193, + 1.5571428571428572, + 1.5581453634085214, + 1.5591478696741854, + 1.5601503759398496, + 1.5611528822055138, + 1.5621553884711779, + 1.563157894736842, + 1.5641604010025063, + 1.5651629072681703, + 1.5661654135338345, + 1.5671679197994988, + 1.568170426065163, + 1.569172932330827, + 1.5701754385964912, + 1.5711779448621555, + 1.5721804511278195, + 1.5731829573934837, + 1.574185463659148, + 1.5751879699248121, + 1.5761904761904761, + 1.5771929824561404, + 1.5781954887218046, + 1.5791979949874686, + 1.5802005012531328, + 1.581203007518797, + 1.582205513784461, + 1.5832080200501253, + 1.5842105263157895, + 1.5852130325814535, + 1.5862155388471177, + 1.587218045112782, + 1.5882205513784462, + 1.5892230576441102, + 1.5902255639097744, + 1.5912280701754387, + 1.5922305764411027, + 1.593233082706767, + 1.5942355889724311, + 1.5952380952380953, + 1.5962406015037593, + 1.5972431077694236, + 1.5982456140350878, + 1.5992481203007518, + 1.600250626566416, + 1.6012531328320803, + 1.6022556390977445, + 1.6032581453634085, + 1.6042606516290727, + 1.6052631578947367, + 1.606265664160401, + 1.6072681704260652, + 1.6082706766917294, + 1.6092731829573934, + 1.6102756892230576, + 1.6112781954887219, + 1.6122807017543859, + 1.61328320802005, + 1.6142857142857143, + 1.6152882205513786, + 1.6162907268170426, + 1.6172932330827068, + 1.618295739348371, + 1.619298245614035, + 1.6203007518796992, + 1.6213032581453635, + 1.6223057644110277, + 1.6233082706766917, + 1.624310776942356, + 1.62531328320802, + 1.6263157894736842, + 1.6273182957393484, + 1.6283208020050126, + 1.6293233082706766, + 1.6303258145363408, + 1.631328320802005, + 1.632330827067669, + 1.6333333333333333, + 1.6343358395989975, + 1.6353383458646618, + 1.6363408521303258, + 1.63734335839599, + 1.6383458646616542, + 1.6393483709273182, + 1.6403508771929824, + 1.6413533834586467, + 1.642355889724311, + 1.643358395989975, + 1.6443609022556391, + 1.6453634085213031, + 1.6463659147869674, + 1.6473684210526316, + 1.6483709273182958, + 1.6493734335839598, + 1.650375939849624, + 1.6513784461152883, + 1.6523809523809523, + 1.6533834586466165, + 1.6543859649122807, + 1.655388471177945, + 1.656390977443609, + 1.6573934837092732, + 1.6583959899749374, + 1.6593984962406014, + 1.6604010025062657, + 1.6614035087719299, + 1.6624060150375939, + 1.6634085213032581, + 1.6644110275689223, + 1.6654135338345863, + 1.6664160401002506, + 1.6674185463659148, + 1.668421052631579, + 1.6694235588972433, + 1.6704260651629073, + 1.6714285714285715, + 1.6724310776942355, + 1.6734335839598997, + 1.674436090225564, + 1.6754385964912282, + 1.6764411027568922, + 1.6774436090225564, + 1.6784461152882206, + 1.6794486215538846, + 1.6804511278195489, + 1.681453634085213, + 1.682456140350877, + 1.6834586466165413, + 1.6844611528822055, + 1.6854636591478696, + 1.6864661654135338, + 1.687468671679198, + 1.6884711779448622, + 1.6894736842105265, + 1.6904761904761905, + 1.6914786967418547, + 1.692481203007519, + 1.693483709273183, + 1.6944862155388472, + 1.6954887218045114, + 1.6964912280701754, + 1.6974937343358396, + 1.6984962406015038, + 1.6994987468671678, + 1.700501253132832, + 1.7015037593984963, + 1.7025062656641605, + 1.7035087719298245, + 1.7045112781954888, + 1.7055137844611528, + 1.706516290726817, + 1.7075187969924812, + 1.7085213032581454, + 1.7095238095238097, + 1.7105263157894737, + 1.711528822055138, + 1.7125313283208021, + 1.7135338345864661, + 1.7145363408521304, + 1.7155388471177946, + 1.7165413533834586, + 1.7175438596491228, + 1.718546365914787, + 1.719548872180451, + 1.7205513784461153, + 1.7215538847117795, + 1.7225563909774437, + 1.7235588972431077, + 1.724561403508772, + 1.725563909774436, + 1.7265664160401002, + 1.7275689223057644, + 1.7285714285714286, + 1.7295739348370929, + 1.7305764411027569, + 1.731578947368421, + 1.7325814536340853, + 1.7335839598997493, + 1.7345864661654136, + 1.7355889724310778, + 1.7365914786967418, + 1.737593984962406, + 1.7385964912280703, + 1.7395989974937343, + 1.7406015037593985, + 1.7416040100250627, + 1.742606516290727, + 1.743609022556391, + 1.7446115288220552, + 1.7456140350877192, + 1.7466165413533834, + 1.7476190476190476, + 1.7486215538847119, + 1.749624060150376, + 1.75062656641604, + 1.7516290726817043, + 1.7526315789473683, + 1.7536340852130325, + 1.7546365914786968, + 1.755639097744361, + 1.756641604010025, + 1.7576441102756892, + 1.7586466165413535, + 1.7596491228070175, + 1.7606516290726817, + 1.761654135338346, + 1.7626566416040101, + 1.7636591478696741, + 1.7646616541353384, + 1.7656641604010024, + 1.7666666666666666, + 1.7676691729323308, + 1.768671679197995, + 1.7696741854636593, + 1.7706766917293233, + 1.7716791979949875, + 1.7726817042606515, + 1.7736842105263158, + 1.77468671679198, + 1.7756892230576442, + 1.7766917293233082, + 1.7776942355889724, + 1.7786967418546367, + 1.7796992481203007, + 1.780701754385965, + 1.7817042606516291, + 1.7827067669172934, + 1.7837092731829574, + 1.7847117794486216, + 1.7857142857142856, + 1.7867167919799498, + 1.787719298245614, + 1.7887218045112783, + 1.7897243107769425, + 1.7907268170426065, + 1.7917293233082707, + 1.7927318295739347, + 1.793734335839599, + 1.7947368421052632, + 1.7957393483709274, + 1.7967418546365914, + 1.7977443609022556, + 1.7987468671679199, + 1.7997493734335839, + 1.800751879699248, + 1.8017543859649123, + 1.8027568922305766, + 1.8037593984962406, + 1.8047619047619048, + 1.8057644110275688, + 1.806766917293233, + 1.8077694235588972, + 1.8087719298245615, + 1.8097744360902257, + 1.8107769423558897, + 1.811779448621554, + 1.812781954887218, + 1.8137844611528822, + 1.8147869674185464, + 1.8157894736842106, + 1.8167919799498746, + 1.8177944862155389, + 1.818796992481203, + 1.819799498746867, + 1.8208020050125313, + 1.8218045112781955, + 1.8228070175438598, + 1.8238095238095238, + 1.824812030075188, + 1.825814536340852, + 1.8268170426065162, + 1.8278195488721805, + 1.8288220551378447, + 1.8298245614035087, + 1.830827067669173, + 1.8318295739348371, + 1.8328320802005011, + 1.8338345864661654, + 1.8348370927318296, + 1.8358395989974938, + 1.8368421052631578, + 1.837844611528822, + 1.8388471177944863, + 1.8398496240601503, + 1.8408521303258145, + 1.8418546365914787, + 1.842857142857143, + 1.8438596491228072, + 1.8448621553884712, + 1.8458646616541352, + 1.8468671679197994, + 1.8478696741854637, + 1.848872180451128, + 1.849874686716792, + 1.8508771929824561, + 1.8518796992481203, + 1.8528822055137844, + 1.8538847117794486, + 1.8548872180451128, + 1.855889724310777, + 1.856892230576441, + 1.8578947368421053, + 1.8588972431077695, + 1.8598997493734335, + 1.8609022556390977, + 1.861904761904762, + 1.8629072681704262, + 1.8639097744360904, + 1.8649122807017544, + 1.8659147869674184, + 1.8669172932330826, + 1.8679197994987469, + 1.868922305764411, + 1.869924812030075, + 1.8709273182957393, + 1.8719298245614036, + 1.8729323308270676, + 1.8739348370927318, + 1.874937343358396, + 1.8759398496240602, + 1.8769423558897242, + 1.8779448621553885, + 1.8789473684210527, + 1.8799498746867167, + 1.880952380952381, + 1.8819548872180452, + 1.8829573934837094, + 1.8839598997493736, + 1.8849624060150376, + 1.8859649122807016, + 1.8869674185463658, + 1.88796992481203, + 1.8889724310776943, + 1.8899749373433585, + 1.8909774436090225, + 1.8919799498746868, + 1.892982456140351, + 1.893984962406015, + 1.8949874686716792, + 1.8959899749373434, + 1.8969924812030075, + 1.8979949874686717, + 1.898997493734336, + 1.9, + 1.900170068027211, + 1.9003401360544216, + 1.9005102040816326, + 1.9006802721088434, + 1.9008503401360544, + 1.9010204081632651, + 1.901190476190476, + 1.901360544217687, + 1.9015306122448978, + 1.9017006802721088, + 1.9018707482993196, + 1.9020408163265305, + 1.9022108843537413, + 1.9023809523809523, + 1.9025510204081633, + 1.902721088435374, + 1.902891156462585, + 1.9030612244897958, + 1.9032312925170067, + 1.9034013605442175, + 1.9035714285714285, + 1.9037414965986394, + 1.9039115646258502, + 1.9040816326530612, + 1.904251700680272, + 1.904421768707483, + 1.904591836734694, + 1.9047619047619047, + 1.9049319727891156, + 1.9051020408163264, + 1.9052721088435374, + 1.9054421768707481, + 1.905612244897959, + 1.90578231292517, + 1.9059523809523808, + 1.9061224489795918, + 1.9062925170068026, + 1.9064625850340136, + 1.9066326530612243, + 1.9068027210884353, + 1.9069727891156463, + 1.907142857142857, + 1.907312925170068, + 1.9074829931972788, + 1.9076530612244897, + 1.9078231292517005, + 1.9079931972789115, + 1.9081632653061225, + 1.9083333333333332, + 1.9085034013605442, + 1.908673469387755, + 1.908843537414966, + 1.9090136054421767, + 1.9091836734693877, + 1.9093537414965986, + 1.9095238095238094, + 1.9096938775510204, + 1.9098639455782311, + 1.9100340136054421, + 1.9102040816326529, + 1.9103741496598639, + 1.9105442176870748, + 1.9107142857142856, + 1.9108843537414966, + 1.9110544217687073, + 1.9112244897959183, + 1.911394557823129, + 1.91156462585034, + 1.911734693877551, + 1.9119047619047618, + 1.9120748299319728, + 1.9122448979591835, + 1.9124149659863945, + 1.9125850340136052, + 1.9127551020408162, + 1.9129251700680272, + 1.913095238095238, + 1.913265306122449, + 1.9134353741496597, + 1.9136054421768707, + 1.9137755102040814, + 1.9139455782312924, + 1.9141156462585034, + 1.9142857142857141, + 1.9144557823129251, + 1.9146258503401359, + 1.9147959183673469, + 1.9149659863945578, + 1.9151360544217686, + 1.9153061224489796, + 1.9154761904761903, + 1.9156462585034013, + 1.915816326530612, + 1.915986394557823, + 1.916156462585034, + 1.9163265306122448, + 1.9164965986394558, + 1.9166666666666665, + 1.9168367346938775, + 1.9170068027210883, + 1.9171768707482992, + 1.9173469387755102, + 1.917517006802721, + 1.917687074829932, + 1.9178571428571427, + 1.9180272108843537, + 1.9181972789115644, + 1.9183673469387754, + 1.9185374149659864, + 1.9187074829931972, + 1.9188775510204081, + 1.919047619047619, + 1.9192176870748299, + 1.9193877551020406, + 1.9195578231292516, + 1.9197278911564626, + 1.9198979591836733, + 1.9200680272108843, + 1.920238095238095, + 1.920408163265306, + 1.9205782312925168, + 1.9207482993197278, + 1.9209183673469388, + 1.9210884353741495, + 1.9212585034013605, + 1.9214285714285713, + 1.9215986394557822, + 1.9217687074829932, + 1.921938775510204, + 1.922108843537415, + 1.9222789115646257, + 1.9224489795918367, + 1.9226190476190474, + 1.9227891156462584, + 1.9229591836734694, + 1.9231292517006802, + 1.9232993197278911, + 1.923469387755102, + 1.9236394557823129, + 1.9238095238095236, + 1.9239795918367346, + 1.9241496598639456, + 1.9243197278911564, + 1.9244897959183673, + 1.924659863945578, + 1.924829931972789, + 1.9249999999999998, + 1.9251700680272108, + 1.9253401360544218, + 1.9255102040816325, + 1.9256802721088435, + 1.9258503401360543, + 1.9260204081632653, + 1.926190476190476, + 1.926360544217687, + 1.926530612244898, + 1.9267006802721087, + 1.9268707482993197, + 1.9270408163265305, + 1.9272108843537414, + 1.9273809523809522, + 1.9275510204081632, + 1.9277210884353742, + 1.927891156462585, + 1.9280612244897959, + 1.9282312925170066, + 1.9284013605442176, + 1.9285714285714284, + 1.9287414965986394, + 1.9289115646258503, + 1.929081632653061, + 1.929251700680272, + 1.9294217687074828, + 1.9295918367346938, + 1.9297619047619046, + 1.9299319727891155, + 1.9301020408163265, + 1.9302721088435373, + 1.9304421768707483, + 1.930612244897959, + 1.93078231292517, + 1.9309523809523808, + 1.9311224489795917, + 1.9312925170068027, + 1.9314625850340135, + 1.9316326530612244, + 1.9318027210884352, + 1.9319727891156462, + 1.9321428571428572, + 1.932312925170068, + 1.932482993197279, + 1.9326530612244897, + 1.9328231292517006, + 1.9329931972789114, + 1.9331632653061224, + 1.9333333333333333, + 1.933503401360544, + 1.933673469387755, + 1.9338435374149658, + 1.9340136054421768, + 1.9341836734693876, + 1.9343537414965986, + 1.9345238095238095, + 1.9346938775510203, + 1.9348639455782313, + 1.935034013605442, + 1.935204081632653, + 1.9353741496598638, + 1.9355442176870747, + 1.9357142857142857, + 1.9358843537414965, + 1.9360544217687075, + 1.9362244897959182, + 1.9363945578231292, + 1.93656462585034, + 1.936734693877551, + 1.936904761904762, + 1.9370748299319727, + 1.9372448979591836, + 1.9374149659863944, + 1.9375850340136054, + 1.9377551020408161, + 1.9379251700680271, + 1.938095238095238, + 1.9382653061224488, + 1.9384353741496598, + 1.9386054421768706, + 1.9387755102040816, + 1.9389455782312925, + 1.9391156462585033, + 1.9392857142857143, + 1.939455782312925, + 1.939625850340136, + 1.9397959183673468, + 1.9399659863945578, + 1.9401360544217687, + 1.9403061224489795, + 1.9404761904761905, + 1.9406462585034012, + 1.9408163265306122, + 1.940986394557823, + 1.941156462585034, + 1.941326530612245, + 1.9414965986394557, + 1.9416666666666667, + 1.9418367346938774, + 1.9420068027210884, + 1.9421768707482991, + 1.9423469387755101, + 1.942517006802721, + 1.9426870748299319, + 1.9428571428571428, + 1.9430272108843536, + 1.9431972789115646, + 1.9433673469387753, + 1.9435374149659863, + 1.9437074829931973, + 1.943877551020408, + 1.944047619047619, + 1.9442176870748298, + 1.9443877551020408, + 1.9445578231292515, + 1.9447278911564625, + 1.9448979591836735, + 1.9450680272108842, + 1.9452380952380952, + 1.945408163265306, + 1.945578231292517, + 1.9457482993197277, + 1.9459183673469387, + 1.9460884353741497, + 1.9462585034013604, + 1.9464285714285714, + 1.9465986394557822, + 1.9467687074829931, + 1.946938775510204, + 1.9471088435374149, + 1.9472789115646258, + 1.9474489795918366, + 1.9476190476190476, + 1.9477891156462583, + 1.9479591836734693, + 1.94812925170068, + 1.948299319727891, + 1.948469387755102, + 1.9486394557823128, + 1.9488095238095238, + 1.9489795918367345, + 1.9491496598639455, + 1.9493197278911563, + 1.9494897959183672, + 1.9496598639455782, + 1.949829931972789, + 1.95, + 1.9501700680272107, + 1.9503401360544217, + 1.9505102040816327, + 1.9506802721088434, + 1.9508503401360544, + 1.9510204081632652, + 1.9511904761904761, + 1.951360544217687, + 1.9515306122448979, + 1.9517006802721089, + 1.9518707482993196, + 1.9520408163265306, + 1.9522108843537413, + 1.9523809523809523, + 1.952551020408163, + 1.952721088435374, + 1.952891156462585, + 1.9530612244897958, + 1.9532312925170068, + 1.9534013605442175, + 1.9535714285714285, + 1.9537414965986393, + 1.9539115646258503, + 1.9540816326530612, + 1.954251700680272, + 1.954421768707483, + 1.9545918367346937, + 1.9547619047619047, + 1.9549319727891155, + 1.9551020408163264, + 1.9552721088435374, + 1.9554421768707482, + 1.9556122448979592, + 1.95578231292517, + 1.9559523809523809, + 1.9561224489795916, + 1.9562925170068026, + 1.9564625850340136, + 1.9566326530612244, + 1.9568027210884353, + 1.956972789115646, + 1.957142857142857, + 1.957312925170068, + 1.9574829931972788, + 1.9576530612244898, + 1.9578231292517005, + 1.9579931972789115, + 1.9581632653061223, + 1.9583333333333333, + 1.9585034013605442, + 1.958673469387755, + 1.958843537414966, + 1.9590136054421767, + 1.9591836734693877, + 1.9593537414965985, + 1.9595238095238094, + 1.9596938775510204, + 1.9598639455782312, + 1.9600340136054422, + 1.960204081632653, + 1.960374149659864, + 1.9605442176870747, + 1.9607142857142856, + 1.9608843537414966, + 1.9610544217687074, + 1.9612244897959183, + 1.961394557823129, + 1.96156462585034, + 1.9617346938775508, + 1.9619047619047618, + 1.9620748299319728, + 1.9622448979591836, + 1.9624149659863945, + 1.9625850340136053, + 1.9627551020408163, + 1.962925170068027, + 1.963095238095238, + 1.963265306122449, + 1.9634353741496597, + 1.9636054421768707, + 1.9637755102040815, + 1.9639455782312925, + 1.9641156462585032, + 1.9642857142857142, + 1.9644557823129252, + 1.964625850340136, + 1.964795918367347, + 1.9649659863945577, + 1.9651360544217686, + 1.9653061224489794, + 1.9654761904761904, + 1.9656462585034014, + 1.9658163265306121, + 1.965986394557823, + 1.9661564625850338, + 1.9663265306122448, + 1.9664965986394556, + 1.9666666666666666, + 1.9668367346938775, + 1.9670068027210883, + 1.9671768707482993, + 1.96734693877551, + 1.967517006802721, + 1.967687074829932, + 1.9678571428571427, + 1.9680272108843537, + 1.9681972789115645, + 1.9683673469387755, + 1.9685374149659862, + 1.9687074829931972, + 1.968877551020408, + 1.969047619047619, + 1.96921768707483, + 1.9693877551020407, + 1.9695578231292517, + 1.9697278911564624, + 1.9698979591836734, + 1.9700680272108844, + 1.9702380952380951, + 1.970408163265306, + 1.9705782312925169, + 1.9707482993197278, + 1.9709183673469386, + 1.9710884353741496, + 1.9712585034013606, + 1.9714285714285713, + 1.9715986394557823, + 1.971768707482993, + 1.971938775510204, + 1.972108843537415, + 1.9722789115646258, + 1.9724489795918367, + 1.9726190476190475, + 1.9727891156462585, + 1.9729591836734692, + 1.9731292517006802, + 1.9732993197278912, + 1.973469387755102, + 1.973639455782313, + 1.9738095238095237, + 1.9739795918367347, + 1.9741496598639454, + 1.9743197278911564, + 1.9744897959183674, + 1.9746598639455781, + 1.974829931972789, + 1.9749999999999999, + 1.9751700680272108, + 1.9753401360544216, + 1.9755102040816326, + 1.9756802721088436, + 1.9758503401360543, + 1.9760204081632653, + 1.976190476190476, + 1.976360544217687, + 1.9765306122448978, + 1.9767006802721088, + 1.9768707482993197, + 1.9770408163265305, + 1.9772108843537415, + 1.9773809523809522, + 1.9775510204081632, + 1.977721088435374, + 1.977891156462585, + 1.978061224489796, + 1.9782312925170067, + 1.9784013605442177, + 1.9785714285714284, + 1.9787414965986394, + 1.9789115646258502, + 1.9790816326530611, + 1.9792517006802721, + 1.9794217687074829, + 1.9795918367346939, + 1.9797619047619046, + 1.9799319727891156, + 1.9801020408163263, + 1.9802721088435373, + 1.9804421768707483, + 1.980612244897959, + 1.98078231292517, + 1.9809523809523808, + 1.9811224489795918, + 1.9812925170068025, + 1.9814625850340135, + 1.9816326530612245, + 1.9818027210884352, + 1.9819727891156462, + 1.982142857142857, + 1.982312925170068, + 1.9824829931972787, + 1.9826530612244897, + 1.9828231292517007, + 1.9829931972789114, + 1.9831632653061224, + 1.9833333333333332, + 1.9835034013605441, + 1.983673469387755, + 1.9838435374149659, + 1.9840136054421769, + 1.9841836734693876, + 1.9843537414965986, + 1.9845238095238094, + 1.9846938775510203, + 1.9848639455782313, + 1.985034013605442, + 1.985204081632653, + 1.9853741496598638, + 1.9855442176870748, + 1.9857142857142855, + 1.9858843537414965, + 1.9860544217687073, + 1.9862244897959183, + 1.9863945578231292, + 1.98656462585034, + 1.986734693877551, + 1.9869047619047617, + 1.9870748299319727, + 1.9872448979591837, + 1.9874149659863944, + 1.9875850340136054, + 1.9877551020408162, + 1.9879251700680272, + 1.988095238095238, + 1.988265306122449, + 1.9884353741496599, + 1.9886054421768706, + 1.9887755102040816, + 1.9889455782312924, + 1.9891156462585033, + 1.989285714285714, + 1.989455782312925, + 1.989625850340136, + 1.9897959183673468, + 1.9899659863945578, + 1.9901360544217686, + 1.9903061224489795, + 1.9904761904761905, + 1.9906462585034013, + 1.9908163265306122, + 1.990986394557823, + 1.991156462585034, + 1.9913265306122447, + 1.9914965986394557, + 1.9916666666666667, + 1.9918367346938775, + 1.9920068027210884, + 1.9921768707482992, + 1.9923469387755102, + 1.992517006802721, + 1.992687074829932, + 1.9928571428571429, + 1.9930272108843536, + 1.9931972789115646, + 1.9933673469387754, + 1.9935374149659864, + 1.993707482993197, + 1.993877551020408, + 1.994047619047619, + 1.9942176870748298, + 1.9943877551020408, + 1.9945578231292516, + 1.9947278911564625, + 1.9948979591836733, + 1.9950680272108843, + 1.9952380952380953, + 1.995408163265306, + 1.995578231292517, + 1.9957482993197277, + 1.9959183673469387, + 1.9960884353741495, + 1.9962585034013605, + 1.9964285714285714, + 1.9965986394557822, + 1.9967687074829932, + 1.996938775510204, + 1.997108843537415, + 1.9972789115646257, + 1.9974489795918366, + 1.9976190476190476, + 1.9977891156462584, + 1.9979591836734694, + 1.9981292517006801, + 1.998299319727891, + 1.9984693877551019, + 1.9986394557823128, + 1.9988095238095238, + 1.9989795918367346, + 1.9991496598639455, + 1.9993197278911563, + 1.9994897959183673, + 1.999659863945578, + 1.999829931972789 + ], + "transient_bytes": [ + 1536, + 3147264, + 6292992, + 3147264, + 6292992, + 6293504, + 9439232, + 6293504, + 6292992, + 3147264, + 6698496, + 40252928, + 40256512, + 68568064, + 102122496, + 73810944, + 73807360, + 40252928, + 37107200, + 33555968, + 33556480, + 33556992, + 33557504, + 33556992, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 33555968, + 67110400, + 33555968, + 1536, + 33555968, + 67110400, + 33555968, + 67110400, + 33555968, + 71435776, + 104990208, + 105027072, + 407016960, + 440571392, + 138581504, + 138544640, + 104990208, + 71435776, + 33555968, + 33556480, + 33556992, + 33557504, + 33556992, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 33555968, + 67110400, + 33555968, + 1536, + 33555968, + 67110400, + 33555968, + 67110400, + 33555968, + 1536, + 8390144, + 1536, + 33555968, + 50333184, + 16778752, + 18875904, + 18876416, + 16779264, + 16778752, + 33555968, + 50333184, + 67110400, + 83887616, + 67110400, + 69207552, + 69208064, + 67110912, + 67110400, + 50333184, + 50333696, + 50334208, + 50336256, + 50336768, + 50337280, + 67114496, + 67113984, + 67114496, + 83891712, + 83891200, + 83893248, + 100670464, + 100668416, + 117445632, + 100668416, + 117445632, + 100668416, + 117445632, + 100668416, + 117445632, + 100668416, + 109057024, + 92279808, + 83891200, + 67113984, + 50336768, + 50336256, + 50334208, + 50333696, + 50333184, + 33555968, + 16778752, + 1536, + 83456, + 116224, + 124416, + 157184, + 148992, + 116224, + 124416, + 8513024, + 8504832, + 16893440, + 8504832, + 8472064, + 8390144, + 19006976, + 35784192, + 35857920, + 111355392, + 128132608, + 52635136, + 52561408, + 35784192, + 27395584, + 16778752, + 16779264, + 16779776, + 16780288, + 16779776, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 16778752, + 33555968, + 16778752, + 1536, + 16778752, + 33555968, + 16778752, + 33555968, + 16778752, + 38012416, + 54789632, + 54937088, + 205932032, + 222709248, + 71714304, + 71566848, + 54789632, + 38012416, + 16778752, + 16779264, + 16779776, + 16780288, + 16779776, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 16778752, + 33555968, + 16778752, + 1536, + 16778752, + 33555968, + 16778752, + 33555968, + 16778752, + 1536, + 4195840, + 1536, + 16778752, + 25167360, + 8390144, + 9438720, + 9439232, + 8390656, + 8390144, + 16778752, + 25167360, + 33555968, + 41944576, + 33555968, + 34604544, + 34605056, + 33556480, + 33555968, + 25167360, + 25167872, + 25168384, + 25170432, + 25170944, + 25171456, + 33560064, + 33559552, + 33560064, + 41948672, + 41948160, + 41950208, + 50338816, + 50336768, + 58725376, + 50336768, + 58725376, + 50336768, + 58725376, + 50336768, + 58725376, + 50336768, + 54531072, + 46142464, + 41948160, + 33559552, + 25170944, + 25170432, + 25168384, + 25167872, + 25167360, + 16778752, + 8390144, + 1536, + 83456, + 148992, + 165376, + 230912, + 214528, + 148992, + 165376, + 4359680, + 4343296, + 8537600, + 4343296, + 4277760, + 4195840, + 10749440, + 19138048, + 19432960, + 57181696, + 65570304, + 27821568, + 27526656, + 19138048, + 14943744, + 8390144, + 8390656, + 8391168, + 8391680, + 8391168, + 8391680, + 8391168, + 8391680, + 8392192, + 8392704, + 8392192, + 8391680, + 8391168, + 8391680, + 8392192, + 8392704, + 8392192, + 8391680, + 8391168, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 8390144, + 16778752, + 8390144, + 1536, + 8390144, + 16778752, + 8390144, + 16778752, + 8390144, + 1536, + 2098688, + 1536, + 8390144, + 12584448, + 4195840, + 4720128, + 4720640, + 4196352, + 4195840, + 8390144, + 12584448, + 16778752, + 20973056, + 16778752, + 17303040, + 17303552, + 16779264, + 16778752, + 12584448, + 12584960, + 12585472, + 12587520, + 12588544, + 12589056, + 16783360, + 16782848, + 16783360, + 20977664, + 20977152, + 20979200, + 25173504, + 25171456, + 29365760, + 25171456, + 29365760, + 25171456, + 29365760, + 25171456, + 29365760, + 25171456, + 27268608, + 23074304, + 20977152, + 16782848, + 12588544, + 12587520, + 12585472, + 12584960, + 12584448, + 8390144, + 4195840, + 1536, + 83456, + 214528, + 247296, + 378368, + 345600, + 214528, + 247296, + 2344448, + 2311680, + 4408832, + 2311680, + 214528, + 83456, + 1536, + 132608, + 198144, + 200704, + 201216, + 198656, + 198144, + 67072, + 72192, + 77312, + 72192, + 82432, + 83456, + 93696, + 92672, + 102912, + 92672, + 93696, + 103936, + 102912, + 92672, + 82432, + 72192, + 82432, + 84480, + 86528, + 87552, + 85504, + 95744, + 96768, + 86528, + 87552, + 88576, + 87552, + 88576, + 89600, + 88576, + 87552, + 86528, + 85504, + 83456, + 73216, + 73728, + 74240, + 73728, + 72704, + 146432, + 146944, + 73216, + 73728, + 73216, + 73728, + 73216, + 75264, + 75776, + 73728, + 74240, + 74752, + 74240, + 74752, + 74240, + 73728, + 83968, + 84992, + 95232, + 94208, + 104448, + 94208, + 95232, + 105472, + 104448, + 94208, + 83968, + 73728, + 75776, + 86016, + 88064, + 89088, + 87040, + 89088, + 87040, + 86016, + 91136, + 86016, + 75776, + 73728, + 76288, + 76800, + 81920, + 82432, + 77312, + 76800, + 142336, + 273408, + 142336, + 76800, + 76288, + 73728, + 8192, + 2105344, + 8192, + 40960, + 2138112, + 2105344, + 4202496, + 2105344, + 8192, + 8396800, + 8192, + 2105344, + 10493952, + 8396800, + 8192, + 8396800, + 8397312, + 8704, + 8192, + 8704, + 8192, + 8396800, + 16785408, + 25174016, + 16785408, + 25174016, + 16785408, + 16785920, + 25174528, + 16785920, + 16785408, + 16785920, + 8397312, + 8397824, + 16786432, + 16785920, + 16786432, + 8397824, + 16786432, + 25175040, + 16786432, + 16786944, + 8398336, + 8398848, + 8398336, + 8398848, + 8398336, + 8398848, + 8398336, + 8397824, + 16786432, + 16786944, + 25175552, + 33564160, + 33563648, + 25175040, + 33563648, + 25175040, + 33563648, + 25175040, + 16786432, + 16786944, + 25175552, + 16786944, + 16786432, + 8397824, + 8397312, + 8396800, + 8192, + 8704, + 8192, + 37756928, + 38051840, + 303104, + 8192, + 8396800, + 21504000, + 97001472, + 97296384, + 101490688, + 25993216, + 25698304, + 21504000, + 8396800, + 8192, + 24576, + 4218880, + 4202496, + 8396800, + 4202496, + 8192, + 16785408, + 8192, + 4202496, + 20979712, + 16785408, + 8192, + 16785408, + 16785920, + 8704, + 8192, + 8704, + 8192, + 16785408, + 33562624, + 50339840, + 33562624, + 50339840, + 33562624, + 33563136, + 50340352, + 33563136, + 33562624, + 33563136, + 16785920, + 16786432, + 33563648, + 33563136, + 33563648, + 16786432, + 33563648, + 50340864, + 33563648, + 33564160, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16786432, + 33563648, + 33564160, + 50341376, + 67118592, + 67118080, + 50340864, + 67118080, + 50340864, + 67118080, + 50340864, + 33563648, + 33564160, + 50341376, + 33564160, + 33563648, + 16786432, + 16785920, + 16785408, + 8192, + 8704, + 8192, + 151003136, + 151150592, + 155648, + 8192, + 16785408, + 38019072, + 189014016, + 189161472, + 205938688, + 54943744, + 54796288, + 38019072, + 16785408, + 8192, + 4202496, + 20979712, + 16785408, + 8192, + 16785408, + 16785920, + 8704, + 8192, + 8704, + 8192, + 16785408, + 33562624, + 50339840, + 33562624, + 50339840, + 33562624, + 33563136, + 50340352, + 33563136, + 33562624, + 33563136, + 16785920, + 16786432, + 33563648, + 33563136, + 33563648, + 16786432, + 33563648, + 50340864, + 33563648, + 33564160, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16786432, + 33563648, + 33564160, + 50341376, + 67118592, + 67118080, + 50340864, + 67118080, + 50340864, + 67118080, + 50340864, + 33563648, + 33564160, + 50341376, + 33564160, + 33563648, + 16786432, + 16785920, + 16785408, + 8192, + 8704, + 8192, + 75505664, + 75579392, + 81920, + 8192, + 16785408, + 38019072, + 189014016, + 189087744, + 197476352, + 46481408, + 46407680, + 38019072, + 16785408, + 8192, + 16384, + 8404992, + 8396800, + 16785408, + 8396800, + 8192, + 33562624, + 8192, + 8396800, + 41951232, + 33562624, + 8192, + 33562624, + 33563136, + 8704, + 8192, + 8704, + 8192, + 33562624, + 67117056, + 100671488, + 67117056, + 100671488, + 67117056, + 67117568, + 100672000, + 67117568, + 67117056, + 67117568, + 33563136, + 33563648, + 67118080, + 67117568, + 67118080, + 33563648, + 67118080, + 100672512, + 67118080, + 67118592, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33563648, + 67118080, + 67118592, + 100673024, + 134227456, + 134226944, + 100672512, + 134226944, + 100672512, + 134226944, + 100672512, + 67118080, + 67118592, + 100673024, + 67118592, + 67118080, + 33563648, + 33563136, + 33562624, + 8192, + 8704, + 8192, + 301998080, + 302034944, + 45056, + 8192, + 33562624, + 71442432, + 373432320, + 373469184, + 407023616, + 105033728, + 104996864, + 71442432, + 33562624, + 8192, + 8396800, + 41951232, + 33562624, + 8192, + 33562624, + 33563136, + 8704, + 8192, + 8704, + 8192, + 33562624, + 67117056, + 100671488, + 67117056, + 100671488, + 67117056, + 67117568, + 100672000, + 67117568, + 67117056, + 67117568, + 33563136, + 33563648, + 67118080, + 67117568, + 67118080, + 33563648, + 67118080, + 100672512, + 67118080, + 67118592, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33563648, + 67118080, + 67118592, + 100673024, + 134227456, + 134226944, + 100672512, + 134226944, + 100672512, + 134226944, + 100672512, + 67118080, + 67118592, + 100673024, + 67118592, + 67118080, + 33563648, + 33563136, + 33562624, + 8192, + 8704, + 8192, + 28319744, + 28323328, + 11776, + 8192, + 11776, + 8192, + 11776, + 15360, + 18944, + 15360, + 11776, + 8192, + 11776, + 15360, + 18944, + 15360, + 18944, + 15360, + 11776, + 8192, + 11776, + 15360, + 18944, + 22528, + 26112, + 22528, + 26112, + 22528, + 18944, + 15360, + 15872, + 16384, + 16896, + 16384, + 15872, + 15360, + 15872, + 16384, + 16896, + 16384, + 16896, + 16384, + 15872, + 15360, + 15872, + 16384, + 16896, + 17408, + 17920, + 17408, + 17920, + 17408, + 16896, + 16384, + 16896, + 17408, + 17920, + 17408, + 16896, + 16384, + 16896, + 17408, + 17920, + 17408, + 17920, + 17408, + 16896, + 16384, + 16896, + 13312, + 13824, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 47104, + 10240, + 47104, + 83968, + 120832, + 83968, + 47104, + 10240, + 47104, + 83968, + 120832, + 83968, + 120832, + 83968, + 47104, + 10240, + 47104, + 46592, + 83456, + 82944, + 119808, + 156672, + 193536, + 156672, + 193536, + 156672, + 119808, + 82944, + 83456, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 82944, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 46592, + 47104, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 83968, + 157696, + 83968, + 157696, + 83968, + 157696, + 231424, + 305152, + 231424, + 157696, + 83968, + 157696, + 231424, + 305152, + 231424, + 305152, + 231424, + 157696, + 83968, + 157696, + 157184, + 230912, + 230400, + 304128, + 377856, + 451584, + 377856, + 451584, + 377856, + 304128, + 230400, + 230912, + 231424, + 231936, + 231424, + 230912, + 230400, + 230912, + 231424, + 231936, + 231424, + 231936, + 231424, + 230912, + 230400, + 230912, + 230400, + 230912, + 230400, + 230912, + 231424, + 231936, + 231424, + 231936, + 231424, + 230912, + 230400, + 156672, + 157184, + 157696, + 158208, + 157696, + 157184, + 156672, + 157184, + 157696, + 158208, + 157696, + 158208, + 157696, + 157184, + 156672, + 157184, + 83456, + 83968, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 157696, + 10240, + 157696, + 305152, + 452608, + 305152, + 157696, + 10240, + 157696, + 305152, + 452608, + 305152, + 452608, + 305152, + 157696, + 10240, + 157696, + 157184, + 304640, + 304128, + 451584, + 599040, + 746496, + 599040, + 746496, + 599040, + 451584, + 304128, + 304640, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 304128, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 157184, + 157696, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 305152, + 10240, + 305152, + 600064, + 894976, + 600064, + 305152, + 10240, + 305152, + 600064, + 894976, + 600064, + 894976, + 600064, + 305152, + 10240, + 305152, + 304640, + 599552, + 599040, + 893952, + 1188864, + 1483776, + 1188864, + 1483776, + 1188864, + 893952, + 599040, + 599552, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 599040, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 304640, + 305152, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 15360, + 10240, + 15360, + 20480, + 25600, + 20480, + 15360, + 10240, + 15360, + 20480, + 25600, + 20480, + 25600, + 20480, + 15360, + 10240, + 15360, + 14848, + 19968, + 19456, + 24576, + 29696, + 34816, + 29696, + 34816, + 29696, + 24576, + 19456, + 19968, + 20480, + 20992, + 20480, + 19968, + 19456, + 19968, + 20480, + 20992, + 20480, + 20992, + 20480, + 19968, + 19456, + 19968, + 19456, + 19968, + 19456, + 19968, + 20480, + 20992, + 20480, + 20992, + 20480, + 19968, + 19456, + 18944, + 18432, + 13312, + 8192, + 3072, + 2560, + 2048, + 1536, + 3147264, + 6292992, + 3147264, + 6292992, + 6293504, + 9439232, + 6293504, + 6292992, + 3147264, + 6698496, + 40252928, + 40256512, + 68568064, + 102122496, + 73810944, + 73807360, + 40252928, + 37107200, + 33555968, + 33556480, + 33556992, + 33557504, + 33556992, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 33555968, + 67110400, + 33555968, + 1536, + 33555968, + 67110400, + 33555968, + 67110400, + 33555968, + 71435776, + 104990208, + 105027072, + 407016960, + 440571392, + 138581504, + 138544640, + 104990208, + 71435776, + 33555968, + 33556480, + 33556992, + 33557504, + 33556992, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 33557504, + 33558016, + 33558528, + 33558016, + 33557504, + 33556992, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 33555968, + 67110400, + 33555968, + 1536, + 33555968, + 67110400, + 33555968, + 67110400, + 33555968, + 1536, + 8390144, + 1536, + 33555968, + 50333184, + 16778752, + 18875904, + 18876416, + 16779264, + 16778752, + 33555968, + 50333184, + 67110400, + 83887616, + 67110400, + 69207552, + 69208064, + 67110912, + 67110400, + 50333184, + 50333696, + 50334208, + 50336256, + 50336768, + 50337280, + 67114496, + 67113984, + 67114496, + 83891712, + 83891200, + 83893248, + 100670464, + 100668416, + 117445632, + 100668416, + 117445632, + 100668416, + 117445632, + 100668416, + 117445632, + 100668416, + 109057024, + 92279808, + 83891200, + 67113984, + 50336768, + 50336256, + 50334208, + 50333696, + 50333184, + 33555968, + 16778752, + 1536, + 83456, + 116224, + 124416, + 157184, + 148992, + 116224, + 124416, + 8513024, + 8504832, + 16893440, + 8504832, + 8472064, + 8390144, + 19006976, + 35784192, + 35857920, + 111355392, + 128132608, + 52635136, + 52561408, + 35784192, + 27395584, + 16778752, + 16779264, + 16779776, + 16780288, + 16779776, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 16778752, + 33555968, + 16778752, + 1536, + 16778752, + 33555968, + 16778752, + 33555968, + 16778752, + 38012416, + 54789632, + 54937088, + 205932032, + 222709248, + 71714304, + 71566848, + 54789632, + 38012416, + 16778752, + 16779264, + 16779776, + 16780288, + 16779776, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 16780288, + 16780800, + 16781312, + 16780800, + 16780288, + 16779776, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 16778752, + 33555968, + 16778752, + 1536, + 16778752, + 33555968, + 16778752, + 33555968, + 16778752, + 1536, + 4195840, + 1536, + 16778752, + 25167360, + 8390144, + 9438720, + 9439232, + 8390656, + 8390144, + 16778752, + 25167360, + 33555968, + 41944576, + 33555968, + 34604544, + 34605056, + 33556480, + 33555968, + 25167360, + 25167872, + 25168384, + 25170432, + 25170944, + 25171456, + 33560064, + 33559552, + 33560064, + 41948672, + 41948160, + 41950208, + 50338816, + 50336768, + 58725376, + 50336768, + 58725376, + 50336768, + 58725376, + 50336768, + 58725376, + 50336768, + 54531072, + 46142464, + 41948160, + 33559552, + 25170944, + 25170432, + 25168384, + 25167872, + 25167360, + 16778752, + 8390144, + 1536, + 83456, + 148992, + 165376, + 230912, + 214528, + 148992, + 165376, + 4359680, + 4343296, + 8537600, + 4343296, + 4277760, + 4195840, + 10749440, + 19138048, + 19432960, + 57181696, + 65570304, + 27821568, + 27526656, + 19138048, + 14943744, + 8390144, + 8390656, + 8391168, + 8391680, + 8391168, + 8391680, + 8391168, + 8391680, + 8392192, + 8392704, + 8392192, + 8391680, + 8391168, + 8391680, + 8392192, + 8392704, + 8392192, + 8391680, + 8391168, + 2560, + 2048, + 1536, + 2048, + 2560, + 2048, + 2560, + 2048, + 1536, + 8390144, + 16778752, + 8390144, + 1536, + 8390144, + 16778752, + 8390144, + 16778752, + 8390144, + 1536, + 2098688, + 1536, + 8390144, + 12584448, + 4195840, + 4720128, + 4720640, + 4196352, + 4195840, + 8390144, + 12584448, + 16778752, + 20973056, + 16778752, + 17303040, + 17303552, + 16779264, + 16778752, + 12584448, + 12584960, + 12585472, + 12587520, + 12588544, + 12589056, + 16783360, + 16782848, + 16783360, + 20977664, + 20977152, + 20979200, + 25173504, + 25171456, + 29365760, + 25171456, + 29365760, + 25171456, + 29365760, + 25171456, + 29365760, + 25171456, + 27268608, + 23074304, + 20977152, + 16782848, + 12588544, + 12587520, + 12585472, + 12584960, + 12584448, + 8390144, + 4195840, + 1536, + 83456, + 214528, + 247296, + 378368, + 345600, + 214528, + 247296, + 2344448, + 2311680, + 4408832, + 2311680, + 214528, + 83456, + 1536, + 132608, + 198144, + 200704, + 201216, + 198656, + 198144, + 67072, + 72192, + 77312, + 72192, + 82432, + 83456, + 93696, + 92672, + 102912, + 92672, + 93696, + 103936, + 102912, + 92672, + 82432, + 72192, + 82432, + 84480, + 86528, + 87552, + 85504, + 95744, + 96768, + 86528, + 87552, + 88576, + 87552, + 88576, + 89600, + 88576, + 87552, + 86528, + 85504, + 83456, + 73216, + 73728, + 74240, + 73728, + 72704, + 146432, + 146944, + 73216, + 73728, + 73216, + 73728, + 73216, + 75264, + 75776, + 73728, + 74240, + 74752, + 74240, + 74752, + 74240, + 73728, + 83968, + 84992, + 95232, + 94208, + 104448, + 94208, + 95232, + 105472, + 104448, + 94208, + 83968, + 73728, + 75776, + 86016, + 88064, + 89088, + 87040, + 89088, + 87040, + 86016, + 91136, + 86016, + 75776, + 73728, + 76288, + 76800, + 81920, + 82432, + 77312, + 76800, + 142336, + 273408, + 142336, + 76800, + 76288, + 73728, + 8192, + 2105344, + 8192, + 40960, + 2138112, + 2105344, + 4202496, + 2105344, + 8192, + 8396800, + 8192, + 2105344, + 10493952, + 8396800, + 8192, + 8396800, + 8397312, + 8704, + 8192, + 8704, + 8192, + 8396800, + 16785408, + 25174016, + 16785408, + 25174016, + 16785408, + 16785920, + 25174528, + 16785920, + 16785408, + 16785920, + 8397312, + 8397824, + 16786432, + 16785920, + 16786432, + 8397824, + 16786432, + 25175040, + 16786432, + 16786944, + 8398336, + 8398848, + 8398336, + 8398848, + 8398336, + 8398848, + 8398336, + 8397824, + 16786432, + 16786944, + 25175552, + 33564160, + 33563648, + 25175040, + 33563648, + 25175040, + 33563648, + 25175040, + 16786432, + 16786944, + 25175552, + 16786944, + 16786432, + 8397824, + 8397312, + 8396800, + 8192, + 8704, + 8192, + 37756928, + 38051840, + 303104, + 8192, + 8396800, + 21504000, + 97001472, + 97296384, + 101490688, + 25993216, + 25698304, + 21504000, + 8396800, + 8192, + 24576, + 4218880, + 4202496, + 8396800, + 4202496, + 8192, + 16785408, + 8192, + 4202496, + 20979712, + 16785408, + 8192, + 16785408, + 16785920, + 8704, + 8192, + 8704, + 8192, + 16785408, + 33562624, + 50339840, + 33562624, + 50339840, + 33562624, + 33563136, + 50340352, + 33563136, + 33562624, + 33563136, + 16785920, + 16786432, + 33563648, + 33563136, + 33563648, + 16786432, + 33563648, + 50340864, + 33563648, + 33564160, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16786432, + 33563648, + 33564160, + 50341376, + 67118592, + 67118080, + 50340864, + 67118080, + 50340864, + 67118080, + 50340864, + 33563648, + 33564160, + 50341376, + 33564160, + 33563648, + 16786432, + 16785920, + 16785408, + 8192, + 8704, + 8192, + 151003136, + 151150592, + 155648, + 8192, + 16785408, + 38019072, + 189014016, + 189161472, + 205938688, + 54943744, + 54796288, + 38019072, + 16785408, + 8192, + 4202496, + 20979712, + 16785408, + 8192, + 16785408, + 16785920, + 8704, + 8192, + 8704, + 8192, + 16785408, + 33562624, + 50339840, + 33562624, + 50339840, + 33562624, + 33563136, + 50340352, + 33563136, + 33562624, + 33563136, + 16785920, + 16786432, + 33563648, + 33563136, + 33563648, + 16786432, + 33563648, + 50340864, + 33563648, + 33564160, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16787456, + 16786944, + 16786432, + 33563648, + 33564160, + 50341376, + 67118592, + 67118080, + 50340864, + 67118080, + 50340864, + 67118080, + 50340864, + 33563648, + 33564160, + 50341376, + 33564160, + 33563648, + 16786432, + 16785920, + 16785408, + 8192, + 8704, + 8192, + 75505664, + 75579392, + 81920, + 8192, + 16785408, + 38019072, + 189014016, + 189087744, + 197476352, + 46481408, + 46407680, + 38019072, + 16785408, + 8192, + 16384, + 8404992, + 8396800, + 16785408, + 8396800, + 8192, + 33562624, + 8192, + 8396800, + 41951232, + 33562624, + 8192, + 33562624, + 33563136, + 8704, + 8192, + 8704, + 8192, + 33562624, + 67117056, + 100671488, + 67117056, + 100671488, + 67117056, + 67117568, + 100672000, + 67117568, + 67117056, + 67117568, + 33563136, + 33563648, + 67118080, + 67117568, + 67118080, + 33563648, + 67118080, + 100672512, + 67118080, + 67118592, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33563648, + 67118080, + 67118592, + 100673024, + 134227456, + 134226944, + 100672512, + 134226944, + 100672512, + 134226944, + 100672512, + 67118080, + 67118592, + 100673024, + 67118592, + 67118080, + 33563648, + 33563136, + 33562624, + 8192, + 8704, + 8192, + 301998080, + 302034944, + 45056, + 8192, + 33562624, + 71442432, + 373432320, + 373469184, + 407023616, + 105033728, + 104996864, + 71442432, + 33562624, + 8192, + 8396800, + 41951232, + 33562624, + 8192, + 33562624, + 33563136, + 8704, + 8192, + 8704, + 8192, + 33562624, + 67117056, + 100671488, + 67117056, + 100671488, + 67117056, + 67117568, + 100672000, + 67117568, + 67117056, + 67117568, + 33563136, + 33563648, + 67118080, + 67117568, + 67118080, + 33563648, + 67118080, + 100672512, + 67118080, + 67118592, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33564672, + 33564160, + 33563648, + 67118080, + 67118592, + 100673024, + 134227456, + 134226944, + 100672512, + 134226944, + 100672512, + 134226944, + 100672512, + 67118080, + 67118592, + 100673024, + 67118592, + 67118080, + 33563648, + 33563136, + 33562624, + 8192, + 8704, + 8192, + 28319744, + 28323328, + 11776, + 8192, + 11776, + 8192, + 11776, + 15360, + 18944, + 15360, + 11776, + 8192, + 11776, + 15360, + 18944, + 15360, + 18944, + 15360, + 11776, + 8192, + 11776, + 15360, + 18944, + 22528, + 26112, + 22528, + 26112, + 22528, + 18944, + 15360, + 15872, + 16384, + 16896, + 16384, + 15872, + 15360, + 15872, + 16384, + 16896, + 16384, + 16896, + 16384, + 15872, + 15360, + 15872, + 16384, + 16896, + 17408, + 17920, + 17408, + 17920, + 17408, + 16896, + 16384, + 16896, + 17408, + 17920, + 17408, + 16896, + 16384, + 16896, + 17408, + 17920, + 17408, + 17920, + 17408, + 16896, + 16384, + 16896, + 13312, + 13824, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 47104, + 10240, + 47104, + 83968, + 120832, + 83968, + 47104, + 10240, + 47104, + 83968, + 120832, + 83968, + 120832, + 83968, + 47104, + 10240, + 47104, + 46592, + 83456, + 82944, + 119808, + 156672, + 193536, + 156672, + 193536, + 156672, + 119808, + 82944, + 83456, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 82944, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 83968, + 84480, + 83968, + 84480, + 83968, + 83456, + 82944, + 83456, + 46592, + 47104, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 83968, + 157696, + 83968, + 157696, + 83968, + 157696, + 231424, + 305152, + 231424, + 157696, + 83968, + 157696, + 231424, + 305152, + 231424, + 305152, + 231424, + 157696, + 83968, + 157696, + 157184, + 230912, + 230400, + 304128, + 377856, + 451584, + 377856, + 451584, + 377856, + 304128, + 230400, + 230912, + 231424, + 231936, + 231424, + 230912, + 230400, + 230912, + 231424, + 231936, + 231424, + 231936, + 231424, + 230912, + 230400, + 230912, + 230400, + 230912, + 230400, + 230912, + 231424, + 231936, + 231424, + 231936, + 231424, + 230912, + 230400, + 156672, + 157184, + 157696, + 158208, + 157696, + 157184, + 156672, + 157184, + 157696, + 158208, + 157696, + 158208, + 157696, + 157184, + 156672, + 157184, + 83456, + 83968, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 157696, + 10240, + 157696, + 305152, + 452608, + 305152, + 157696, + 10240, + 157696, + 305152, + 452608, + 305152, + 452608, + 305152, + 157696, + 10240, + 157696, + 157184, + 304640, + 304128, + 451584, + 599040, + 746496, + 599040, + 746496, + 599040, + 451584, + 304128, + 304640, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 304128, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 305152, + 305664, + 305152, + 305664, + 305152, + 304640, + 304128, + 304640, + 157184, + 157696, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 305152, + 10240, + 305152, + 600064, + 894976, + 600064, + 305152, + 10240, + 305152, + 600064, + 894976, + 600064, + 894976, + 600064, + 305152, + 10240, + 305152, + 304640, + 599552, + 599040, + 893952, + 1188864, + 1483776, + 1188864, + 1483776, + 1188864, + 893952, + 599040, + 599552, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 599040, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 600064, + 600576, + 600064, + 600576, + 600064, + 599552, + 599040, + 599552, + 304640, + 305152, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 10752, + 10240, + 10752, + 10240, + 10752, + 11264, + 11776, + 11264, + 11776, + 11264, + 10752, + 10240, + 15360, + 10240, + 15360, + 20480, + 25600, + 20480, + 15360, + 10240, + 15360, + 20480, + 25600, + 20480, + 25600, + 20480, + 15360, + 10240, + 15360, + 14848, + 19968, + 19456, + 24576, + 29696, + 34816, + 29696, + 34816, + 29696, + 24576, + 19456, + 19968, + 20480, + 20992, + 20480, + 19968, + 19456, + 19968, + 20480, + 20992, + 20480, + 20992, + 20480, + 19968, + 19456, + 19968, + 19456, + 19968, + 19456, + 19968, + 20480, + 20992, + 20480, + 20992, + 20480, + 19968, + 19456, + 18944, + 18432, + 13312, + 8192 + ] + } + ] +} diff --git a/benchmarks/plot.py b/benchmarks/plot.py new file mode 100644 index 0000000..4e4d142 --- /dev/null +++ b/benchmarks/plot.py @@ -0,0 +1,480 @@ +""" +Draw the README's benchmark figures from the results in benchmarks/history. + +Run from the repo root, with matplotlib installed (the package doesn't depend on it): + + python3 benchmarks/plot.py v1.1.0 v1.2.0 + +Writes light and dark PNGs to notebooks/assets/benchmarks: + + benchmark_throughput, benchmark_convergence, benchmark_vram from / + benchmark_release, old against new per GPU from both versions + +Each GPU's release panel has its own scale and compares only with itself, so every panel +is a true A/B on one machine. On the convergence figure, any paths_fused_.json adds +that GPU's fused path as a dashed curve; the title and equal-loss line stay on paths.json. +""" +import argparse +import itertools +import json +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.ticker import FuncFormatter, LogLocator + +HISTORY = Path(__file__).resolve().parent / "history" +ASSETS = Path(__file__).resolve().parents[1] / "notebooks" / "assets" / "benchmarks" + +plt.rcParams["font.family"] = ["DejaVu Sans"] +plt.rcParams["axes.unicode_minus"] = False + +DPI = 200 +FIGSIZE = (8.0, 4.5) +SZ_TITLE, SZ_CAPTION, SZ_VALUE, SZ_UNIT = 14.0, 9.5, 21.0, 10.5 +SZ_AXIS, SZ_TICK, SZ_FOOTER = 9.5, 10.0, 8.5 +MIB = 1024.0 * 1024.0 +TIMES = "×" +SEED = 42 # build_model's manual_seed + +THEMES = { + "light": { + "suffix": "", + "surface": "#fcfcfb", + "legend_bg": "#f4f4f0", + "primary": "#0b0b0b", + "secondary": "#52514e", + "muted": "#898781", + "grid": "#e1e0d9", + "axis": "#c3c2b7", + "raw_alpha": 0.18, + "series": {"cupy-fused": "#2a9d5c", "cupy-fallback": "#2a78d6", "numpy-cpu": "#4a3aa7", + "other-gpu-fused": "#5da938"}, + "stages": {"forward": "#2a9d5c", "backward": "#16643a"}, + }, + "dark": { + "suffix": "-dark", + "surface": "#1a1a19", + "legend_bg": "#222220", + "primary": "#ffffff", + "secondary": "#c3c2b7", + "muted": "#898781", + "grid": "#2c2c2a", + "axis": "#383835", + "raw_alpha": 0.38, + "series": {"cupy-fused": "#3dbb74", "cupy-fallback": "#3987e5", "numpy-cpu": "#9085e9", + "other-gpu-fused": "#81d262"}, + "stages": {"forward": "#3dbb74", "backward": "#9fe8bd"}, + }, +} +STAGE_LABELS = {"forward": "Forward", "backward": "Backward", "rest": "Everything else"} +THROUGHPUT_ORDER = ("numpy-cpu", "cupy-fallback", "cupy-fused") +VRAM_DRAW_ORDER = ("cupy-fallback", "cupy-fused") # the taller area goes down first +MIN_LOSS_DROP = 0.25 # a curve that falls less than this didn't converge +DASHES = (0, (2, 1.2)) # in line widths; the legend handle ends on a dash, not a sliver +PHASE_LABEL_ROW ={"forward": 0, "loss": 1, "backward": 0, "optimizer": 1} + + +def fmt_ms(ms): + return f"{ms:.0f}" if ms >= 100 else f"{ms:.1f}" if ms >= 10 else f"{ms:.2f}" + + +def fmt_time(seconds): + if seconds >= 60: + return f"{seconds / 60:.1f} min" + return f"{seconds:.1f} s" if seconds >= 1 else f"{seconds:.2g} s" + + +def fmt_multiple(ratio): + return f"{ratio:.0f}{TIMES}" if ratio >= 10 else f"{ratio:.1f}{TIMES}" + + +def fmt_mib(num_bytes): + mib = num_bytes / MIB + return f"{mib:,.0f}" if mib >= 100 else f"{mib:,.1f}" + + +def new_figure(theme, title, caption, footer, height=FIGSIZE[1]): + """A figure with the title block and footer at the same size on any height.""" + fig = plt.figure(figsize=(FIGSIZE[0], height), dpi=DPI) + fig.patch.set_facecolor(theme["surface"]) + fig.text(0.042, 1 - 0.2025 / height, title, fontsize=SZ_TITLE, color=theme["primary"], + fontweight="bold", va="top") + fig.text(0.042, 1 - 0.5535 / height, caption, fontsize=SZ_CAPTION, color=theme["secondary"], va="top") + fig.text(0.5, 0.054 / height, footer, fontsize=SZ_FOOTER, color=theme["muted"], va="bottom", + ha="center") + return fig + + +def new_axes(fig, rect, theme): + ax = fig.add_axes(rect) + ax.set_facecolor(theme["surface"]) + ax.spines[["top", "right"]].set_visible(False) + ax.spines[["left", "bottom"]].set_color(theme["axis"]) + ax.tick_params(colors=theme["secondary"], labelsize=SZ_TICK, length=0) + ax.set_axisbelow(True) + return ax + + +def footer_text(meta, extra=""): + return (f"{meta['gpu']} · CuPy {meta['cupy']} · CIFAR-10 · " + f"batch {meta['batch_size']} · seed {SEED} · {meta['commit'][:7]}{extra}") + + +# --- Throughput --------------------------------------------------------------------- + +def figure_throughput(paths, theme): + results = {r["key"]: r for r in paths["results"]} + entries = [results[key] for key in THROUGHPUT_ORDER] + fused_ms = results["cupy-fused"]["ms_per_step"]["median"] + + def delta(entry): + if entry["key"] == "cupy-fused": + return "baseline" + return f"{fmt_multiple(entry['ms_per_step']['median'] / fused_ms)} slower" + + gain = results["cupy-fallback"]["ms_per_step"]["median"] / fused_ms + fig = new_figure( + theme, + f"Fused kernels deliver {fmt_multiple(gain)} higher throughput on GPU", + "Same CNN, same seed, same fp16 policy. Conv2d can only honor fp16 on the fused path.", + footer_text(paths["meta"], f" · median of {results['cupy-fused']['repeats']} GPU runs"), + ) + + tiles = fig.add_axes([0.042, 0.585, 0.935, 0.215]) + tiles.set_xlim(0, len(entries)) + tiles.set_ylim(0, 1) + tiles.axis("off") + fig.canvas.draw() + for i, entry in enumerate(entries): + color = theme["series"][entry["key"]] + tiles.plot([i + 0.035], [0.845], marker="o", markersize=8.5, color=color, clip_on=False) + tiles.text(i + 0.105, 0.845, entry["label"], fontsize=SZ_CAPTION, + color=theme["secondary"], va="center") + value = tiles.text(i, 0.38, fmt_ms(entry["ms_per_step"]["median"]), fontsize=SZ_VALUE, + color=theme["primary"], fontweight="bold") + right = value.get_window_extent(fig.canvas.get_renderer()).x1 + tiles.text(tiles.transData.inverted().transform((right, 0))[0] + 0.024, 0.38, "ms", + fontsize=SZ_UNIT, color=theme["secondary"]) + tiles.text(i, 0.09, delta(entry), fontsize=SZ_CAPTION, color=theme["secondary"]) + fig.add_artist(plt.Line2D([0.042, 0.977], [0.556, 0.556], color=theme["grid"], linewidth=1.0)) + + ax = new_axes(fig, [0.248, 0.190, 0.729, 0.318], theme) + ax.set_xscale("log") + ax.grid(axis="x", color=theme["grid"], linewidth=1.0) + rows = np.arange(len(entries))[::-1] + for row, entry in zip(rows, entries): + ms = entry["ms_per_step"]["median"] + color = theme["series"][entry["key"]] + ax.plot([fused_ms, ms], [row, row], color=color, linewidth=2.0, solid_capstyle="round") + ax.plot([ms], [row], marker="o", markersize=11, color=color, + markeredgecolor=theme["surface"], markeredgewidth=2.0) + ax.annotate(delta(entry), xy=(ms, row), xytext=(12, 0), textcoords="offset points", + va="center", fontsize=SZ_TICK, color=theme["primary"]) + ax.set_yticks(rows, [entry["label"] for entry in entries], color=theme["primary"]) + ax.set_ylim(-0.72, len(entries) - 0.28) + ax.set_xlim(fused_ms / 1.7, max(e["ms_per_step"]["median"] for e in entries) * 4.2) + ax.xaxis.set_major_locator(LogLocator(base=10.0)) + ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g}")) + ax.set_xlabel("milliseconds per training step (log scale, lower is better)", + fontsize=SZ_AXIS, color=theme["secondary"], labelpad=9) + return fig + + +# --- Convergence -------------------------------------------------------------------- + +def smooth(values): + window = max(2, min(len(values) // 12, 25)) + padded = np.pad(values, (window // 2, window - 1 - window // 2), mode="edge") + return np.convolve(padded, np.ones(window) / window, mode="valid") + + +def equal_loss_threshold(traces): + """A round loss every path crosses, about a third of the way down their shared range.""" + converging = [trace for trace in traces if not is_flat(trace)] + floor = max(trace["trend"][-1] for trace in converging) + ceiling = min(trace["trend"][0] for trace in converging) + if ceiling <= floor: + raise ValueError(f"[aether] Loss curves share no range: {ceiling} <= {floor}.") + return round((floor + (ceiling - floor) * 0.35) * 20) / 20 + + +def crossing_time(trace, threshold): + return trace["t"][np.nonzero(trace["trend"] <= threshold)[0][0]] + + +def is_flat(trace): + return trace["trend"][0] - trace["trend"][-1] < MIN_LOSS_DROP + + +def read_trace(result, label, linestyle="-"): + loss = np.asarray(result["loss"]) + return {"key": result["key"], "label": label, "linestyle": linestyle, "loss": loss, + "trend": smooth(loss), "t": np.asarray(result["elapsed_s"])} + + +def figure_convergence(paths, other_gpus, theme): + results = {r["key"]: r for r in paths["results"]} + main_traces = [read_trace(results[key], results[key]["label"]) for key in THROUGHPUT_ORDER] + threshold = equal_loss_threshold(main_traces) + main_times = [crossing_time(tr, threshold) for tr in main_traces if not is_flat(tr)] + fastest, slowest = min(main_times), max(main_times) + + # Another GPU's fused path gets its own lime shade so it doesn't blur into the AMD green. + other_traces = [{**read_trace(other["results"][0], f"{other['meta']['gpu']}, fused", DASHES), + "key": "other-gpu-fused"} for other in other_gpus] + traces = main_traces + other_traces + # Another GPU that never reaches the line still gets its curve, just no marker. + crossings = sorted(((crossing_time(tr, threshold), tr) for tr in traces + if not is_flat(tr) and tr["trend"].min() <= threshold), + key=lambda item: item[0]) + + fig = new_figure( + theme, + f"Same loss reached {fmt_multiple(slowest / fastest)} sooner on fused kernels", + f"One epoch, {len(traces[0]['t'])} steps, same batch order and seed. Elapsed time " + "with a sync every step, log scale.", + footer_text(paths["meta"]), + ) + ax = new_axes(fig, [0.092, 0.205, 0.865, 0.615], theme) + ax.set_xscale("log") + ax.grid(color=theme["grid"], linewidth=1.0) + + handles = [] + for trace in traces: + color = theme["series"][trace["key"]] + ax.plot(trace["t"], trace["loss"], color=color, linewidth=1.0, alpha=theme["raw_alpha"]) + handles.append(ax.plot(trace["t"], trace["trend"], color=color, linewidth=2.0, + linestyle=trace["linestyle"], label=f"{trace['label']} ({fmt_time(trace['t'][-1])})")[0]) + ax.plot([trace["t"][-1]], [trace["trend"][-1]], marker="o", markersize=7.5, color=color, + markeredgecolor=theme["surface"], markeredgewidth=2.0, zorder=5) + ax.set_ylabel("training loss", fontsize=SZ_AXIS, color=theme["secondary"], labelpad=6) + ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g} s")) + ax.set_xlim(min(tr["t"][0] for tr in traces) / 1.25, max(tr["t"][-1] for tr in traces) * 1.8) + low, high = ax.get_ylim() + # Headroom for the legend, which grows by a row per extra GPU, so it stays off the curves. + ax.set_ylim(low - (high - low) * 0.14, high + (high - low) * (0.18 + 0.14 * len(other_traces))) + low, high = ax.get_ylim() + + for trace in filter(is_flat, traces): + ax.annotate("loss flat - did not converge", xy=(trace["t"][0], trace["trend"].mean()), + xytext=(-9, 0), textcoords="offset points", fontsize=SZ_AXIS, + color=theme["muted"], va="center", ha="right") + + ax.axhline(threshold, color=theme["axis"], linewidth=1.0, zorder=1) + ax.annotate(f"loss {threshold:g}", xy=(0.006, threshold), xycoords=("axes fraction", "data"), + xytext=(0, 6), textcoords="offset points", fontsize=SZ_AXIS, + color=theme["secondary"]) + previous, tier = None, 0 + for t, trace in crossings: + # Labels too close on the log axis to share a row step down onto the next. + tier = (tier + 1) % 3 if previous and np.log10(t / previous) < 0.72 else 0 + previous = t + label_y = low + (high - low) * (0.075 + 0.065 * tier) + color = theme["series"][trace["key"]] + ax.plot([t, t], [label_y, threshold], color=color, linewidth=1.0, alpha=0.55) + ax.plot([t], [threshold], marker="o", markersize=7, color=color, zorder=6, + markeredgecolor=theme["surface"], markeredgewidth=2.0) + ax.annotate(fmt_time(t), xy=(t, label_y), xytext=(0, -3), textcoords="offset points", + fontsize=SZ_AXIS, color=theme["primary"], va="top", ha="center") + + legend = ax.legend(handles=handles[::-1], loc="upper right", framealpha=0.85, + facecolor=theme["legend_bg"], edgecolor=theme["axis"], fontsize=9.0, + handlelength=1.2, labelcolor=theme["primary"]) + legend.set_zorder(7) + fig.text(0.5, 0.055, + f"Equal loss ({threshold:g}) reached in {fmt_time(fastest)} vs {fmt_time(slowest)} " + "elapsed", + fontsize=SZ_CAPTION, color=theme["primary"], va="bottom", ha="center") + return fig + + +# --- Transient VRAM ----------------------------------------------------------------- + +def figure_vram(vram, theme): + results = {r["key"]: r for r in vram["results"]} + peaks = {key: r["peak_transient_bytes"] for key, r in results.items()} + floors = " · ".join(f"{key.split('-')[1]} {fmt_mib(r['baseline_bytes'])} MiB" + for key, r in results.items()) + gain = peaks["cupy-fallback"] / peaks["cupy-fused"] + fig = new_figure( + theme, + f"Fused kernels hold {fmt_multiple(gain)} less transient VRAM", + "Same CNN, same batches, same fp16 policy. Conv2d can only honor fp16 on the fused path.", + footer_text(vram["meta"]), + ) + fig.text(0.5, 0.092, + "x-axis is position within the training step, not elapsed time - " + "phase widths are fixed layout, not durations.", + fontsize=SZ_CAPTION, color=theme["secondary"], va="bottom", ha="center") + fig.text(0.5, 0.048, + f"Transient only, above a resident floor of {floors} " + "(parameters, AdamW state, batches)", + fontsize=SZ_FOOTER, color=theme["muted"], va="bottom", ha="center") + + steps = vram["meta"]["recorded_steps"] + ax = new_axes(fig, [0.092, 0.205, 0.865, 0.615], theme) + ax.grid(axis="y", color=theme["grid"], linewidth=1.0) + for key in VRAM_DRAW_ORDER: + result = results[key] + color = theme["series"][key] + # A step function: the last value holds until the end of the last step. + x = [*result["x"], float(steps)] + y = [*result["transient_bytes"], result["transient_bytes"][-1]] + ax.fill_between(x, 0.0, y, step="post", color=color, alpha=0.28, linewidth=0) + ax.plot(x, y, color=color, linewidth=1.4, drawstyle="steps-post") + + # Zero-based, so the two filled areas stand in the ratio the title states. + top = max(peaks.values()) * 1.34 + ax.set_xlim(0.0, steps) + ax.set_ylim(0.0, top) + tick = next(s for s in (5, 10, 20, 25, 50, 100, 200, 250, 500, 1000) + if top / (s * MIB) <= 6.5) * MIB + ax.set_yticks(np.arange(0.0, top, tick)) + ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v / MIB:,.0f}")) + ax.set_ylabel("transient VRAM (MiB)", fontsize=SZ_AXIS, color=theme["secondary"], labelpad=6) + ax.set_xticks([step + 0.5 for step in range(steps)], + [f"step {step + 1}" for step in range(steps)], color=theme["secondary"]) + + position = 0.0 + for name, width in itertools.chain.from_iterable( + itertools.repeat(tuple(vram["phase_widths"].items()), steps)): + ax.annotate(name, xy=(position + width / 2, 0.985 - 0.062 * PHASE_LABEL_ROW[name]), + xycoords=("data", "axes fraction"), fontsize=SZ_FOOTER, + color=theme["secondary"], va="top", ha="center") + position += width + if position < steps: + step_seam = abs(position - round(position)) < 1e-6 + ax.axvline(position, color=theme["axis"], linewidth=1.2 if step_seam else 1.0, + linestyle=(0, (2, 2)) if step_seam else (0, (1, 2)), zorder=1) + + for key in VRAM_DRAW_ORDER: + result = results[key] + x, peak = result["peak_x"], result["peak_transient_bytes"] + color = theme["series"][key] + ax.plot([x, x], [0.0, peak], color=color, linewidth=1.0, alpha=0.55) + ax.plot([x], [peak], marker="o", markersize=7, color=color, zorder=6, + markeredgecolor=theme["surface"], markeredgewidth=2.0) + flip = x > steps * 0.55 + ax.annotate(f"{result['label']} \u00b7 {fmt_mib(peak)} MiB peak", xy=(x, peak), xytext=(-9 if flip else 9, 7), + textcoords="offset points", fontsize=SZ_AXIS, color=theme["primary"], + ha="right" if flip else "left", zorder=7, + bbox={"boxstyle": "round,pad=0.28", "facecolor": theme["surface"], + "edgecolor": "none", "alpha": 0.82}) + return fig + + +# --- Release comparison ------------------------------------------------------------- + +def stage_ms(breakdown): + """GPU ms per step, grouped: forward, backward (layers only) and everything else.""" + stages = dict.fromkeys(STAGE_LABELS, 0.0) + for row in breakdown["rows"]: + stage = row["stage"] if row["stage"] in ("forward", "backward") else "rest" + stages[stage] += row["gpu_ms"]["median"] + return stages + + +def release_panel(ax, old_version, new_version, old, new, gain, theme): + # The remainder is small and carries no story, so it stays neutral. + colors = {**theme["stages"], "rest": theme["muted"]} + ax.grid(axis="x", color=theme["grid"], linewidth=1.0) + ax.spines["left"].set_visible(False) + ax.set_title(new["meta"]["gpu"], loc="left", fontsize=SZ_CAPTION, fontweight="bold", + color=theme["primary"], pad=5) + for row, breakdown in ((1, old), (0, new)): + left = 0.0 + for stage, ms in stage_ms(breakdown).items(): + ax.barh(row, ms, left=left, height=0.42, color=colors[stage], + edgecolor=theme["surface"], linewidth=1.0, label=STAGE_LABELS[stage]) + left += ms + label = f"{fmt_ms(breakdown['baseline_ms']['median'])} ms/step" + if row == 0: + label += f" · {fmt_multiple(gain)} faster" + ax.annotate(label, xy=(left, row), xytext=(7, 0), textcoords="offset points", + va="center", fontsize=SZ_AXIS, color=theme["primary"], + fontweight="bold" if row == 0 else "normal") + ax.set_yticks([1, 0], [old_version, new_version], color=theme["primary"]) + ax.set_ylim(-0.5, 1.5) + ax.set_xlim(0, sum(stage_ms(old).values()) * 1.2) + ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g} ms")) + + +def figure_release(old_version, new_version, runs, theme): + gains = [old["baseline_ms"]["median"] / new["baseline_ms"]["median"] for old, new in runs] + headline = (f"{min(gains):.1f}–{max(gains):.1f}{TIMES}" if len(gains) > 1 + else fmt_multiple(gains[0])) + first_old, first_new = runs[0] + + # Everything is laid out in inches, so the panels keep their size as GPUs are added. + panel, gap, top, bottom = 0.85, 0.62, 1.32, 0.60 + height = top + bottom + panel * len(runs) + gap * (len(runs) - 1) + fig = new_figure( + theme, + f"{new_version} trains {headline} faster than {old_version}", + "Same CNN and fp16 policy. Segments are GPU time per stage, totals are ms/step." + + (" Each GPU has its own scale." if len(runs) > 1 else ""), + f"CIFAR-10 · batch {first_new['meta']['batch_size']} · seed {SEED} · " + f"median of {first_new['meta']['repeats']} runs · " + f"{first_old['meta']['commit'][:7]} vs {first_new['meta']['commit'][:7]}", + height=height, + ) + for i, ((old, new), gain) in enumerate(zip(runs, gains)): + y = bottom + (panel + gap) * (len(runs) - 1 - i) + ax = new_axes(fig, [0.10, y / height, 0.86, panel / height], theme) + release_panel(ax, old_version, new_version, old, new, gain, theme) + + handles, labels = ax.get_legend_handles_labels() + fig.legend(handles[:3], labels[:3], loc="upper left", bbox_to_anchor=(0.03, 1 - 0.76 / height), + ncols=3, frameon=False, fontsize=SZ_FOOTER, handlelength=1.0, + labelcolor=theme["secondary"]) + return fig + + +def load_release_runs(old_version, new_version): + """(old, new) breakdowns for every GPU, fastest first. All runs must share one pair + of commits, or the footer and the title would be describing different code.""" + runs = [] + for new_path in sorted((HISTORY / new_version).glob("breakdown_*.json")): + old_path = HISTORY / old_version / new_path.name + if not old_path.exists(): + raise FileNotFoundError(f"[aether] {old_path} is missing; {new_path} has no baseline.") + runs.append((json.loads(old_path.read_text()), json.loads(new_path.read_text()))) + if not runs: + raise FileNotFoundError(f"[aether] No breakdown_*.json in {HISTORY / new_version}.") + commits = {(old["meta"]["commit"], new["meta"]["commit"]) for old, new in runs} + if len(commits) > 1: + raise ValueError(f"[aether] Release runs measured different commits: {sorted(commits)}.") + return sorted(runs, key=lambda run: run[1]["baseline_ms"]["median"]) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[1]) + parser.add_argument("old_version", help="folder in benchmarks/history, e.g. v1.1.0") + parser.add_argument("new_version", help="folder in benchmarks/history, e.g. v1.2.0") + args = parser.parse_args() + + results = HISTORY / args.new_version + paths = json.loads((results / "paths.json").read_text()) + vram = json.loads((results / "vram.json").read_text()) + runs = load_release_runs(args.old_version, args.new_version) + other_gpus = [json.loads(path.read_text()) + for path in sorted(results.glob("paths_fused_*.json"))] + + for theme in THEMES.values(): + figures = { + "benchmark_throughput": figure_throughput(paths, theme), + "benchmark_convergence": figure_convergence(paths, other_gpus, theme), + "benchmark_vram": figure_vram(vram, theme), + "benchmark_release": figure_release(args.old_version, args.new_version, runs, theme), + } + for name, fig in figures.items(): + fig.savefig(ASSETS / f"{name}{theme['suffix']}.png", facecolor=theme["surface"]) + plt.close(fig) + + +if __name__ == "__main__": + main() diff --git a/notebooks/assets/benchmark_convergence-dark.png b/notebooks/assets/benchmark_convergence-dark.png deleted file mode 100644 index 06b136e..0000000 Binary files a/notebooks/assets/benchmark_convergence-dark.png and /dev/null differ diff --git a/notebooks/assets/benchmark_convergence.png b/notebooks/assets/benchmark_convergence.png deleted file mode 100644 index 428124b..0000000 Binary files a/notebooks/assets/benchmark_convergence.png and /dev/null differ diff --git a/notebooks/assets/benchmark_throughput-dark.png b/notebooks/assets/benchmark_throughput-dark.png deleted file mode 100644 index c1c292a..0000000 Binary files a/notebooks/assets/benchmark_throughput-dark.png and /dev/null differ diff --git a/notebooks/assets/benchmark_throughput.png b/notebooks/assets/benchmark_throughput.png deleted file mode 100644 index 91e8f0d..0000000 Binary files a/notebooks/assets/benchmark_throughput.png and /dev/null differ diff --git a/notebooks/assets/benchmark_vram-dark.png b/notebooks/assets/benchmark_vram-dark.png deleted file mode 100644 index 68c7203..0000000 Binary files a/notebooks/assets/benchmark_vram-dark.png and /dev/null differ diff --git a/notebooks/assets/benchmark_vram.png b/notebooks/assets/benchmark_vram.png deleted file mode 100644 index 7948aee..0000000 Binary files a/notebooks/assets/benchmark_vram.png and /dev/null differ diff --git a/notebooks/assets/benchmarks/benchmark_convergence-dark.png b/notebooks/assets/benchmarks/benchmark_convergence-dark.png new file mode 100644 index 0000000..dd1dceb Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_convergence-dark.png differ diff --git a/notebooks/assets/benchmarks/benchmark_convergence.png b/notebooks/assets/benchmarks/benchmark_convergence.png new file mode 100644 index 0000000..21dce0e Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_convergence.png differ diff --git a/notebooks/assets/benchmarks/benchmark_release-dark.png b/notebooks/assets/benchmarks/benchmark_release-dark.png new file mode 100644 index 0000000..ef833d1 Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_release-dark.png differ diff --git a/notebooks/assets/benchmarks/benchmark_release.png b/notebooks/assets/benchmarks/benchmark_release.png new file mode 100644 index 0000000..e4dce37 Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_release.png differ diff --git a/notebooks/assets/benchmarks/benchmark_throughput-dark.png b/notebooks/assets/benchmarks/benchmark_throughput-dark.png new file mode 100644 index 0000000..e3cad14 Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_throughput-dark.png differ diff --git a/notebooks/assets/benchmarks/benchmark_throughput.png b/notebooks/assets/benchmarks/benchmark_throughput.png new file mode 100644 index 0000000..aa6d539 Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_throughput.png differ diff --git a/notebooks/assets/benchmarks/benchmark_vram-dark.png b/notebooks/assets/benchmarks/benchmark_vram-dark.png new file mode 100644 index 0000000..f0572db Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_vram-dark.png differ diff --git a/notebooks/assets/benchmarks/benchmark_vram.png b/notebooks/assets/benchmarks/benchmark_vram.png new file mode 100644 index 0000000..567dff8 Binary files /dev/null and b/notebooks/assets/benchmarks/benchmark_vram.png differ