An instrumented baseline, then an optimisation ladder where every rung changes exactly one thing and the order is chosen by the profile rather than by a blog post. Negative results stay in the table.
No GPU here, so the study runs DDP over gloo across real OS processes on CPU. Genuinely distributed; labelled CPU/gloo rather than dressed up as a GPU result.
Weak scaling (per-worker batch fixed - "more data in the same time"):
| world | global batch | lr | samples/s (mean ± std) | speedup | efficiency |
|---|---|---|---|---|---|
| 1 | 32 | 0.0100 | 43.6 ± 1.6 | 1.00x | 100% |
| 2 | 64 | 0.0200 | 69.8 ± 1.7 | 1.60x | 80% |
| 4 | 128 | 0.0400 | 103.7 ± 13.9 | 2.38x | 59% |
Strong scaling (global batch fixed - "the same job, faster"):
| world | per-worker batch | lr | samples/s (mean ± std) | speedup | efficiency |
|---|---|---|---|---|---|
| 1 | 128 | 0.0100 | 51.2 ± 4.0 | 1.00x | 100% |
| 2 | 64 | 0.0100 | 71.8 ± 8.3 | 1.40x | 70% |
| 4 | 32 | 0.0100 | 101.0 ± 6.4 | 1.97x | 49% |
Strong is worse at every size, as it should be: the per-worker batch shrinks, so each worker does less compute per synchronisation and fixed per-step overhead is amortised over less work.
The LR was scaling against the per-worker batch - right by accident under
weak scaling, where they coincide. Under strong scaling it printed
lr 0.01 → 0.02 → 0.04 while the global batch never moved from 128. That is a
hyperparameter change wearing a scaling study's clothes. Fixed to reference a
fixed base batch; strong now holds lr = 0.01 throughout.
An earlier version of this README stated "16.7% is measured communication"
and attributed the scaling gap to it. That number was noise and the claim was
wrong. Repeating the paired no_sync comparison seven times per worker shows
why:
| world | comm fraction (median) | spread across 7 trials |
|---|---|---|
| 2 | 6.3% | ±20.7 points |
| 4 | 2.3% | ±22.3 points |
The spread is three to ten times the median; consecutive runs of the same config gave 15.2% and 1.2%. On a contended CPU this estimator is a coin flip. Communication cost is unmeasured on this hardware, and the median is now reported with its spread so a reader can see it is unusable.
With communication unmeasurable, the remaining explanation was CPU contention
from the in-process decode. The ablation is one command - rerun world 4 with
--decode-cost 0:
| configuration | efficiency at world 4 |
|---|---|
| decode_cost = 60 (normal) | 59% ± 13.9 |
| decode_cost = 0 (ablation) | 65% ± 0.9 |
Removing all CPU-side data work recovers ~6 points, and the two overlap within the baseline's spread. Decode contention is not the dominant term either.
Both candidate explanations have been tested and neither accounts for the ~35-41% loss. What remains untested is core oversubscription (4 workers plus gloo threads on 4 physical cores) and DDP's gradient-bucketing overhead.
Naming an unexplained gap is more useful than assigning it to whichever cause was measured last. Full write-up in docs/SCALING.md.
python -m trainlab.profile_trace writes Chrome traces to results/traces/,
loadable in chrome://tracing or Perfetto. A screenshot is a claim about a
trace; the trace is the evidence, so the trace is what is committed.
| config | data_wait | forward | backward | optimizer |
|---|---|---|---|---|
| baseline | 25.2% | 33.2% | 38.7% | 2.9% |
| +prefetch | 0.4% | 47.7% | 48.3% | 3.6% |
Data-wait collapses from a quarter of step time to essentially nothing, which is the entire argument for attacking the dataloader before touching precision or layout.
You do not get to optimise anything you have not measured.
Enforced in code, not by discipline:
_assert_single_changefails the run if a rung changes more than one knob (batch size and its pre-declared LR rule count as one change, because the LR follows from the batch).- Warmup steps are excluded from every measurement, and how many were excluded is recorded in the row.
- Every row records its device, timer kind, and torch version, so rows measured on different hardware can never be silently merged into one table.
- Rungs that cannot run on the current device are recorded as skipped with a reason; rungs that crash are recorded as failed with the error. A missing row and a deliberately omitted row look identical in a table, so neither is allowed to be missing.
pip install -r requirements.txt
make test # 12 tests, all about the instrument
make ladder # run every rung, 3 repeats each
make report # regenerate REPORT.md + figures from the ledgerOn a GPU box the same command runs the CUDA-only rungs (pin_memory, tf32)
that are skipped on CPU, and the timer switches from perf_counter to
torch.cuda.Event automatically - wall-clock timing around an async kernel
launch measures how fast Python queued the work, not how long the GPU took.
The instrumented FP32 baseline spends ~37% of step time in data wait. That single number set the entire rung order: while the device is idle a third of the time waiting for input, precision and layout optimisations cannot help, because kernels are not the constraint.
The obvious first move - mixed precision - would have been the wrong one here, and the profile said so before any time was spent on it. That is the whole argument for the word order in "profiling-driven distributed training".
Measured results, the generated figures, and the separability analysis are in
REPORT.md, regenerated from results/ladder_cpu.json by
python -m trainlab.report. Nothing in that file is typed by hand.
SyntheticImages carries a tunable per-sample CPU cost that does real numpy
work, not sleep. That matters: a sleep is trivially hidden by any number of
workers and would make the dataloader rung look better than it is, whereas real
work competes for cores and the GIL the way JPEG decode and augmentation actually
do. The cost is set so the baseline reproduces the most common real training
bottleneck (CPU input pipeline starving the accelerator) on any machine, without
downloading ImageNet.
The trade this makes is stated plainly: these are not ImageNet numbers, the model is deliberately small, and the absolute throughput means nothing outside this workload. What transfers is the method and the rung order, not the values.
amp_dtype defaults to bf16. bf16 keeps fp32's exponent range and loses
mantissa bits; fp16 keeps mantissa and loses range, which is why fp16 needs a
GradScaler to stop small gradients flushing to zero. The code enables the
scaler only for fp16 (scaler = GradScaler(enabled=amp and dtype is fp16)) - running a scaler with bf16 is a common cargo-cult that costs a little time and
buys nothing.
What breaks if this choice is wrong: on hardware without native bf16 (pre-Ampere NVIDIA, most CPUs without AMX), bf16 autocast is emulated and is slower than fp32 - which is exactly what the measured ladder on this CPU shows, and why that rung is a negative result in the table rather than a deleted row.
The direct compute cost of this study was $0: it ran on local hardware.
python -m trainlab.cost --ledger results/ladder_cpu.json --rate 0.35 turns any
ledger into a cost-to-target table and a break-even for scaling, but the rate is
a required input rather than a hard-coded constant, because published on-demand
prices are not what anyone's bill actually says.
Everything above this line is CPU. A rented NVIDIA L4 (g6.2xlarge spot, us-east-1, ~$0.92/hr) ran the same ladder on real hardware for the first time - and the three rungs this machine could never execute all ran.
| rung | samples/s | ± | data-wait | vs prev | separable? |
|---|---|---|---|---|---|
| baseline | 789.8 | 0.7 | 95% | - | - |
| workers | 3146.4 | 71.2 | 79% | +298% | yes |
| workers+persistent | 3003.6 | 23.7 | 80% | −4.5% | yes |
| +prefetch | 3072.8 | 20.4 | 79% | +2.3% | yes |
+pin_memory |
3121.0 | 82.2 | 81% | +1.6% | no |
+tf32 |
3174.7 | 83.6 | 81% | +1.7% | no |
| +amp (bf16) | 3682.7 | 655.2 | 79% | +16.0% | no |
| +channels_last | 3527.6 | 771.0 | 79% | −4.2% | no |
| +batch128 | 3644.1 | 666.5 | 88% | +3.3% | no |
+compile |
3257.8 | 672.4 | 90% | −10.6% | no |
Zero rungs skipped, zero failed. +pin_memory and +tf32 had been marked
"CUDA-only, skipped" since the repo was written; +compile had been a failed row
carrying an InductorError about a missing MSVC toolchain. On Linux with gcc it
compiles fine.
workers alone is +298%. Everything after it sums to about +17%, and only
three of those eight rungs clear their own spread. The cumulative 4.66× is really
3.98× from a single change and noise thereafter.
That is the finding worth the rental. On CPU the dataloader was one bottleneck among several. On the L4 compute got roughly four times faster and the dataloader did not, so the GPU is starved for the entire ladder - 95% at baseline, still 79-90% after every kernel-level optimisation.
Which is why +tf32, +amp, +channels_last and +compile land inside the
noise: they make the GPU faster at a job it is already waiting to be given. The
ladder's ordering thesis - profile first, fix the stall, then touch kernels - holds harder on a GPU than it did on the CPU that motivated it.
On CPU, +amp was the ladder's headline negative: 3.3× slower. On the L4 it
is +16% - the largest single post-workers gain.
It is also not separable. The spread is ±655 samples/s, 18% of the measurement, so a +16% move does not clear it. The honest statement is that the sign reversed exactly as hardware predicted and the magnitude remains unestablished at three repeats. Reporting "bf16 gives +16% on GPU" from this data would be repeating the CPU mistake with better hardware.
−10.6%, with Not enough SMs to use max_autotune_gemm mode - the L4 has too few
SMs for Inductor's autotuning path. Another negative result, and one that could
not previously be had: the row was blocked on a missing compiler, so "does
compile help here" had never been a question with an answer.
Every number in the optimisation ladder is samples per second. That is right for "is the training loop efficient?" and wrong for the question anyone pays for, which is "what does it cost to get a model this good?"
cost to target = samples_to_target / samples_per_second x rate
Both factors. The ladder measures only the denominator, so it is structurally blind to any change that moves throughput and convergence in opposite directions - and so is every "we got a 2x speedup" measured the same way.
make accuracy trains each configuration until held-out accuracy crosses 85% on
two consecutive evaluations, at $0.35/hour:
| config | samples/s | steps to target | samples to target | $ to target | rank by throughput | rank by cost |
|---|---|---|---|---|---|---|
| batch64 | 106.0 | 187.5 | 12,000 | $0.0110 | 1 | 1 |
| batch256_linear | 83.0 | 137.5 | 35,200 | $0.0412 | 2 | 4 |
| batch128_linear | 71.2 | 162.5 | 20,800 | $0.0284 | 3 | 3 |
| batch128_unscaled | 68.2 | 137.5 | 17,600 | $0.0251 | 4 | 2 |
| batch128_bf16 | 31.7 | 162.5 | 20,800 | $0.0639 | 5 | 5 |
The winners agree and the orderings do not. batch256_linear is the
second-fastest configuration and the second-most-expensive route to the target - a throughput ladder promotes it while it costs 3.75x the cheapest path to the
same accuracy. Across the table the spread is 5.8x, and none of it is visible
in samples/s.
The mechanism is clean and it is exactly the textbook one: bigger batches did cut the number of updates needed (187.5 → 137.5 steps) and raised the number of samples 2.9x doing it. On CPU the throughput gain never arrived to compensate.
The first version compared only rank_by_throughput[0] against
rank_by_cost[0], found them equal, and printed "the throughput winner is also
the cheapest here". True, and it buried the finding: a ladder does not just pick
a winner, it ranks every rung, and the result lives entirely in the rungs that
move. The comparison now covers full orderings and names the configuration
throughput over-rates - the one a ladder would promote and a budget would not.
The rented L4 also re-ran the cost-to-accuracy study - at the actual price paid, $0.9221/hr - and it is the sharpest version of this repo's argument.
Same target (0.85), same step cap, same repeats. Only the device changed.
| config | samples/s | rank by throughput | samples to target | $ to target | rank by cost |
|---|---|---|---|---|---|
| batch64 | 8795.3 | 4th | 12,000 | $0.000349 | 1st |
| batch128_bf16 | 11968.0 | 1st | 22,400 | $0.000479 | 2nd |
| batch128_unscaled | 9153.5 | 3rd | 17,600 | $0.000492 | 3rd |
| batch128_linear | 9297.4 | 2nd | 20,800 | $0.000573 | 4th |
| batch256_linear | 7303.0 | 5th | 35,200 | $0.001235 | 5th |
winners_agree: false. batch64 is fourth of five by throughput and first by
cost - it moves three places. The dearest configuration costs 3.5x the
cheapest, and none of that spread is visible in samples/s.
On the CPU run, winners_agree was true: batch64 was both the fastest and
the cheapest, so throughput was a perfectly adequate proxy and the study had
nothing to say. The inversion is not a property of the code. It is a property of
the hardware the code runs on.
The mechanism is visible in the two middle columns. samples_to_target is almost
identical on both devices - 12,000 / 17,600 / 20,800 / 35,200, differing only for
bf16 - because how many samples an optimiser needs to reach an accuracy is a
property of the optimisation, not of the chip. Throughput is the opposite: it
reordered completely. Cost is one divided by the other, so on a device where
throughput happens to agree with sample-efficiency you can measure the wrong
thing and be right anyway.
That is the failure mode worth naming: throughput was not a bad proxy on the CPU, it was a lucky one, and nothing in a samples/s dashboard tells you which of the two you are currently enjoying.
bf16 makes it concrete. It is the fastest configuration on the L4 by 36% over
batch64 - and 37% more expensive to get to the same accuracy, because it needs
22,400 samples instead of 12,000. Buying throughput and paying it back in extra
samples is invisible to every metric except this one.
Running this on an accelerator for the first time exposed two defects that CPU execution structurally could not:
- The device was hardcoded to
cpu. A study whose unit is dollars-per-hour of a machine could never be run on the machine that costs money. It silently measured vCPUs for fourteen minutes on a rented GPU beforenvidia-smishowed 0% utilisation. - No
torch.cuda.synchronize(). CUDA kernels are asynchronous:perf_counter()afterbackward()returns once the work is queued, not once it is done. Every timing would have measured launch overhead and reported it as compute. This is the same shape as trusting Kafka'sflush()to mean the writes landed - an async boundary read as if it were synchronous - and it is the kind of bug that makes a benchmark faster and wronger at once.
Both are fixed, and the result above is post-fix. The first bug is embarrassing; the second is the one that would have produced confident, publishable, wrong numbers.
The rented L4 instance is also the "quiet dedicated machine" this study had been asking for, and running the pinning experiment on it overturned the laptop result.
| laptop (contended) | dedicated box | |
|---|---|---|
| measured noise floor | 11.8% | 1.0% |
| pinning vs control | +41.3% | +0.6% |
| separable from control? | yes | no |
| disjointness effect | +6.1% | +0.5% |
The noise floor fell twelve-fold and the effect vanished with it. On a machine that is not fighting anything else, pinning does nothing measurable - +0.6% against a 1.0% floor.
So the laptop's +41.3%, which cleared its own noise floor by 3.5× and looked like the study's clean positive result, was contention. The calibration arm was right to be suspicious and was not suspicious enough: it correctly reported the floor as 11.8%, and an effect can clear a floor that high and still be an artifact of whatever else the machine was doing.
The thing that makes this checkable rather than embarrassing is that both runs carry their own noise floor, measured the same way, from two arms that are secretly the same configuration. Without that number the two results would just be two different answers.
The ladder's biggest win was num_workers=4. On an 8-core box that leaves the
main process sharing all 8 cores with 4 decode workers - everybody oversubscribed,
threads migrating, caches thrown away. Does partitioning the cores beat sharing
them, even though partitioning gives each side strictly fewer?
Four arms, because pinned_split bundles two interventions and an experiment
that cannot separate its own interventions has measured neither:
| arm | main | workers | threads | samples/s (median of 7) | range |
|---|---|---|---|---|---|
default |
all | all | default | 87.8 | 79.2-105.2 |
threads_capped |
all | all | 4 | 99.5 | 65.3-129.0 |
pinned_split |
0-3 | 4-7 | 4 | 140.6 | 129.4-146.3 |
pinned_half_shared |
0-3 | 0-3 | 4 | 132.5 | 113.9-138.5 |
Pinning is worth +41.3%, an effect 3.5x the measured noise floor.
The mechanism is not the one predicted. pinned_half_shared puts both sides
on the same four cores - same core count, shared instead of disjoint - and
captures nearly all of the gain. Disjointness is worth a further +6.1%, which is
inside the noise floor and not separable. So what helps is confining the
training process to four cores at all; whether the dataloader gets its own
four is not established. Without that fourth arm the obvious reading of "+41.3%"
is "partitioning removed contention", and this data does not support it.
PyTorch already defaults to 4 threads here - 8 logical cores, 4 physical - so
threads_capped is a no-op, and default and threads_capped are the same
configuration measured twice. That accident is the most useful thing in the
table, twice over.
It measures the noise floor. The gap between two identical configurations is not an effect, it is error: 11.8% at 7 repeats. Every other number has to clear it, and the disjointness result does not.
It validates the separability test itself. At 3 repeats those two
identical arms came out non-overlapping - 48.9-67.0 against 81.9-92.5. So
non-overlap did not imply a real effect: the test scored a false positive on
the one chance it was given, and every separable flag in that run was unsound.
Two 3-repeat runs duly disagreed on the sign of every effect - disjointness
was +8.4% and "separable" in one, −15.2% and not in the next.
So the module computes separability_test_valid and refuses to issue a verdict
when the calibration arms fail their own overlap check. At 3 repeats it reports
NO CONCLUSION; at 7 the arms overlap as they must and the verdicts become
readable. The precondition is checkable before the result is read, which is the
whole point - otherwise the 3-repeat run would have shipped whichever sign it
happened to produce, with a confident range test behind it.
worker_init_fnwas a closure, and the docstring justified that choice on the grounds that it runs in the worker process under spawn. The reasoning was right and the conclusion backwards: spawn pickles the init function to reach the child, and a local function is not picklable. It failed instantly withPicklingError: Can't pickle local object. It is a module-level class now.- The dataloader kwargs were unguarded for
num_workers=0. Torch raises onprefetch_factorthere rather than ignoring it. The study never runs with zero workers - contention is the whole point - so it only fired under test, which is precisely where an unguarded version would have sat unnoticed until someone reused the helper.
The scaling study ran on gloo/CPU for its whole life, and the honesty note made a specific, falsifiable claim about what would happen on real GPUs:
The shape - efficiency falling as the comm fraction grows - is what transfers; the percentages are not a prediction about NCCL.
Two NVIDIA A40s (rented, ~$0.80/hr) make that checkable. Same code, same
model, same batches - only --backend nccl and rank-to-device placement differ.
| run | backend | ws=1 | ws=2 | speedup | efficiency | comm fraction |
|---|---|---|---|---|---|---|
| weak | gloo/CPU | 43.6 | 69.8 | 1.60x | 80% | 6.3% ±20.7 |
| weak | nccl/A40 | 2224.3 | 3076.4 | 1.38x | 69% | 29.1% ±13.8 |
| strong | gloo/CPU | 51.2 | 71.8 | 1.40x | 70% | 15.2% |
| strong | nccl/A40 | 2610.8 | 3771.7 | 1.44x | 72% | 25.1% ±6.3 |
| no-decode | nccl/A40 | 4330.5 | 5439.1 | 1.26x | 63% | 24.8% ±8.8 |
Both halves of the prediction held. Throughput did not transfer at all - it is 51x higher (43.6 -> 2224.3 samples/s), exactly the "percentages are not a prediction" part. And the shape did transfer: efficiency still falls as the comm fraction rises, and the run with the highest comm fraction is still the one with the worst efficiency.
The strong-scaling efficiencies landing at 70% and 72% is a coincidence, not a confirmation, and it would be dishonest to present it as one - the weak-scaling pair is 80% vs 69% on the same two backends. The band transferred; the numbers inside it did not.
On gloo the comm fraction was 1.2-15.2% with spreads up to ±22 points - the README already called that a coin flip rather than a measurement. On NCCL it is 25-29% with spreads of ±6-14: both substantially larger and substantially more measurable.
Nothing about the communication changed. The model is the same, so the gradient volume per all-reduce is identical. What changed is that compute got ~51x faster and the interconnect did not, so the same fixed communication cost went from a rounding error to a quarter of the step.
That is the whole argument for why distributed-training work is interconnect work, and it only becomes visible when the compute side stops being the bottleneck.
nvidia-smi topo -m reports SYS between GPU0 and GPU1, and
nvidia-smi nvlink -s reports all links inactive. These A40s talk over PCIe
through the CPU root complex, and across NUMA nodes - GPU0 is on NUMA 0,
GPU1 on NUMA 1.
So this is close to the worst-case GPU interconnect, and the 25-29% comm fraction should be read as an upper bound. On NVLink the same experiment would show a markedly smaller number. The original honesty note said these figures were "not GPUs across NVLink" - that is still true. What changed is that the backend is now NCCL rather than gloo; the interconnect is still not NVLink, and the note has been updated to say which of the two limitations was retired and which was not.
Two GPUs also means world sizes 1 and 2 only. The gloo study reached 4 workers and showed efficiency falling further (59% weak, 49% strong); whether NCCL bends the same way at 4 is untested here.
The backend is now a flag rather than an assumption, which meant touching the code path every existing gloo number came from. Re-running gloo after the change gives 92% efficiency at ws=2 with 2.3% comm on the 96-core box - consistent with the CPU curve, so the refactor did not quietly invalidate the results it was extending.
The same torch.cuda.synchronize() bug from the cost study applies here and is
handled: without it the all-reduce would appear free, because it would not have
happened yet when perf_counter() was read.
Things this repository does not prove, and what each one would need.
| limitation | why |
|---|---|
| NCCL over NVLink, and world size 4 | not available: the rented pair is PCIe/SYS across NUMA, and it is two GPUs |
- The GPU numbers come from a single rented L4 on spot, in one session, at
three repeats. The
workersrung and the data-wait share are far outside the noise and are safe to quote; the kernel-level rungs are not, and the table says which is which. - Most measured numbers in this repo are CPU.
torch.cuda.is_available()isFalseand no GPU number is quoted anywhere. - Two torch versions are represented. The ladder, the traces and the scaling
studies were measured on
2.11.0+cpu; the cost-to-accuracy and CPU-pinning studies on2.13.0+cpu, because the toolchain moved underneath this repo mid-build. Each result file records its own version. Nothing is compared across that boundary, and the two are not silently pooled into one table. - The scaling study now has both backends, and neither is NVLink. The gloo
figures (100% / 80% / 59%) are CPU across real OS processes; the NCCL figures
(100% / 69%) are two A40s over PCIe across NUMA nodes, which
nvidia-smi topo -mreports asSYS. The prediction that the shape would transfer and the percentages would not was tested and held. The comm fractions are an upper bound - NVLink would be lower - and world size 4 is untested on NCCL. (An earlier version of this note said there was no scaling number at all. That was true when it was written and stopped being true when the study ran, and a stale honesty note is worse than none: it is a false claim in the section whose whole job is not making them.) - The
+compilerung genuinely fails on this machine (InductorError: Compiler: cl is not found) because Inductor's CPU backend needs MSVC. That is in the ledger as a failed row with the error, not omitted. - Absolute throughput here moved by more than 2x between sessions.
pinned_splitmeasured 63.8 samples/s while this laptop was carrying unrelated work at 100% CPU and 140.6 on a quiet one. Only within-session comparisons carry weight, which is why every arm is re-run inside each invocation rather than compared against a stored baseline. separable_from_controlis thinner than the effect size suggests: 129.4 against 129.0 is a margin of 0.4 samples/s. The claim that pinning helps rests on the effect being 3.5x the noise floor, not on that hair-width range test.- The cost-to-accuracy task is synthetic, so its convergence shape belongs to the generator. What transfers is that the two rankings can disagree and that only measuring both catches it - not the crossover point.
- Stopping resolution differs across batch sizes. Accuracy is evaluated every
25 steps, so
samples_to_targetis quantised to 1,600 samples at batch 64 and 6,400 at batch 256 - up to 18% of that row. It does not explain a 2.9x gap, and it is a real bias in the large-batch direction. - Absolute throughput is meaningless outside this synthetic workload. The rung order is the transferable result; the rung values are not.