Taming OS and hardware noise for repeatable benchmarks.\nOptimizing code starts with measuring it, and a measurement is only\nuseful if it is repeatable: a 2% improvement is invisible under 5% of\nnoise. Yet on an untuned machine the same binary can easily run several\npercent faster or slower between runs. In this post we take a tiny\nbenchmark and tune the machine step by step, re-measuring after every\nchange, until runs become deterministic.<sup>1</sup> <sup>1</sup>\nNote that tuning for\nbenchmarking is not the same as tuning for performance: a benchmark\nwants the machine repeatable, even at the cost of some peak speed. A\nproduction box, however, wants every last bit of speed.
\nOur running example sums an array of doubles, in short bursts. Real\nservices rarely hammer the CPU continuously: they handle a request, sit\nidle, and wake up for the next one. Each timed iteration here runs a\nburst of 256 sums after a 2 ms idle gap, with the gap excluded from the\nmeasurement<sup>2</sup> <sup>2</sup>\nPauseTiming / ResumeTiming keep the sleep out of the\nmeasured time, and DoNotOptimize keeps the result alive past the\noptimizer; without it the compiler deletes the entire loop.
static auto BM_Sum(benchmark::State& state) -> void {\n alignas(64) static std::array<double, 4096> data;\n std::iota(data.begin(), data.end(), 0.0);\n for (auto _ : state) {\n state.PauseTiming(); // Idle between bursts, like a real service\n std::this_thread::sleep_for(std::chrono::milliseconds(2));\n state.ResumeTiming();\n for (auto i = 0; i < 256; ++i) {\n auto sum = std::accumulate(data.cbegin(), data.cend(), 0.0);\n benchmark::DoNotOptimize(sum);\n }\n }\n}\nBENCHMARK(BM_Sum);Compile it in release with all optimizations, -O3, and -march=native -mtune=native -flto -ffast-math. Then run ten repetitions and\naggregate them
$ ./benchmark --benchmark_repetitions=10 --benchmark_min_time=100x\nBM_Sum_mean 99575 ns\nBM_Sum_stddev 2704 ns\nBM_Sum_cv 2.72 %The interesting line is cv, the coefficient of variation: standard\ndeviation divided by mean. Almost 3% of run-to-run noise—any\noptimization smaller than that is invisible. Let’s bring it down.
Before turning any knob, look at what you are tuning. lstopo draws\nthe whole machine in one picture: caches, cores, SMT pairs, and the PCIe\ndevices hanging off them. Start with my laptop
Here the choice of core changes what you measure: land on CPU 4 and you get an E-core at lower clocks; on CPU 12 you lose the L3 too. Now compare that against my homelab server
\nOn the server every core is as good as any other: homogeneous machines make better benchmarking boxes. The PCIe side matters once a benchmark touches I/O: it shows which NVMe or NIC you are exercising and, on multi-socket machines, which NUMA node it hangs off.
\nThe scheduler is free to migrate the benchmark between cores, and every migration throws away warm caches. On hybrid CPUs it’s worse: performance and efficiency cores run the same code at very different speeds, so results turn bimodal depending on where the process lands. Pin the benchmark to a single core (on hybrid parts, a P-core)
\n$ taskset -c 2 ./benchmark ...The mean falls to 55.3 µs and the CV better than halves, to 1.06%.\nThe win is bigger than migration costs alone would suggest: every burst\nnow wakes the same core, so that core’s clock never has time to sag\nbetween bursts.<sup>3</sup> <sup>3</sup>\nPinning puts the benchmark onto the core but does\nnot keep other tasks off it. On a busy box, go further and reserve\nthe core for the benchmark alone, either on the kernel command line\n(isolcpus=2 nohz_full=2 rcu_nocbs=2) or at runtime with a cpuset\ncgroup.
By default Linux scales the CPU frequency with load, so the benchmark\nstarts on a cold, slow clock and finishes on a hot, fast one. Switch\nthe frequency governor to performance to keep clocks locked high
$ sudo cpupower frequency-set --governor performanceand verify it took effect
\n$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor\nperformanceRe-measuring gives a mean of 54.9 µs and a CV of 0.79%. The\nincrement looks modest only because pinning already kept our core’s\nclock warm: on its own, the performance governor takes the unpinned\nbaseline from 99.6 µs straight to 54.5 µs. Either way, no burst ever\nwakes up on a cold clock again.
\nCPU still shares its execution units and L1/L2 caches with its SMT sibling: anything the scheduler places there perturbs our measurement. Disable SMT entirely
\n$ echo off | sudo tee /sys/devices/system/cpu/smt/controlThe CV drops to 0.26%, three times better: the core now has its\nexecution units and caches all to itself.
\nEven with the performance governor, turbo frequencies vary with temperature and power budget: the same run on a warm machine clocks lower than on a cool one. Disable turbo for stable clocks
\n$ echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boostOn this machine nothing changes, since our short bursts never gave the\nsilicon time to boost anyway. On a machine where turbo does engage,\nexpect the mean to climb instead: you are giving up peak performance.\nThat trade is fine, since when optimizing we care about relative\nnumbers, and those are now comparable across runs.<sup>4</sup> <sup>4</sup>\nLow-latency\nproduction tuning makes the opposite call and keeps turbo on: there,\nevery nanosecond counts. The most latency-sensitive trading shops go\nfurther and run overclocked servers, locked at a fixed all-core\nfrequency above stock—speed and stable clocks, bought with better\ncooling.
\nHere is the whole journey in one table, each row adding one change on\ntop of all the previous ones. We went from almost 3% of noise down to\n0.26%, and got 1.8x faster along the way; differences of half a\npercent are now real, measurable signal.<sup>5</sup> <sup>5</sup>\nFeel free to reproduce on\nyour machine using the benchmark from my CppPlayground repository.
\n| Step | Mean | StdDev | CV |
|---|---|---|---|
| Untuned | 99.6 µs | 2.70 µs | 2.72% |
| + pinned to one core | 55.3 µs | 0.59 µs | 1.06% |
| + performance governor | 54.9 µs | 0.43 µs | 0.79% |
| + hyperthreading off | 55.3 µs | 0.15 µs | 0.26% |
| + turbo disabled | 55.5 µs | 0.14 µs | 0.26% |
On busier machines there is a longer tail of knobs worth trying:\ndisabling address space layout randomization, the NMI watchdog, or\ntransparent huge pages. The bench-remote.sh script applies all. None\nof it survives a reboot, which is exactly what you want: tune, measure,\nand reboot back to a normal machine.
\nLong live reproducible benchmarks!
","headings":[{"level":1,"text":"Tuning a Server for Benchmarking","id":"tuning-a-server-for-benchmarking"},{"level":2,"text":"A noisy baseline","id":"a-noisy-baseline"},{"level":2,"text":"Know your hardware","id":"know-your-hardware"},{"level":2,"text":"Pin to a core","id":"pin-to-a-core"},{"level":2,"text":"Lock the CPU frequency","id":"lock-the-cpu-frequency"},{"level":2,"text":"Disable hyperthreading","id":"disable-hyperthreading"},{"level":2,"text":"Disable turbo boost","id":"disable-turbo-boost"},{"level":2,"text":"Summary","id":"summary"}]}}