Benchmark In Milliseconds · Listed Articles

Alex Kladov (matklad) explains his rule of thumb for micro benchmarks: tune the input size until a run takes about 300ms, so results read as easy integer milliseconds, fixed costs don't skew them, the numbers stay in a human-perceptible range, and repeated runs to eyeball variance stay fast.

How long should a micro benchmark run? My rule of thumb is to tweak the input size until the benchmark takes about 300ms, for the following reasons:

Milliseconds are integers ranging from 1 to 999. Enough precision to notice even a small improvement, and easy to scan visually. No need for different units or floating points (compare 1.31s with239ms ).

Anything faster than, say, 10ms risks being skewed by fixed costs (e.g, interpreter startup). Hundreds of milliseconds is an eternity for a computer, usually enough to make one-off overheads irrelevant without using fancier (= less robust) techniques to explicitly account for them.

For a human, hundreds of milliseconds is fast, but noticeable. Pushing numbers into human-perceptible range allows me to use my intuitive sense of time and speed, it doesn’t rely exclusively on numeracy. It’s plain fun to see, as a result of optimization work, how a previously lagging CLI command becomes “instant”.

But anything longer than a second makes iterating on the benchmark slower than it needs to be. Running a benchmark 10 times in a row to eyeball variance should be fast!

The imminently-to-be-stated assumption here is that the purpose of benchmarking isn’t so much a precise measurement of performance, but rather providing the author with enough intuition to make a correct decision.