Active Benchmarking


Passive Benchmarking

Debugging benchmarks is something I've done for many years, and I've seen an amazing and comical variety of failure modes. The problem is that benchmark tools are often run without understanding what they are testing or checking that the results are valid. This can lead to poor development or architectural choices that haunt you later on. I previously summarized this situation as:

casual benchmarking: you benchmark A, but actually measure B, and conclude you've measured C.

Accurate benchmarking is important. Your company may choose a different server platform, programming stack, database, or application vendor, based on benchmark results. Accurate results enable good choices to be made: picking options that deliver the best price performance, and are most likely to scale under load.