Pitfalls of Benchmarking on Modern Systems
Summary
The article discusses the pitfalls of benchmarking modern systems, using fictitious benchmarks to illustrate how results can vary due to hardware, OS scheduling, and runtime optimizations. It emphasizes that performance data often represents a distribution rather than a single point and recommends methodological practices like multiple runs and controlling variables. It also outlines practical factors to consider, including JVM/JIT behavior, garbage collection, memory, and environmental factors.