Benchmarking in Optimization: Best Practice and Open Issues
Thomas Bartz-Beielstein, Carola Doerr, Daan van den Berg, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, William La Cava, Manuel Lopez-Ibanez, Katherine M. Malan, Jason H. Moore, Boris Naujoks, Patryk Orzechowski, Vanessa Volz, Markus Wagner, Thomas Weise
TL;DR
This survey articulates a comprehensive, practice-oriented framework for benchmarking optimization algorithms. It defines eight core topics—goals, problem instances, algorithms, performance measures, analysis, experimental design, presentation, and reproducibility—and provides concrete guidelines, exemplary best practices, and open issues for each. By detailing how to select problems and algorithms, measure performance, analyze results, and ensure reproducibility, the paper aims to improve the reliability, comparability, and impact of benchmark studies. It positions benchmarking as a living, evolving discipline that requires community engagement, standardized interfaces, and transparent reporting to bridge theory and practice in optimization.
Abstract
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.