Pitfalls in Machine Learning Research: Reexamining the Development Cycle

TL;DR

The paper analyzes three bottlenecks in ML development—design, data collection, and evaluation—through concrete case studies (Gender and ML, Tiny Images, NAS) to show how assumptions, biases, and weak baselines yield non-replicable or harmful results. It proposes practical remedies, including stakeholder-driven design, hypothesis-driven data collection, dataset documentation and revision, statistical validation with seed variation, and third‑party replication. The work advocates cultural and process changes, such as integrating ethical review and slow science principles, to improve reliability and real-world impact. Overall, it offers a structured, action-oriented roadmap to reduce illusory findings and promote more rigorous, responsible data science.

Abstract

Machine learning has the potential to fuel further advances in data science, but it is greatly hindered by an ad hoc design process, poor data hygiene, and a lack of statistical rigor in model evaluation. Recently, these issues have begun to attract more attention as they have caused public and embarrassing issues in research and development. Drawing from our experience as machine learning researchers, we follow the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements. At each step, case studies are introduced to highlight how these pitfalls occur in practice, and where things could be improved.