The Unreasonable Effectiveness of Open Science in AI: A Replication Study

TL;DR

This paper tackles the reproducibility crisis in AI by conducting a systematic replication of 30 highly cited AI studies, focusing on R3 (data) and R4 (code+data) reproducibility types. It employs structured code and data retrieval procedures and a 40-hour replication limit per study to assess replication feasibility, yielding 11 successful or partially successful replications out of 22 attempts. The key findings show an 86% repro success rate when both code and data are available, versus 33% with data alone, and highlight data documentation quality as a stronger predictor of success than code documentation. The work argues for stronger open-science norms and concrete replication artifacts, with implications for AI research practices and policy, including notable relevance to LLM evaluations where code and data may be proprietary.

Abstract

A reproducibility crisis has been reported in science, but the extent to which it affects AI research is not yet fully understood. Therefore, we performed a systematic replication study including 30 highly cited AI studies relying on original materials when available. In the end, eight articles were rejected because they required access to data or hardware that was practically impossible to acquire as part of the project. Six articles were successfully reproduced, while five were partially reproduced. In total, 50% of the articles included was reproduced to some extent. The availability of code and data correlate strongly with reproducibility, as 86% of articles that shared code and data were fully or partly reproduced, while this was true for 33% of articles that shared only data. The quality of the data documentation correlates with successful replication. Poorly documented or miss-specified data will probably result in unsuccessful replication. Surprisingly, the quality of the code documentation does not correlate with successful replication. Whether the code is poorly documented, partially missing, or not versioned is not important for successful replication, as long as the code is shared. This study emphasizes the effectiveness of open science and the importance of properly documenting data work.

Figures (2)

• Figure 1: An R3 type reproducibility study is conducted where only the research report and data are shared publicly, so the code implementing both the AI method and experiment as well as hardware an ancillary software differ from the original experiment, as illustrated by differing icons. Many assumptions must be made when conducting a reproducibility study as the research report cannot possibly cover all design decisions. Difference in implementation as well as differences in hardware and ancillary software can introduce differences between the output produced by the original and the reproducibility experiments. Sometimes the differences lead to different conclusions even after doing the exact same analysis. This is indicated for one of the reproducibility experiments by the black light bulb icon. When the original article describes several experiments, the overall result of the reproducibility study depends on whether all the conclusions of all these experiments agree or not. If only a subset of the conclusions is the same, the reproducibility study is considered Partial Success.
• Figure 2: Overview of the results.