Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies

TL;DR

This work critically evaluates the reproducibility of LLM-centric empirical software engineering studies by analyzing 85 ICSE 2024 and ASE 2024 papers, with a focus on artefact availability and OpenAI-based experiments. Through a replication framework and Bayesian bootstrapping, it finds that only 5 studies were initially fit for reproduction and none could be fully reproduced, highlighting pervasive artefact gaps, dependency/version issues, and model deprecations. The study further reveals that ACM artefact badges do not reliably signal reproducibility over time and argues for stronger, standardized artefact practices and transparent reporting. The authors propose concrete recommendations for authors, venues, and funding agencies to improve long-term reproducibility and artefact accessibility in a rapidly evolving LLM landscape.

Abstract

Large Language Models have gained remarkable interest in industry and academia. The increasing interest in LLMs in academia is also reflected in the number of publications on this topic over the last years. For instance, alone 78 of the around 425 publications at ICSE 2024 performed experiments with LLMs. Conducting empirical studies with LLMs remains challenging and raises questions on how to achieve reproducible results, for both researchers and practitioners. One important step towards excelling in empirical research on LLM and their application is to first understand to what extent current research results are eventually reproducible and what factors may impede reproducibility. This investigation is within the scope of our work. We contribute an analysis of the reproducibility of LLM-centric studies, provide insights into the factors impeding reproducibility, and discuss suggestions on how to improve the current state. In particular, we studied the 85 articles describing LLM-centric studies, published at ICSE 2024 and ASE 2024. Of the 85 articles, 18 provided research artefacts and used OpenAI models. We attempted to replicate those 18 studies. Of the 18 studies, only five were sufficiently complete and executable. For none of the five studies, we were able to fully reproduce the results. Two studies seemed to be partially reproducible, and three studies did not seem to be reproducible. Our results highlight not only the need for stricter research artefact evaluations but also for more robust study designs to ensure the reproducible value of future publications.

Figures (2)

• Figure 1: Summary of Research Approach
• Figure 2: Top 10 llms Employed in the Analyzed 85 Articles.