Automated Reproducibility Has a Problem Statement Problem

TL;DR

This work addresses the lack of a general, formal problem statement for reproducibility in empirical AI and proposes a graph-based representation of a study as hypotheses linked to experiments, with data, methods, outcomes, analyses, and interpretations. It implements a proof-of-concept extraction using a prompt-driven LLM to auto-derive this structure from 20 publications and validates the outputs with the original authors. Results show the representation can capture core elements in most cases, though it misses some details for longer or visually-driven results, highlighting areas for improvement such as figure handling and finer-grained outputs. The study demonstrates feasibility of automated, cross-study reproducibility via a unified framework and points to future work on enhancing extraction fidelity,Prompt-engineering, and expanding to broader domains and multi-agent setups.

Abstract

Background. Reproducibility is essential to the scientific method, but reproduction is often a laborious task. Recent works have attempted to automate this process and relieve researchers of this workload. However, due to varying definitions of reproducibility, a clear problem statement is missing. Objectives. Create a generalisable problem statement, applicable to any empirical study. We hypothesise that we can represent any empirical study using a structure based on the scientific method and that this representation can be automatically extracted from any publication, and captures the essence of the study. Methods. We apply our definition of reproducibility as a problem statement for the automatisation of reproducibility by automatically extracting the hypotheses, experiments and interpretations of 20 studies and assess the quality based on assessments by the original authors of each study. Results. We create a dataset representing the reproducibility problem, consisting of the representation of 20 studies. The majority of author feedback is positive, for all parts of the representation. In a few cases, our method failed to capture all elements of the study. We also find room for improvement at capturing specific details, such as results of experiments. Conclusions. We conclude that our formulation of the problem is able to capture the concept of reproducibility in empirical AI studies across a wide range of subfields. Authors of original publications generally agree that the produced structure is representative of their work; we believe improvements can be achieved by applying our findings to create a more structured and fine-grained output in future work.

Figures (5)

• Figure 1: A general overview of our problem statement of reproducing an empirical study. We model the problem as a graph: A study contains one or more hypotheses, evaluated and tested through multiple experiments. Outcomes are analysed and interpreted to support or repudiate a given hypothesis. The analysis is reduced to elements needed for assessing the outcome of experiments. The interpretation element is graphically distinguished, since we treat it as static, whereas traditionally, these can be more flexible.
• Figure 2: Evaluation of the hypotheses captured by the LLM by the original authors, using a 7-point Likert scale, including missing hypotheses supplemented by the authors.
• Figure 3: Evaluation of the extracted experiment descriptions by the original authors, using a 5-point Likert scale plot. This includes missing experiments supplemented by the authors.
• Figure 4: Evaluation of the extracted experiment details by the original authors, using a 5-point Likert scale.
• Figure 5: Evaluation of the extracted experiment interpretations by the original authors, using a 5-point Likert scale. Authors were given the opportunity to adapt the phrasing.