Standardization of Post-Publication Code Verification by Journals is Possible with the Support of the Community

TL;DR

The paper addresses the reproducibility gap in ML by proposing a journal-level post-publication verification framework inspired by ACM badges. It advocates for independent replication reports to earn up to two publicly visible badges embedded in article metadata, extending verification beyond pre-publication checks. The authors outline a concrete workflow, discuss benefits for credibility, training, and risk management, and acknowledge challenges such as workload, partial reproducibility, and potential misuse. If adopted, the framework would complement existing reproducibility efforts and promote sustainable, community-driven verification in ML research.

Abstract

Reproducibility remains a challenge in machine learning research. While code and data availability requirements have become increasingly common, post-publication verification in journals is still limited and unformalized. This position paper argues that it is plausible for journals and conference proceedings to implement post-publication verification. We propose a modification to ACM pre-publication verification badges that allows independent researchers to submit post-publication code replications to the journal, leading to visible verification badges included in the article metadata. Each article may earn up to two badges, each linked to verified code in its corresponding public repository. We describe the motivation, related initiatives, a formal framework, the potential impact, possible limitations, and alternative views.

Figures (1)

• Figure 1: Reproducibility policies in journals, conferences and ACM-affiliated venues.