Cleaning up the Mess
Haocong Luo, Ataberk Olgun, Maria Makeenkova, F. Nisa Bostanci, Geraldo F. Oliveira, A. Giray Yaglikci, Onur Mutlu
TL;DR
The Mess paper claimed Ramulator 2.0 poorly matched real memory performance and introduced a new benchmark; this work shows those conclusions arose from configuration and methodology errors. By building a corrected Ramulator 2.0 setup and a Mess-like workload generator, the authors demonstrate latency-bandwidth curves that closely resemble real hardware, debunking the Mess paper's central claims. They also reveal irreproducible and erroneous DAMOV results caused by missing sources and improper statistics usage, underscoring the need for rigorous validation, honest artifact reporting, and proactive communication with simulator authors. The contribution provides open-source tools and a clarified evaluation methodology that improve reproducibility and reliability in memory-system benchmarking.
Abstract
A MICRO 2024 best paper runner-up publication (the Mess paper) with all three artifact badges awarded (including "Reproducible") proposes a new benchmark to evaluate real and simulated memory system performance. In this paper, we demonstrate that the Ramulator 2.0 simulation results reported in the Mess paper are incorrect and, at the time of the publication of the Mess paper, irreproducible. We find that the authors of Mess paper made multiple trivial human errors in both the configuration and usage of the simulators. We show that by correctly configuring Ramulator 2.0, Ramulator 2.0's simulated memory system performance actually resembles real system characteristics well, and thus a key claimed contribution of the Mess paper is factually incorrect. We also identify that the DAMOV simulation results in the Mess paper use wrong simulation statistics that are unrelated to the simulated DRAM performance. Moreover, the Mess paper's artifact repository lacks the necessary sources to fully reproduce all the Mess paper's results. Our work corrects the Mess paper's errors regarding Ramulator 2.0 and identifies important issues in the Mess paper's memory simulator evaluation methodology. We emphasize the importance of both carefully and rigorously validating simulation results and contacting simulator authors and developers, in true open source spirit, to ensure these simulators are used with correct configurations and as intended. We encourage the computer architecture community to correct the Mess paper's errors. This is necessary to prevent the propagation of inaccurate and misleading results, and to maintain the reliability of the scientific record. Our investigation also opens up questions about the integrity of the review and artifact evaluation processes. To aid future work, our source code and scripts are openly available at https://github.com/CMU-SAFARI/ramulator2/tree/mess.