Artifact Evaluation for Distributed Systems: Current Practices and Beyond

TL;DR

The paper addresses the reproducibility crisis in science as it relates to distributed systems by surveying the current artifact evaluation practices (AET) at major conferences and journals. It contrasts these practices with established guidelines from other computing disciplines, highlighting a lag in unification and a bias toward simulation over emulation due to infrastructure and time costs. The authors synthesize state-of-practice findings and propose practical checklists and recommendations for artifact authors, AE committees, and the broader community, including process improvements, resource considerations, and cloud access strategies. Through these contributions, the work aims to raise artifact quality and submission rates in distributed systems while fostering dialogue that can generalize to other domains.

Abstract

Although repeatability and reproducibility are essential in science, failed attempts to replicate results across diverse fields made some scientists argue for a reproducibility crisis. In response, several high-profile venues within computing established artifact evaluation tracks, a systematic procedure for evaluating and badging research artifacts, with an increasing number of artifacts submitted. This study compiles recent artifact evaluation procedures and guidelines to show how artifact evaluation in distributed systems research lags behind other computing disciplines and/or is less unified and more complex. We further argue that current artifact assessment criteria are uncoordinated and insufficient for the unique challenges of distributed systems research. We examine the current state of the practice for artifacts and their evaluation to provide recommendations to assist artifact authors, reviewers, and track chairs. We summarize the recommendations and best practices as checklists for artifact authors and evaluation committees. Although our recommendations alone will not resolve the repeatability and reproducibility crisis, we want to start a discussion in our community to increase the number of submitted artifacts and their quality over time.

Figures (2)

• Figure 1: Conferences and journals with AETs.
• Figure 2: Mapping between challenges (see \ref{['challenges']}) and the existing state of practices (shown in gray, see \ref{['best-practices']}) and our recommendations (shown in pink, see \ref{['our-recommendations']}). Some of the practices and recommendations do not impact the challenges.