Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis

TL;DR

This paper investigates reproducibility challenges in deep learning software and surveys existing guidelines. It then tests and extends these guidelines through a case study of TRUNK, a hierarchical Tree-Based Unidirectional Neural Network, to evaluate how environment replication, end-to-end training, architecture disclosure, and data-processing transparency influence reproducibility. Through experiments on EMNIST, CIFAR-10, and SVHN, the authors highlight the critical roles of software environment manifests, training pipelines, and sensitivity analyses, revealing substantial variability due to non-determinism and missing details in the original Artefacts. They propose concrete enhancements to guidelines, including GPU-focused manifests, integrated end-to-end training, and high-level configuration files, to bridge the gap between research and practical deployment of deep learning models.

Abstract

The field of deep learning has witnessed significant breakthroughs, spanning various applications, and fundamentally transforming current software capabilities. However, alongside these advancements, there have been increasing concerns about reproducing the results of these deep learning methods. This is significant because reproducibility is the foundation of reliability and validity in software development, particularly in the rapidly evolving domain of deep learning. The difficulty of reproducibility may arise due to several reasons, including having differences from the original execution environment, incompatible software libraries, proprietary data and source code, lack of transparency, and the stochastic nature in some software. A study conducted by the Nature journal reveals that more than 70% of researchers failed to reproduce other researchers experiments and over 50% failed to reproduce their own experiments. Irreproducibility of deep learning poses significant challenges for researchers and practitioners. To address these concerns, this paper presents a systematic approach at analyzing and improving the reproducibility of deep learning models by demonstrating these guidelines using a case study. We illustrate the patterns and anti-patterns involved with these guidelines for improving the reproducibility of deep learning models. These guidelines encompass establishing a methodology to replicate the original software environment, implementing end-to-end training and testing algorithms, disclosing architectural designs, and enhancing transparency in data processing and training pipelines. We also conduct a sensitivity analysis to understand the model performance across diverse conditions. By implementing these strategies, we aim to bridge the gap between research and practice, so that innovations in deep learning can be effectively reproduced and deployed within software.

Figures (12)

• Figure 1: Even with the provided source code and datasets, reproducibility challenges in deep learning still persist pineau2020.
• Figure 2: (a) Monolithic Architectures vs (b) TRUNK TRUNK_goel. We will use TRUNK, a type of hierarchical neural network, as a case study to demonstrate our guidelines for reproducibility.
• Figure 3: Experiment Methodology used to Test the Reproducibility of TRUNK
• Figure 4: We choose TRUNK for our case study analysis due to its complexity of the network architecture design varying by dataset (a) EMNIST (b) CIFAR-10 (c) SVHN. The red node is the root node of the tree, the gray nodes are the supergroups, and the green nodes are the leaf nodes.
• Figure 5: Comparison of the EMNIST (a) Tree Structure Developed by TRUNK Authors with (b) Our Reproduced Tree Structure. The red node is the root node of the tree, the gray nodes are the supergroups, and the green nodes are the leaf nodes.