Empirical Characterization of Logging Smells in Machine Learning Code

TL;DR

The paper tackles the problem of understanding logging quality in ML-based software by combining large-scale mining of open-source ML repositories with a practitioner survey. It proposes a two-phase methodology to identify ML-specific logging smells, extending traditional log smell taxonomies to account for ML workflows, experiment tracking, and data pipelines. The study uses an LLM-assisted identification workflow, iterative human validation, and a subsequent survey to measure relevance, frequency, and severity, aiming to produce a formal taxonomy of smells and practical guidelines. Its findings are expected to improve observability and reproducibility in ML deployments by guiding tooling and best practices for ML logging.

Abstract

\underline{Context:} Logging is a fundamental yet complex practice in software engineering, essential for monitoring, debugging, and auditing software systems. With the increasing integration of machine learning (ML) components into software systems, effective logging has become critical to ensure reproducibility, traceability, and observability throughout model training and deployment. Although various general-purpose and ML-specific logging frameworks exist, little is known about how these tools are actually used in practice or whether ML practitioners adopt consistent and effective logging strategies. To date, no empirical study has systematically characterized recurring bad logging practices--or logging smells--in ML System. \underline{Goal:} This study aims to empirically identify and characterize logging smells in ML systems, providing an evidence-based understanding of how logging is implemented and challenged in practice. \underline{Method:} We propose to conduct a large-scale mining of open-source ML repositories hosted on GitHub to catalogue recurring logging smells. Subsequently, a practitioner survey involving ML engineers will be conducted to assess the perceived relevance, severity, and frequency of the identified smells. \underline{Limitations:} % While The study's limitations include that While our findings may not be generalizable to closed-source industrial projects, we believe our study provides an essential step toward understanding and improving logging practices in ML development.