Open-Source Drift Detection Tools in Action: Insights from Two Use Cases
Rieke Müller, Mohamed Abdelaal, Davor Stjelja
TL;DR
Data drift challenges ML reliability; the paper presents D3Bench, a microbenchmark comparing open-source drift-detection tools (Evidently AI, NannyML, Alibi-Detect) on two real-world smart-building time-series use cases. The study evaluates functional and non-functional criteria, revealing Evidently AI’s strength in detecting general drift and NannyML’s precision in drift timing and its effect on predictive accuracy, with Alibi-Detect offering niche capabilities in certain domains. The findings guide tool selection and advocate a combined workflow to robustly detect and interpret data drift in time-series environments. The work also highlights limitations to univariate methods and suggests future extensions to multivariate and image data drift detection.
Abstract
Data drifts pose a critical challenge in the lifecycle of machine learning (ML) models, affecting their performance and reliability. In response to this challenge, we present a microbenchmark study, called D3Bench, which evaluates the efficacy of open-source drift detection tools. D3Bench examines the capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world data from two smart building use cases.We prioritize assessing the functional suitability of these tools to identify and analyze data drifts. Furthermore, we consider a comprehensive set of non-functional criteria, such as the integrability with ML pipelines, the adaptability to diverse data types, user-friendliness, computational efficiency, and resource demands. Our findings reveal that Evidently AI stands out for its general data drift detection, whereas NannyML excels at pinpointing the precise timing of shifts and evaluating their consequent effects on predictive accuracy.