DREAM: Debugging and Repairing AutoML Pipelines

TL;DR

DREAM tackles critical AutoML bugs by introducing three mechanisms: search space expansion, feedback monitoring, and a feedback-driven search that uses rich training-time feedback to guide repairs. It monitors model training and evaluation, detects performance and ineffective-search bugs, and automatically expands the search space and adapts the search strategy accordingly. Empirical results across four datasets show DREAM achieving an average best accuracy of $83.21\%$ in 25 trials, substantially outperforming AutoKeras baselines and reaching target scores far faster. The work demonstrates a practical, scalable approach to debugging AutoML pipelines with publicly available code and data, potentially reducing compute and carbon footprints in AutoML workflows.

Abstract

Deep Learning models have become an integrated component of modern software systems. In response to the challenge of model design, researchers proposed Automated Machine Learning (AutoML) systems, which automatically search for model architecture and hyperparameters for a given task. Like other software systems, existing AutoML systems suffer from bugs. We identify two common and severe bugs in AutoML, performance bug (i.e., searching for the desired model takes an unreasonably long time) and ineffective search bug (i.e., AutoML systems are not able to find an accurate enough model). After analyzing the workflow of AutoML, we observe that existing AutoML systems overlook potential opportunities in search space, search method, and search feedback, which results in performance and ineffective search bugs. Based on our analysis, we design and implement DREAM, an automatic debugging and repairing system for AutoML systems. It monitors the process of AutoML to collect detailed feedback and automatically repairs bugs by expanding search space and leveraging a feedback-driven search strategy. Our evaluation results show that DREAM can effectively and efficiently repair AutoML bugs.

Figures (10)

• Figure 1: The Basic Steps of the AutoML Pipeline
• Figure 2: Performance bug in AutoML When Search for Target Score
• Figure 3: Ineffective Search Trials of the AutoKeras Pipeline
• Figure 4: Overarching Design of DREAM
• Figure 5: Evaluation Results in 25 Trials on CIFAR-100 and Tiny ImageNet Datasets