Online Meta-learning for AutoML in Real-time (OnMAR)

TL;DR

This work tackles real-time AutoML by introducing OnMAR, a model-agnostic online meta-learning framework that predicts the quality of current designs and triggers GA-driven redesign only when needed, reducing runtime without sacrificing performance. A complementary OffMAR baseline tests offline meta-learning in a two-phase pipeline, providing a rigorous comparison. Across three real-time AutoML applications—clustering algorithm composition, CNN configuration, and video classification pipeline configuration—OnMAR generally matches or surpasses state-of-the-art baselines while delivering faster runtimes, with XGBoost often emerging as a strong default meta-learner. The approach leverages extensive meta-features (including Exploratory Landscape Analysis) and evaluates multiple meta-learners (kNN, RF, XGBoost) to demonstrate model-agnostic applicability and practical impact for real-time ML design. These findings suggest OnMAR as a versatile, efficient pathway for deploying adaptive AutoML in real-time tasks.

Abstract

Automated machine learning (AutoML) is a research area focusing on using optimisation techniques to design machine learning (ML) algorithms, alleviating the need for a human to perform manual algorithm design. Real-time AutoML enables the design process to happen while the ML algorithm is being applied to a task. Real-time AutoML is an emerging research area, as such existing real-time AutoML techniques need improvement with respect to the quality of designs and time taken to create designs. To address these issues, this study proposes an Online Meta-learning for AutoML in Real-time (OnMAR) approach. Meta-learning gathers information about the optimisation process undertaken by the ML algorithm in the form of meta-features. Meta-features are used in conjunction with a meta-learner to optimise the optimisation process. The OnMAR approach uses a meta-learner to predict the accuracy of an ML design. If the accuracy predicted by the meta-learner is sufficient, the design is used, and if the predicted accuracy is low, an optimisation technique creates a new design. A genetic algorithm (GA) is the optimisation technique used as part of the OnMAR approach. Different meta-learners (k-nearest neighbours, random forest and XGBoost) are tested. The OnMAR approach is model-agnostic (i.e. not specific to a single real-time AutoML application) and therefore evaluated on three different real-time AutoML applications, namely: composing an image clustering algorithm, configuring the hyper-parameters of a convolutional neural network, and configuring a video classification pipeline. The OnMAR approach is effective, matching or outperforming existing real-time AutoML approaches, with the added benefit of a faster runtime.

Figures (10)

• Figure 1: A basic illustration of real-time AutoML showing how the design algorithm is used to find a new design for each timestep of the application algorithm
• Figure 2: The OnMAR approach.
• Figure 3: The first phase of the OffMAR approach
• Figure 4: The second phase of the OffMAR approach
• Figure 5: Accuracy of the clustering algorithm designed using B-Comp, OffMAR (blue) and OnMAR (maroon).