Lift What You Can: Green Online Learning with Heterogeneous Ensembles

TL;DR

The paper tackles sustainable online learning by enabling resource-aware training of heterogeneous ensembles (HEROS) in data streams. It formalizes training under budget as a Markov decision process and introduces the $\zeta$-policy, a greedy, threshold-based method that prioritizes low-cost models within a near-optimal performance band, with theoretical guarantees and empirical validation on 11 datasets. The theoretical analysis shows that for small $\zeta$, the $\zeta$-policy achieves higher asymptotic performance and lower resource costs than baseline methods, while remaining within $\zeta$ of the best possible performance. Experiments demonstrate that $\zeta$-policy often yields higher AUROC and substantially lower energy consumption than alternatives, and it effectively adapts to concept drift, offering a practical approach to green online learning. Overall, the framework provides a principled, tunable balance between predictive quality and environmental impact in streaming scenarios.

Abstract

Ensemble methods for stream mining necessitate managing multiple models and updating them as data distributions evolve. Considering the calls for more sustainability, established methods are however not sufficiently considerate of ensemble members' computational expenses and instead overly focus on predictive capabilities. To address these challenges and enable green online learning, we propose heterogeneous online ensembles (HEROS). For every training step, HEROS chooses a subset of models from a pool of models initialized with diverse hyperparameter choices under resource constraints to train. We introduce a Markov decision process to theoretically capture the trade-offs between predictive performance and sustainability constraints. Based on this framework, we present different policies for choosing which models to train on incoming data. Most notably, we propose the novel $ζ$-policy, which focuses on training near-optimal models at reduced costs. Using a stochastic model, we theoretically prove that our $ζ$-policy achieves near optimal performance while using fewer resources compared to the best performing policy. In our experiments across 11 benchmark datasets, we find empiric evidence that our $ζ$-policy is a strong contribution to the state-of-the-art, demonstrating highly accurate performance, in some cases even outperforming competitors, and simultaneously being much more resource-friendly.

Figures (6)

• Figure 1: Trade-off performance and resource costs in kWh of HEROS under different policies in terms of the mean ranks over 11 data streams and 3 random repetitions. Our $\zeta$-policy has a high predictive performance and a low resource cost while training. The critical differences, determined using the Wilcoxon signed-rank test with $95\%$ confidence level and adjusted with Holm's method, are positioned along the axis -- above for the performance metric and to the right for resources.
• Figure 2: Schematic overview of HEROS to handle the incoming data stream, involving the model selection for prediction $f(x)$ and the associated model selection for training using a policy $\pi$ within the model pool $f$.
• Figure 3: Probability density of $X_i=L(f_i)$ under $\zeta$-policy (with parameter $\zeta=0.01$ and $\epsilon=0.1$) and cand-policy, for data stream WISDM with different $k$, model pool size $M=20$ and base learner (\ref{['subfig:dist_mlp']}) MLP and (\ref{['subfig:dist_ht']}) Hoeffding tree. A Beta distribution Beta($\alpha, \beta$) is overlaid to visualize the approximation of the observed $X_i$.
• Figure 4: Distribution of average time (in s) used for model evaluation and policy execution per instance for data stream electricity, $P=50$ and $k=30$ for base learners HT and MLP. (\ref{['subfig:resource_distribution_ht']}) For cheapest and expensive, policy computation accounts for only $2-3\%$ of total costs, keeping overall resource use low due to inexpensive models. In contrast, for Cand, policy computation makes up over $32\%$ , and $\zeta$ policy under $18\%$, leading to higher total resource use. (\ref{['subfig:resource_distribution_mlp']}) With MLP base learner, policy computation is small (around $1\%$), allowing for greater resource savings.
• Figure 5: Resource consumption per training step with $M=50$, $k=30$ for base learner MLP on data stream AGR$_g$. Gradual drift is marked with vertical lines and width $\leftarrow$ drift $\rightarrow$. Resources are smoothed with a sliding window of size $300$.

Theorems & Definitions (14)

• Lemma 4.1
• Lemma 4.2
• Theorem 4.3
• Theorem 4.4
• Theorem 4.5
• Theorem 4.6
• proof : Proof of Lemma \ref{['lemma:cand']}
• proof : Proof of Lemma \ref{['lemma:zeta']}
• proof : Proof of Theorem \ref{['theorem:performance-cand']}
• proof : Proof of Theorem \ref{['theorem:ressource_cand']}