Boosting Deep Ensembles with Learning Rate Tuning

TL;DR

This paper presents a novel framework, LREnsemble, to effectively leverage effective learning rate tuning to boost deep ensemble performance, and proposes LREnsemble, a framework that utilizes the synergy of LR tuning and deep ensemble techniques to enhance deep learning performance.

Abstract

The Learning Rate (LR) has a high impact on deep learning training performance. A common practice is to train a Deep Neural Network (DNN) multiple times with different LR policies to find the optimal LR policy, which has been widely recognized as a daunting and costly task. Moreover, multiple times of DNN training has not been effectively utilized. In practice, often only the optimal LR is adopted, which misses the opportunities to further enhance the overall accuracy of the deep learning system and results in a huge waste of both computing resources and training time. This paper presents a novel framework, LREnsemble, to effectively leverage effective learning rate tuning to boost deep ensemble performance. We make three original contributions. First, we show that the LR tuning with different LR policies can produce highly diverse DNNs, which can be supplied as base models for deep ensembles. Second, we leverage different ensemble selection algorithms to identify high-quality deep ensembles from the large pool of base models with significant accuracy improvements over the best single base model. Third, we propose LREnsemble, a framework that utilizes the synergy of LR tuning and deep ensemble techniques to enhance deep learning performance. The experiments on multiple benchmark datasets have demonstrated the effectiveness of LREnsemble, generating up to 2.34% accuracy improvements over well-optimized baselines.

Figures (7)

• Figure 1: Visualization of deep learning training paths with different LR policies: different LR policies can lead to different optimization trajectories and produce diverse deep learning models.
• Figure 2: Accuracy improvements by LREnsemble (WRN-28-10 on CIFAR-100): LREnsemble can leverage Learning Rate (LR) tuning to generate diverse individual models (blue or gray bars in Figure \ref{['fig:first_page_cifar100_lrensemble']}) and select the complementary member models (blue bars: selected models) to boost ensemble accuracy (red bar: LREnsemble with over 1.74% accuracy improvements). Figure \ref{['fig:first_page_cifar100_lrpolicy']} shows the accuracy of individual models trained using different LR policies with different initial LR values using our LREnsemble framework.
• Figure 3: Overview of LREnsemble architecture
• Figure 4: Performance distribution across all ensemble team sizes in different tasks.
• Figure 5: Two image examples to illustrate the voting processes in LREnsemble for training WRN-28-10 on CIFAR-10. The ensemble used here is from Table \ref{['table:ensemble-comparison-cifar10-cifar100']} with the team size of 4. All four member models are listed in Table \ref{['table:lr-comparison']}, which are ordered from model 1 to model 4: MultiStepLR with $k_0$ = 0.1, OneCycleLR with $k_0$ = 0.2, WarmupCosineAnnealing with $k_0$ = 0.1, and WarmupCosineAnnealing with $k_0$ = 0.05.