Enhancing Adaptive Deep Networks for Image Classification via Uncertainty-aware Decision Fusion

TL;DR

The paper targets adaptive deep networks with multiple exits by revealing that the last exit is not universally optimal for all classes. It introduces Collaborative Decision Making (CDM), which uses evidential deep learning to quantify per-classifier uncertainty $u^c$ and fuses decisions from earlier heads via an uncertainty-aware mechanism, with a balance term to prevent fusion saturation/unfairness. To further boost performance, Guided Collaborative Decision Making (GCDM) regularizes training by aligning early classifiers with the final one using Jensen–Shannon divergence and temperature scaling, improving both accuracy and diversity. Across MSDNet, RANet, and IMTA on ImageNet and CIFAR datasets, CDM/GCDM yield consistent gains under anytime and budgeted resource constraints with negligible overhead, demonstrating practical benefits for resource-adaptive inference in large-scale vision tasks.

Abstract

Handling varying computational resources is a critical issue in modern AI applications. Adaptive deep networks, featuring the dynamic employment of multiple classifier heads among different layers, have been proposed to address classification tasks under varying computing resources. Existing approaches typically utilize the last classifier supported by the available resources for inference, as they believe that the last classifier always performs better across all classes. However, our findings indicate that earlier classifier heads can outperform the last head for certain classes. Based on this observation, we introduce the Collaborative Decision Making (CDM) module, which fuses the multiple classifier heads to enhance the inference performance of adaptive deep networks. CDM incorporates an uncertainty-aware fusion method based on evidential deep learning (EDL), that utilizes the reliability (uncertainty values) from the first c-1 classifiers to improve the c-th classifier' accuracy. We also design a balance term that reduces fusion saturation and unfairness issues caused by EDL constraints to improve the fusion quality of CDM. Finally, a regularized training strategy that uses the last classifier to guide the learning process of early classifiers is proposed to further enhance the CDM module's effect, called the Guided Collaborative Decision Making (GCDM) framework. The experimental evaluation demonstrates the effectiveness of our approaches. Results on ImageNet datasets show CDM and GCDM obtain 0.4% to 2.8% accuracy improvement (under varying computing resources) on popular adaptive networks. The code is available at the link https://github.com/Meteor-Stars/GCDM_AdaptiveNet.

Figures (17)

• Figure 1: Motivation analysis: (a) Accuracy of different classifiers of MSDNet on randomly sampled classes with CIFAR100 dataset. (b) Agreement measurement on 10 classifiers of MSDNet on ImageNet100 with regularized training. A lower value represents higher diversity. The values in bold denote that the agreement value decreases after regularized training.
• Figure 2: Comparison between anytime prediciton and budgeted batch prediction settings.
• Figure 3: Overview of our methods for adaptive deep network. Our proposed CDM fusion is suitable for both anytime prediction and budgeted batch prediction settings as shown in Figure \ref{['fig:task_ana']}.
• Figure 4: Accuracy (top-1) of budgeted batch prediction on ImageNet100 and ImageNet1000. With the same computational resources, existing methods equipped with the proposed GCDM can achieve better performance.
• Figure 5: Accuracy (top-1) of budgeted batch prediction on CIFAR10 and CIFAR100.