Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition

TL;DR

This paper introduces Big-Little Net (bL-Net), a simple yet powerful multi-scale CNN architecture that uses a high-complexity Big-Branch operating at low resolution and a lightweight Little-Branch at high resolution, with frequent merging to build rich multi-scale features. By learning per-scale capacities and fusing outputs via addition, the approach achieves substantial FLOP reductions (around 30% in object recognition and up to 30% in speech) while maintaining or improving accuracy across backbones like ResNet, ResNeXt, and SEResNeXt. The authors demonstrate strong results on ImageNet and Switchboard, showing notable speedups and cross-domain generalization, and provide ablations and comparisons indicating advantages over pruning, dynamic routing, and other acceleration methods. The method is compatible with other architectures and can be combined with CNN acceleration techniques, making it practically impactful for real-time vision and speech systems.

Abstract

In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy. This is achieved by using a multi-branch network, which has different computational complexity at different branches. Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation. The proposed approach demonstrates improvement of model efficiency and performance on both object recognition and speech recognition tasks,using popular architectures including ResNet and ResNeXt. For object recognition, our approach reduces computation by 33% on object recognition while improving accuracy with 0.9%. Furthermore, our model surpasses state-of-the-art CNN acceleration approaches by a large margin in accuracy and FLOPs reduction. On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains. The codes are available at https://github.com/IBM/BigLittleNet

Figures (5)

• Figure 1: Our proposed Big-Little Net (bL-Net) for efficient multi-scale feature representations. (a) The bL-Net stacks several Big-Little Modules. A bL-module include $K$ branches ($K=2$ in this illustration) where the $k^{th}$ branch represents an image scale of $1/2^{k}$. 'M' here denotes a merging operation. (b) Our implementation of the Big-Little Module includes two branches. The Big-Branch has the same structure as the baseline model while the Little-Branch reduces the convolutional layers and feature maps by $\alpha$ and $\beta$, respectively. Larger values of $\alpha$ and $\beta$ lead to lower computational complexity in Big-Little Net.
• Figure 2: Comparison with the ResNet and ResNeXt related works.
• Figure 3: Comparison performance among other types of networks. (a) FLOPs. (b) GPU Speed.
• Figure 4: Prediction results for bL-ResNet-50 and ResNet-50-lowres. True labels, predicted labels and their probability are listed in the table. When both models predicts correctly ((a) and (b)), bL-ResNet-50 achieves much higher probability; on the other hand, bL-ResNet-50 captures the details on the object and then predicts correctly ((c) and (d)).
• Figure 5: Comparison memory requirement at the training and test phases among other types of networks. (a) Training. (b) Test. (The batch size is 8.)