Tiny Models are the Computational Saver for Large Models

TL;DR

The paper tackles the rising compute demands of large vision models by introducing TinySaver, a dynamic compression framework that uses pre-trained tiny models as independent savers at the first exit. This approach decouples from backbone design, enabling model-agnostic deployment and straightforward extension with an Exit Sequence Network (ESN) to blend saver signals with backbone features. Through theoretical analysis and large-scale experiments on ImageNet-1k (and COCO), TinySaver achieves substantial compute reductions (up to ~90% FLOPs) with negligible accuracy degradation and demonstrates favorable comparisons to EE and MoE baselines. The work provides a practical, flexible path toward scalable AI with reduced energy use and faster inference, while maintaining competitive performance across diverse architectures. It also introduces a principled saver-selection metric, $ riangle C_{tr}$, and a concrete ESN framework to preserve advantages of traditional EE methods when desired.

Abstract

This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Distinct from traditional compression techniques, dynamic methods like TinySaver can leverage the difficulty differences to allow certain inputs to complete their inference processes early, thereby conserving computational resources. Most existing early exit designs are implemented by attaching additional network branches to the model's backbone. Our study, however, reveals that completely independent tiny models can replace a substantial portion of the larger models' job with minimal impact on performance. Employing them as the first exit can remarkably enhance computational efficiency. By searching and employing the most appropriate tiny model as the computational saver for a given large model, the proposed approaches work as a novel and generic method to model compression. This finding will help the research community in exploring new compression methods to address the escalating computational demands posed by rapidly evolving AI models. Our evaluation of this approach in ImageNet-1k classification demonstrates its potential to reduce the number of compute operations by up to 90\%, with only negligible losses in performance, across various modern vision models.

Figures (5)

• Figure 1: Illustration of TinySaver with the comparison to related concept of Early Exit and Mixture of Experts.
• Figure 2: TinySaver with the Exit Sequence Network
• Figure 3: Comparison of the top-1 accuracy on ImageNet-1k between TinySaver enabled Models and related work
• Figure 4: Accuracy vs early exit ratio on the ImageNet-1k training/validation set. The intersection of solid lines denotes the ratio where the saver and base model have the equivalent accuracy on the validation set. The intersection of dashed lines are for the training set. Every plot is labelled as Base - Saver.
• Figure 5: Detailed ESN