Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEE

TL;DR

QuEE addresses the high inference cost of large pretrained models by enabling per-sample adaptation that combines quantization (width reduction) and early exiting (depth reduction) in a fully post-training framework. It reframes the problem as predicting per-path error probabilities and uses lightweight gates to route each input along a cost-aware sequence of quantization levels and exits. By discretizing the feature space to generate targets for error-probability predictors and training small neural predictors that operate on compact features, QuEE achieves a flexible operating curve without backbone retraining. Across ImageNet, CIFAR, and SVHN with ViT-based backbones, QuEE generally outperforms single-technique baselines, especially at higher cost budgets, while maintaining stability and practicality for deployment on foundation-model-style architectures.

Abstract

Machine learning models can solve complex tasks but often require significant computational resources during inference. This has led to the development of various post-training computation reduction methods that tackle this issue in different ways, such as quantization which reduces the precision of weights and arithmetic operations, and dynamic networks which adapt computation to the sample at hand. In this work, we propose a more general dynamic network that can combine both quantization and early exit dynamic network: QuEE. Our algorithm can be seen as a form of soft early exiting or input-dependent compression. Rather than a binary decision between exiting or continuing, we introduce the possibility of continuing with reduced computation. This complicates the traditionally considered early exiting problem, which we solve through a principled formulation. The crucial factor of our approach is accurate prediction of the potential accuracy improvement achievable through further computation. We demonstrate the effectiveness of our method through empirical evaluation, as well as exploring the conditions for its success on 4 classification datasets.

Figures (10)

• Figure 1: Early exit decreases inference costs along the depth, while quantization decreases inference costs along the width. QuEE can integrate both, and learn to employ different computation reduction methods to classify different classes. On CIFAR-10, quantization is mainly used for automobiles and ships, while early exiting is mainly used for dogs and frogs.
• Figure 2: Depiction of how QuEE predicts the next step at inference. $h_{\theta}$ is evaluated for each considered paths (2 paths shown in grey were not sampled), then the algorithm takes a step towards the path $\pi^*$ minimizing the predicted loss $\ell_{01c}$. In this example, at gate $j$$\pi^2$ has been identified as the best path.
• Figure 3: Accuracy vs BitFLOPS. A) Imagenet t2t-Vit-14, B) Imagenet Vit-14 with DinoV2, C) EE vs quantization performance, D) CIFAR100 t2t-Vit-14, E) CIFAR10 t2t-Vit-7, F) SVHN t2t-Vit-7.
• Figure 4: Accuracy vs cost curves of predictors with varying RMSE performance on the SVHN dataset.
• Figure 5: A) Predicted probability of the ground truth label and entropy of a few clusters for $K=20$ on the SVHN datasets. We can see that there is a correlation between the clustering and those two metrics. B) Expected calibration error of $\tilde{pe}$ generated from varying cluster numbers $K$ for the SVHN dataset on T2T-ViT-7 with a fitted second order polynomial. Increasing the number of clusters (up to 50) reduces the calibration error. We discuss this further in Appendix \ref{['app:cluster']}C) Accuracy-cost trade-off for different values of $K$ for the SVHN dataset on T2T-ViT-7.