SQUAD: Scalable Quorum Adaptive Decisions via ensemble of early exit neural networks

TL;DR

SQUAD addresses the reliability and latency issues of traditional early-exit neural networks by introducing a quorum-based inference scheme that aggregates intermediate predictions from a distributed ensemble and halts computation only upon statistically robust consensus. To ensure the ensemble is effective at every exit, QUEST performs a neural architecture search that optimizes hierarchical diversity among early exits, producing complementary learners across depths. The approach yields statistically robust improvements in accuracy (up to 5.95%) and substantial latency reductions (up to 70.60%) against static ensembles at comparable cost, demonstrated on CIFAR-10, CIFAR-100, and ImageNet-16-120. Combined, SQUAD and QUEST offer a principled, automation-friendly framework for scalable, uncertainty-aware dynamic inference with strong practical impact for edge and resource-constrained deployments.

Abstract

Early-exit neural networks have become popular for reducing inference latency by allowing intermediate predictions when sufficient confidence is achieved. However, standard approaches typically rely on single-model confidence thresholds, which are frequently unreliable due to inherent calibration issues. To address this, we introduce SQUAD (Scalable Quorum Adaptive Decisions), the first inference scheme that integrates early-exit mechanisms with distributed ensemble learning, improving uncertainty estimation while reducing the inference time. Unlike traditional methods that depend on individual confidence scores, SQUAD employs a quorum-based stopping criterion on early-exit learners by collecting intermediate predictions incrementally in order of computational complexity until a consensus is reached and halting the computation at that exit if the consensus is statistically significant. To maximize the efficacy of this voting mechanism, we also introduce QUEST (Quorum Search Technique), a Neural Architecture Search method to select early-exit learners with optimized hierarchical diversity, ensuring learners are complementary at every intermediate layer. This consensus-driven approach yields statistically robust early exits, improving the test accuracy up to 5.95% compared to state-of-the-art dynamic solutions with a comparable computational cost and reducing the inference latency up to 70.60% compared to static ensembles while maintaining a good accuracy.

Figures (6)

• Figure 1: Overview of the first stage ($e$=1) of SQUAD with $K=3$ and $k\in\{1,...,3\}$ denotes the branch index. The Exit Test incrementally collects the class confidences from the EECs denoted as $C_{1k}$, ordered by the computational cost of $B_{1k}$. If the stopping criterion of the Exit Test is satisfied, an intermediate prediction is done, and the input representations are not forwarded to the following branches. Otherwise, the output of the blocks $B_{1k}$ is propagated through the following blocks. The test result acts as a stopping signal when a decision in the Exit Test has been taken.
• Figure 2: Overview of QUEST consisting of the Arch Adapter, the Search Space, the Trainer, and the Search Strategy.
• Figure 3: Left panel: Early exit ratios by dataset. The majority of samples in simple datasets (CIFAR-10) exit early, while complex datasets (ImageNet) require deeper processing. Right panel: Branch pivot (the decisive vote) ratios by dataset. Branch b1 dominates for easy samples, while Branch b2 becomes critical for reaching consensus on hard samples.
• Figure 4: Mean vs T-student early exit criterion on CIFAR-10 with different $\tau_{conf}$
• Figure 5: Mean vs T-student early exit criterion on CIFAR-100 with different $\tau_{conf}$