Low-Regret and Low-Complexity Learning for Hierarchical Inference

TL;DR

This work introduces Hierarchical Inference (HI) for edge intelligence, where a compact Local-ML on-device collaborates with a high-accuracy Remote-ML on an edge server. By modeling the probability of correct local inference as an increasing function of the Local-ML confidence, the authors design two UCB-based policies, HI-LCB and HI-LCB-lite, to strategically offload samples and learn the misclassification likelihood and offloading cost in nonstationary environments. They prove order-optimal regret bounds of O(log T) for both policies under adversarial and stochastic arrivals, with HI-LCB-lite offering O(1) per-sample computation perfect for devices with limited resources. Numerical results on ImageNet1k, CIFAR-10, and CIFAR-100 show superior performance over state-of-the-art HI methods, confirming the practical viability of their approach. The work advances offloading decisions by exploiting a structural link between confidence and accuracy, yielding both theoretical guarantees and real-world efficiency improvements.

Abstract

This work focuses on Hierarchical Inference (HI) in edge intelligence systems, where a compact Local-ML model on an end-device works in conjunction with a high-accuracy Remote-ML model on an edge-server. HI aims to reduce latency, improve accuracy, and lower bandwidth usage by first using the Local-ML model for inference and offloading to the Remote-ML only when the local inference is likely incorrect. A critical challenge in HI is estimating the likelihood of the local inference being incorrect, especially when data distributions and offloading costs change over time -- a problem we term Hierarchical Inference Learning (HIL). We introduce a novel approach to HIL by modeling the probability of correct inference by the Local-ML as an increasing function of the model's confidence measure, a structure motivated by empirical observations but previously unexploited. We propose two policies, HI-LCB and HI-LCB-lite, based on the Upper Confidence Bound (UCB) framework. We demonstrate that both policies achieve order-optimal regret of $O(\log T)$, a significant improvement over existing HIL policies with $O(T^{2/3})$ regret guarantees. Notably, HI-LCB-lite has an $O(1)$ per-sample computational complexity, making it well-suited for deployment on devices with severe resource limitations. Simulations using real-world datasets confirm that our policies outperform existing state-of-the-art HIL methods.

Figures (6)

• Figure 1: Hierarchical Inference on an edge intelligence system comprising a compact on-device Local-ML model and an edge-server with a larger, more accurate Remote-ML model.
• Figure 2: Empirical relationship between confidence measure (max softmax) and inference accuracy for multi-class classification. Accuracy increases with the value of the confidence measure, with rare exceptions.
• Figure 3: Runtime vs. cardinality of confidence measure set ($|\Phi|$).
• Figure 4: Simulation results for ImageNet1k, CIFAR-10, and CIFAR-100. The legend for sub-figures (a), (b), and (c) is provided at the top of the page. The legend for sub-figure (d) is provided just above the sub-figure.
• Figure 5: Simulation results for CIFAR-100 using the ground-truth as the baseline.

Theorems & Definitions (45)

• Remark 2.1
• Remark 2.2
• Remark 2.3
• Remark 2.4
• Definition 3.1
• Lemma 3.1
• proof
• Remark 3.1
• Remark 3.2
• Remark 3.3