SneakPeek: Data-Aware Model Selection and Scheduling for Inference Serving on the Edge

TL;DR

This work tackles efficient inference serving at the edge when hardware elasticity is unavailable and multiple model variants must be selected per request. It introduces SneakPeek, a data-aware mechanism to sharpen per-model accuracy estimates using real-time data, integrated with a priority-based and grouped scheduling framework that also supports short-circuit inference. The proposed approach decomposes scheduling into request ordering and locally-optimal model selection, enabling inference batching and reduced GPU swaps, with strong empirical gains across three real-world applications. Results show up to a twofold improvement in a combined utility metric (accuracy subject to deadline penalties) and substantially fewer deadline violations, highlighting practical benefits for latency-sensitive, multi-application edge deployments.

Abstract

Modern applications increasingly rely on inference serving systems to provide low-latency insights with a diverse set of machine learning models. Existing systems often utilize resource elasticity to scale with demand. However, many applications cannot rely on hardware scaling when deployed at the edge or other resource-constrained environments. In this work, we propose a model selection and scheduling algorithm that implements accuracy scaling to increase efficiency for these more constrained deployments. We show that existing schedulers that make decisions using profiled model accuracy are biased toward the label distribution present in the test dataset. To address this problem, we propose using ML models -- which we call SneakPeek models -- to dynamically adjust estimates of model accuracy, based on the underlying data. Furthermore, we greedily incorporate inference batching into scheduling decisions to improve throughput and avoid the overhead of swapping models in and out of GPU memory. Our approach employs a new notion of request priority, which navigates the trade-off between attaining high accuracy and satisfying deadlines. Using data and models from three real-world applications, we show that our proposed approaches result in higher-utility schedules and higher accuracy inferences in these hardware-constrained environments.

Figures (15)

• Figure 1: SneakPeek system model for inference serving
• Figure 2: SneakPeek model updating for fall detection
• Figure 3: Illustration of request priority.
• Figure 4: Groups are split based on the computed SneakPeek probabilities
• Figure 5: Comparison of schedule utility across approaches