SLIDE: Simultaneous Model Downloading and Inference at the Wireless Network Edge

TL;DR

SLIDE addresses high end-to-end latency in on-device inference caused by large model downloads by overlapping model provisioning with inference. It formulates a joint optimization of model provisioning $x_{k,i}$, bandwidth allocation $y_k$, and per-layer GPU allocations $z_{k,l_i}$ to maximize task throughput under latency deadlines $\bar{T}_k$ and energy budgets $Q_k$, with end-to-end latency defined recursively as $t_{k,L_i}= \max\left\{\sum_{l_i=1}^{L_i}\tau_{k,l_i}, t_{k,L_i-1}\right\}+T_{k,i}(z_{k,L_i})$. The problem is transformed to a bandwidth-allocation problem $\mathcal{P}3$ and solved in polynomial time via a two-stage algorithm, with a greedy procedure achieving the optimum. Numerical results show SLIDE markedly improves served-user throughput and robustness to mobility, device heterogeneity, and mixed-precision model libraries, compared to conventional DAI. The approach enables real-time edge AI under constrained spectrum while accommodating diverse architectures and energy budgets.

Abstract

To support on-device inference, the next-generation mobile networks are expected to support real-time model downloading services to mobile users. However, powerful AI models typically have large model sizes, resulting in excessive end-to-end (E2E) downloading-and-inference (DAI) latency. To address this issue, we propose a simultaneous model downloading and inference (SLIDE) framework, which allows users to perform inference with downloaded layers while simultaneously receiving the remaining layers of the model. To this end, we formulate a task throughput maximization problem by jointly optimizing model provisioning, spectrum bandwidth allocation, and computing resource allocation for multi-user downlink systems. Unlike traditional DAI frameworks, SLIDE introduces recursive dependencies across layers, where inference latency depends recursively on the downloading bandwidth and computing resource allocation for each of the preceding layers. To solve this challenging problem, we design an efficient algorithm that acquires the optimal solution with polynomial-time complexity. Simulation results demonstrate that the proposed SLIDE framework significantly improves task throughput under latency and communication resource constraints compared with the conventional model downloading schemes.

Figures (10)

• Figure 1: The E2E latency in wireless networks, where an end user downloads an AI model from a BS to perform local inference. The E2E latency comprises downloading and inference latency. We assume the BS downloads models with a transmit power spectral density of -29 dBm/Hz 3gpp.38.104 over a 30-MHz channel, and the distance is set to 100 m. The inference is executed on a Jetson Orin Nano with a GPU frequency of 624.75 MHz, using the CIFAR-10 dataset krizhevsky2009learning, a batch size of 1, and models from the ResNet family he2016deep.
• Figure 2: The proposed SLIDE framework, where users start inference with downloaded layers while simultaneously receiving the remaining layers.
• Figure 3: The procedures of conventional DAI and the proposed SLIDE framework, assuming that user $k$ is provisioned with model $i$. In conventional DAI, user $k$ begins inference only after downloading the entire model. In contrast, SLIDE enables user $k$ to start inference for each layer as soon as the layer is downloaded and the inference of the previous layer is completed.
• Figure 4: Experimental hardware system with an edge server (functioning as a BS) and multiple edge devices, including Jetson Orin Nano with 4 GB RAM and Jetson Orin NX with 16 GB RAM.
• Figure 5: Served user ratio of SLIDE, evaluated on the Jetson Orin Nano and Jetson Orin NX running at GPU frequencies of 624.75 MHz and 918 MHz, respectively. The default values of $B$, $K$, $\bar{T}_{k}$, $\theta$, and $\beta_{k}$, are set to 400 MHz, 80, 800 ms, 60%, and 26%, respectively.

Theorems & Definitions (20)

• Remark 1
• Remark 2
• Remark 3
• Proposition 1
• proof
• Proposition 2
• proof
• Proposition 3
• proof
• Corollary 1