HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network

TL;DR

HALO tackles efficient distributed LLM inference on lossy edge networks by assigning less critical neuron groups to unreliable devices, enabling relaxed synchronization through semantic-aware prediction. It integrates three mechanisms—semantic-aware predictors (SAPs), an overlap-based generation scheme, and a PLR-aware load-balancing scheduler—to maximize resource utilization under packet loss. Evaluations on a Raspberry Pi cluster with LLaMA-series models show HALO achieving up to 3.41x end-to-end speedup while preserving accuracy across tasks and PLR settings, outperforming prior approaches in lossy conditions. This work demonstrates practical approaches for privacy-preserving, low-latency edge AI in heterogeneous and unreliable networks.

Abstract

The deployment of large language models' (LLMs) inference at the edge can facilitate prompt service responsiveness while protecting user privacy. However, it is critically challenged by the resource constraints of a single edge node. Distributed inference has emerged to aggregate and leverage computational resources across multiple devices. Yet, existing methods typically require strict synchronization, which is often infeasible due to the unreliable network conditions. In this paper, we propose HALO, a novel framework that can boost the distributed LLM inference in lossy edge network. The core idea is to enable a relaxed yet effective synchronization by strategically allocating less critical neuron groups to unstable devices, thus avoiding the excessive waiting time incurred by delayed packets. HALO introduces three key mechanisms: (1) a semantic-aware predictor to assess the significance of neuron groups prior to activation. (2) a parallel execution scheme of neuron group loading during the model inference. (3) a load-balancing scheduler that efficiently orchestrates multiple devices with heterogeneous resources. Experimental results from a Raspberry Pi cluster demonstrate that HALO achieves a 3.41x end-to-end speedup for LLaMA-series LLMs under unreliable network conditions. It maintains performance comparable to optimal conditions and significantly outperforms the state-of-the-art in various scenarios.

Figures (9)

• Figure 1: Smart home scenario empowered by HALO: trusted home devices collaboratively serve LLM inference with high accuracy and efficiency in lossy network.
• Figure 2: Activation Distribution of Neuron Groups.
• Figure 3: Time Cost of Different Scheduling Strategies.
• Figure 4: Overview of the System Design. Three phases are executed in sequence. Four edge devices collaboratively inference one LLM. In the Assignment phase, their four colors and the number of filled squares indicate the assigned workload. $A()$ refers to the attention computation in the MHA block and activation function computation in the MLP block.
• Figure 5: Generation Phase with Overlapped Processes. Load&Comp and Pred&Comm are overlapped across layers during runtime. Four processes have four colors, and the block lines, containing detailed illustrations, are in the same color. For illustration simplicity, we ignore the LM Head, which is essentially similar to the MLP block.