SLICE: SLO-Driven Scheduling for LLM Inference on Edge Computing Devices

TL;DR

This work tackles the problem of scheduling LLM inference on edge devices under heterogeneous SLOs, where traditional throughput-focused batching causes TPOT/TTFT violations. It introduces SLICE, a two-stage offline-online framework comprising utility-based task selection and a decode-mask matrix for per-task rate allocation, ensuring real-time tasks meet deadlines while maximizing overall SLO attainment. The approach achieves up to 35× higher SLO attainment and 3.4× faster task completion compared with Orca and FastServe on a 4060Ti-based edge platform using ChatGLM2-6B-INT4. The results highlight the practical value of SLO-aware decoding rate control for diverse edge AI workloads, enabling reliable, latency-sensitive LLM inference at the edge.

Abstract

Large Language Models (LLMs), as the foundational architecture for next-generation interactive AI applications, not only power intelligent dialogue systems but also drive the evolution of embodied intelligence on edge devices, including humanoid robots, smart vehicles, and other scenarios. The applications running on these edge devices impose differentiated Service Level Objectives (SLO) requirements on LLM services, specifically manifested as distinct constraints on Time to First Token (TTFT) and Time Per Output Token (TPOT) as well as end-to-end latency. Notably, edge devices typically handle real-time tasks that are extremely sensitive to latency, such as machine control and navigation planning. However, existing scheduling service systems still prioritize maximizing output token throughput as the sole optimization objective, failing to adequately address the diversity of SLO requirements. This ultimately results in persistently high violation rates for end-to-end latency or TPOT related SLOs. This paper proposes SLICE, an innovative scheduling solution designed for edge computing scenarios with differentiated SLO requirements. By combining a utility-maximizing request scheduling algorithm with a dynamic iterative control mechanism for generation rates, SLICE significantly improves LLM inference service SLO attainment. Experimental results demonstrate that compared to state-of-the-art solutions Orca and FastServe, SLICE achieves up to 35x higher SLO attainment and 3.4x advantage in task completion time than the other two solutions. This version is temporarily hosted anonymously for double-blind review.

Figures (11)

• Figure 1: We conducted a batch processing performance evaluation of the ChatGLM2-6B model on an NVIDIA RTX 4060 Ti 16GB GPU. Experimental results demonstrate that as the batch size increases from 1 to 9, the per-token latency shows near-linear growth while throughput improves only marginally. When batch size exceeds 9, although the per-token latency stabilizes and throughput continues to scale linearly with batch size, the absolute latency spikes above 120ms, reducing the token generation rate per task to below 10 tokens/s. This performance bottleneck will significantly impact latency-sensitive real-time tasks, making it impossible to meet strict SLO.
• Figure 2: (a) The current scheduling system cannot perform fine-grained, decode-level scheduling. Each decoding process batches all scheduled tasks together, which increases decoding latency and consequently leads to violations of task-level SLOs. (b) By adopting decode-level batch scheduling and implementing an on-demand allocation approach, more tasks can meet their SLO requirements.
• Figure 3: In LLM inference, the prefill and decode phases utilize a dynamically maintained Key-Value (KV) cache. This cache acts as a working memory, being continuously stored, loaded, and updated throughout the generation process.
• Figure 4: Rate allocation example. The rate allocation algorithm achieves differentiated rate allocation by constructing a 4×6 token generation mask matrix. The system first sorts the four tasks (task0-task3) in descending order of their decoding rate requirements (6/4/2/1 tokens/s) and builds a 4×6 binary scheduling matrix: the row index corresponds to the task ID, where the 0th row (all 1s) ensures task0 is scheduled 6 times per cycle, the 1st row (first 4 bits as 1s) enables task1 to be scheduled 4 times per cycle, the 2nd row (first 2 bits as 1s) realizes task2 being scheduled 2 times per cycle, and the 3rd row (first bit as 1) meets task3's requirement of 1 scheduling per cycle. During execution, a periodic column-wise scanning mechanism is adopted—each scheduling cycle sequentially scans columns 0 to 5, dynamically grouping tasks with a value of 1 in the current column for batch decoding (e.g., when scanning column 2, task0 and task1 corresponding to the 0th and 1st rows are grouped). This matrix-based scheduling approach not only guarantees that each task meets its SLO-required decoding rate but also enables efficient utilization of system resources.
• Figure 5: Overview of SLICE implementation. The system consists of the SLICE scheduler and the preemption controller.