InferCept: Efficient Intercept Support for Augmented Large Language Model Inference

TL;DR

InferCept tackles the inefficiency of serving augmented LLMs when external interactions interrupt decoding. By introducing min-waste interception strategies—swap pipelining, chunked recomputation, and adaptive inter-request scheduling—the framework minimizes GPU memory waste while maintaining high throughput. Empirical results across multiple models show consistent gains in throughput and reduced end-to-end latency, with substantial reductions in waste from recomputation and memory movements. The approach is implemented on top of vLLM and is designed to be modular for integration with other LLM serving systems, enabling practical deployment in real-world augmented-LM workflows.

Abstract

Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.

Figures (6)

• Figure 1: InferCept and Alternative Approaches.Vertical: GPU resources occupied or wasted by intercepted requests and other normal ones. Horizontal: timeline divided by iterations (dotted vertical lines). Hatch parts represent memory waste. InferCept adaptively combines preserve, our optimized discard, and our optimized swap. All green parts represent InferCept's techniques.
• Figure 2: End-to-end Performance on Mixed Workload.First Row: Normalized Latency. Lower right is better, i.e., sustains higher serving load. Second Row: Throughput. Expressed as completed requests per second. Higher is better. Third Row: Time-to-first-token (TTFT). Lower is better, i.e., shorter response time.
• Figure 3: InferCept Technique Breakdown.Each bar group adds one technique over its left bar group, with the leftmost being vanilla vLLM and the rightmost being the full InferCept.
• Figure 4: CDF Results of Short APIs.Each line plots the CDF distribution of all the calls of one API type.
• Figure 5: CDF Results of Long APIs.Each line plots the CDF distribution of all the calls of one API type.