A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency

TL;DR

This survey addresses the urgent need for efficient, scalable LLM inference by evaluating 25 open-source and commercial engines through a framework-centric lens. It maps a broad set of optimization techniques—batching, caching, quantization, and attention and kernel-level accelerations—to the capabilities of each engine, while contrasting edge and server deployments and incorporating hardware heterogeneity. The work provides empirical insights from server and edge benchmarking under OpenAI-compatible interfaces, highlighting throughput-latency trade-offs, and identifies practical criteria for engine selection and deployment. It also outlines future directions, including long-context memory management, complex reasoning support, security, multimodal inference, and cloud orchestration, to guide researchers and practitioners in building optimized LLM serving infrastructures.

Abstract

Large language models (LLMs) are widely applied in chatbots, code generators, and search engines. Workload such as chain-of-throught, complex reasoning, agent services significantly increase the inference cost by invoke the model repeatedly. Optimization methods such as parallelism, compression, and caching have been adopted to reduce costs, but the diverse service requirements make it hard to select the right method. Recently, specialized LLM inference engines have emerged as a key component for integrating the optimization methods into service-oriented infrastructures. However, a systematic study on inference engines is still lacking.This paper provides a comprehensive evaluation of 25 open-source and commercial inference engines. We examine each inference engine in terms of ease-of-use, ease-of-deployment, general-purpose support, scalability, and suitability for throughput- and latency-aware computation. Furthermore, we explore the design goals of each inference engine by investigating the optimization techniques it supports. In addition, we assess the ecosystem maturity of open source inference engines and handle the performance and cost policy of commercial solutions.We outline future research directions that include support for complex LLM-based services, support of various hardware, and enhanced security, offering practical guidance to researchers and developers in selecting and designing optimized LLM inference engines. We also provide a public repository to continually track developments in this fast-evolving field: \href{https://github.com/sihyeong/Awesome-LLM-Inference-Engine}{https://github.com/sihyeong/Awesome-LLM-Inference-Engine}.

Figures (47)

• Figure 1: Taxonomy of LLM Inference Engines and Optimizations
• Figure 2: Overview of Decoder-only Transformer Architecture
• Figure 3: Attention Mechanism
• Figure 4: LLM Inference Process
• Figure 5: Inference and Serving Process of LLM