Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward

TL;DR

This survey confronts the problem of efficient inference for large language models by surveying model compression and system-level optimization techniques and validating them through empirical experiments on LLaMA-/2-7B. It covers pruning, quantization, distillation, and low-rank methods, alongside system engines and deployment frameworks, highlighting tradeoffs between memory, speed, and perplexity. Key findings include the strong performance of training-free pruning approaches like FLaP, the effectiveness of 4-bit quantization across multiple engines, and the complementary benefits of system-level optimizations, while also outlining practical bottlenecks such as rank selection, quantization overhead, and evaluation metrics. The paper argues for a pragmatic, hardware-aware path to deployable, efficient LLMs and provides a public codebase to reproduce the results, setting the stage for future research in scalable, high-fidelity inference.

Abstract

Despite the impressive performance of LLMs, their widespread adoption faces challenges due to substantial computational and memory requirements during inference. Recent advancements in model compression and system-level optimization methods aim to enhance LLM inference. This survey offers an overview of these methods, emphasizing recent developments. Through experiments on LLaMA(/2)-7B, we evaluate various compression techniques, providing practical insights for efficient LLM deployment in a unified setting. The empirical analysis on LLaMA(/2)-7B highlights the effectiveness of these methods. Drawing from survey insights, we identify current limitations and discuss potential future directions to improve LLM inference efficiency. We release the codebase to reproduce the results presented in this paper at https://github.com/nyunAI/Faster-LLM-Survey