FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

TL;DR

FlexInfer tackles the memory bottleneck of on-device LLM inference by introducing a triad of techniques—asynchronous prefetching, balanced memory locking, and flexible tensor preservation—to enable memory-efficient, budget-driven offloading. The framework overlaps I/O with computation, distributes memory more evenly across layers, and selectively retains tensor groups based on memory budgets, achieving substantial throughput gains (up to 12.5x) over traditional mmap-based offloading. The approach is implemented as an extension to llama.cpp and validated across multiple Llama2 and Codellama models, with detailed ablations showing the contribution of each component. This work enables running large LLMs on resource-constrained devices, enhancing privacy, customization, and autonomous operation in edge settings.

Abstract

Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within user-specified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices.

Figures (5)

• Figure 1: FlexInfer architecture.
• Figure 2: Asynchronous prefetching.
• Figure 3: Balanced Memory Locking.
• Figure 4: Evaluation result.
• Figure 5: Ablation study for flexible tensor preservation.