FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework

TL;DR

FastCache tackles the latency and memory challenges of KV-cache compression in multimodal inference by introducing two core innovations: a dynamic multi-stage batching framework and a KV-cache memory pool that eliminates memory fragmentation. The system includes a modality-aware compressor trained via self-supervised learning, and a dynamic scheduler that adapts batch sizes across prefill, compression, and decode stages based on real-time resource availability. Empirical results on GQA and MileBench show up to $19.3\times$ TTFT reduction, up to $12.1\times$ throughput gain, and a $20\%$ reduction in average memory usage, while maintaining stable TPOT and robust performance under high concurrency (up to 40 req/s). These improvements demonstrate FastCache as a practical, scalable solution for real-world KV-cache compressed serving, enabling responsive multimodal LLM applications with tighter memory footprints.

Abstract

Multi-modal Large Language Models (MLLMs) serving systems commonly employ KV-cache compression to reduce memory footprint. However, existing compression methods introduce significant processing overhead and queuing delays, particularly in concurrent serving scenarios. We present \texttt{FastCache}, a novel serving framework that effectively addresses these challenges through two key innovations: (1) a dynamic batching strategy that optimizes request scheduling across prefill, compression, and decode stages, and (2) an efficient KV-cache memory pool mechanism that eliminates memory fragmentation while maintaining high GPU utilization. Our comprehensive experiments on the GQA and MileBench datasets demonstrate that \texttt{FastCache} achieves up to 19.3$\times$ reduction in Time-To-First-Token (TTFT) and 12.1$\times$ improvement in throughput compared to state-of-the-art baselines. The system maintains stable performance under high-concurrency scenarios (up to 40 req/s) while reducing average memory consumption by 20\%. These results establish \texttt{FastCache} as an efficient solution for real-world LLM serving systems with KV-cache compression.

Figures (14)

• Figure 1: Performance comparison of the state-of-the-art KV-cache compress methods li2024snapkvdevoto2024simpleeffectivekvpress2024 for LLaVA-1.5-7B under real-time serving system.
• Figure 2: Illustration of KV-cache transmission in different serving scenarios. Left: Traditional serving pipeline with prefill (P) and decode (D) stages; Right: Serving pipeline with compression, where a compression stage (C) is introduced between prefill and decode stages, requiring careful design to balance memory savings and computational overhead.
• Figure 3: Processing time breakdown showing three stages (decode, compression, and prefill) under different scenarios. Left: Time distribution with varying request rates (4-10 req/s); Right: Time distribution across different batch sizes (1-16), where the maximum input_length = 128, and we chose the advanced kv-compress method kvpress2024.
• Figure 4: Comparison of effective time utilization ratios between vanilla sequential processing (left) and decoupled prefill-decoding (right) systems across different request rates with KV-cache compression enabled.
• Figure 5: Latency breakdown for the SnapKV compression method within a prefill-decode separated architecture.