Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading

TL;DR

InferSave addresses the challenge of cost-efficient cloud serving for large language models by jointly optimizing VM selection and KV Cache Offloading under Service Level Objectives. It introduces a four-stage framework, including memory-aware candidate identification, performance-cost prediction with a Compute Time Calibration Function (CTCF), and SLO-based optimization to select the most cost-efficient instance. The approach demonstrates significant online and offline cost savings on AWS GPUs while maintaining target TPS, and shows that KV offloading is particularly beneficial for memory-constrained or batch-heavy workloads. The framework is implemented in Python on top of FlexGen, validated with online/offline workloads, and designed for easy extension to broader cloud environments. Overall, InferSave provides a practical, automated solution to balance cost and performance in LLM inference services.

Abstract

LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads.

Figures (6)

• Figure 1: Analysis of KV Cache size growth across different models in response to increasing batch sizes.
• Figure 2: Comparison of cost efficiency per GPU instance based on SLO constraints and batch size, based on experimental results using the OPT-2.7B model on an AWS g4dn.xlarge instance with an input length of 512 tokens and an output length of 128 tokens.
• Figure 3: Comparison of FLOPS provided by the GPU manufacturer (NVIDIA) and the actual FLOPS utilized when calculating Prefill time on AWS GPU VMs. The results present TFLOPS measurements for three different GPU VMs using the OPT-2.7B model with an input size of 512 tokens and an output size of 128 tokens as batch size grows.
• Figure 4: CTCF accuracy analysis. The results illustrate the predicted time (blue), actual time (red), and CTCF-adjusted values (green) for Prefill and Decode times as batch size increases on two different GPU VMs. Additionally, the Error Rate between the CTCF-adjusted time and actual time is presented.
• Figure 5: Comparison of average TPS and cost for different InferSave configurations and the baseline configuration under varying SLO constraints for online inference workloads (Left: Average TPS, Right: Cost).