EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse

TL;DR

EFIM identifies a bottleneck in infilling with large language models: traditional FIM prompts cause KV cache invalidation as the insertion region grows. It then presents two core innovations: a prompt transformation (EFIM) that relocates the incremental content to the end to maximize cross-request KV cache reuse, and a fragment tokenization training method that enables universal subtoken generation without architectural changes. Empirical results on two representative LLMs show EFIM achieves substantial speedups (52% latency reduction, 98% throughput gain) while preserving infilling capability, though subtokens require the fragment-tokenization augmentation to be universally reliable. The work also demonstrates favorable cost efficiency, suggesting practical viability for scalable LLM inference in infilling services. Overall, EFIM provides a pragmatic pathway to faster, scalable infilling serving by rethinking prompt structure and training data processing toward better KV-cache utilization and subtoken generation capability.

Abstract

Large language models (LLMs) are often used for infilling tasks, which involve predicting or generating missing information in a given text. These tasks typically require multiple interactions with similar context. To reduce the computation of repeated historical tokens, cross-request key-value (KV) cache reuse, a technique that stores and reuses intermediate computations, has become a crucial method in multi-round interactive services. However, in infilling tasks, the KV cache reuse is often hindered by the structure of the prompt format, which typically consists of a prefix and suffix relative to the insertion point. Specifically, the KV cache of the prefix or suffix part is frequently invalidated as the other part (suffix or prefix) is incrementally generated. To address the issue, we propose EFIM, a transformed prompt format of FIM to unleash the performance potential of KV cache reuse. Although the transformed prompt can solve the inefficiency, it exposes subtoken generation problems in current LLMs, where they have difficulty generating partial words accurately. Therefore, we introduce a fragment tokenization training method which splits text into multiple fragments before tokenization during data processing. Experiments on two representative LLMs show that LLM serving with EFIM can lower the latency by 52% and improve the throughput by 98% while maintaining the original infilling capability. EFIM's source code is publicly available at https://github.com/gty111/EFIM.

Figures (12)

• Figure 1: A python code snippet where a programmer wants to insert code inside a function. The prefix/suffix part represents content before/after the insertion point. The middle part is the content expected to infill.
• Figure 2: Comparison of PSM and SPM. $<$P$>$ follows prefix part, $<$S$>$ follows suffix part, $<$M$>$ follows middle part and $<$E$>$ marks the end of infilling span.
• Figure 3: Reusable part between PSM, SPM and EFIM when the growth ($inc$) happens either at the prefix tail or at the suffix head. The reusable part includes the content before $inc$.
• Figure 4: Elapsed time breakdown of prefill and decode stage for infilling serving (average input/output length is 2100/32) between FIM and EFIM.
• Figure 5: Subtoken generation ability between different prompt formats considering prompt "code comp[] models".