InfiniPot: Infinite Context Processing on Memory-Constrained LLMs

TL;DR

InfiniPot enables infinite-context processing on memory-constrained LLMs by introducing Continual Context Distillation (CCD), a memory-pot style KV-cache management that compresses incoming context to a fixed size using Catalyst Prompt (CaP) for representative importance and Novelty under Compression (NuC) for past novelty, complemented by Context-Reset Rotary Position Embedding (CR-RoPE). By iteratively distilling context and balancing CaP and NuC signals, InfiniPot maintains critical information within a limited KV-cache, enabling long-context tasks without retraining. Empirical results on LongBench and NIH show InfiniPot achieving competitive or superior performance relative to memory-unconstrained baselines and outperforming several memory-constrained approaches, while maintaining efficiency in memory usage and latency. This work extends the practical applicability of pre-trained LLMs to edge and mobile settings, where strict memory budgets previously limited context length, opening avenues for on-device tasks such as document understanding and multi-hop reasoning at unprecedented scales.

Abstract

Handling long input contexts remains a significant challenge for Large Language Models (LLMs), particularly in resource-constrained environments such as mobile devices. Our work aims to address this limitation by introducing InfiniPot, a novel KV cache control framework designed to enable pre-trained LLMs to manage extensive sequences within fixed memory constraints efficiently, without requiring additional training. InfiniPot leverages Continual Context Distillation (CCD), an iterative process that compresses and retains essential information through novel importance metrics, effectively maintaining critical data even without access to future context. Our comprehensive evaluations indicate that InfiniPot significantly outperforms models trained for long contexts in various NLP tasks, establishing its efficacy and versatility. This work represents a substantial advancement toward making LLMs applicable to a broader range of real-world scenarios.

Figures (7)

• Figure 1: Illustration of KV-cache control methods in long context scenario. (a) Previous memory-unconstrained methods (denoted as SnapKV) showing full context processing. (b) Our memory-constrained KV-cache control using the proposed CCD, where only a limited length of context fits within the memory pot regardless of the total context length. (c) Proposed token importance scoring from perspectives of past and future contexts. Numbers inside the boxes indicate positional indices.
• Figure 2: Top: Hit rate between Continual Context Distillation (CCD) with Catalyst Prompt general (CaP-G) and question (CaP-Q), Bottom: Selected token frequency in memory pot per attention head (left: 1st layer, right: 30th layer) Mistral-inst-v0.3-4K used with HotpotQA task.
• Figure 3: Top: Comparison of hit rates between CaP and CaP w/ NuP across the CCD cycle. Bottom: Comparison of the summation of selected token's entropy across various CCD configurations (w/ CaP, w/ CaP + NuC). This includes a global token entropy comparison, summing globally selected tokens entropy.
• Figure 4: Accuracy comparison on the Needle in a haystack (NIH) benchmark at varying context lengths from 4K to 1M. InfiniPot-integrated models (Ours) show superior scalability and maintain high accuracy even at extremely long contexts.
• Figure 5: Retrieval accuracy of Mistral-7B-v0.3-4K for the Needle in a Haystack (NIH) passkey task across varying context lengths from 4K to 1M. The task involved hiding a passkey at different depths (start, middle, end corresponding to depths 0.1, 0.5, 0.9) and measuring retrieval accuracy as the context length increased.