DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs

TL;DR

This work tackles the challenge of maintaining answer quality and low latency in long-form dialogue by addressing context management without relying on fixed-topic segmentation. It introduces DyCP, a dynamic context pruning method that identifies query-relevant memory spans at runtime using a Kadane-inspired segmentation (KadaneDial) to preserve discourse continuity while minimizing input size. Empirical results across LoCoMo, MT-Bench+, and SCM4LLMs, using multiple closed- and open-source LLMs, show DyCP consistently improves answer quality and reduces first-token latency, with robust retrieval behavior and generalization to open-source models. The analysis reveals that high recall in retrieval is beneficial for quality and that even as LLMs expand context windows, effective context management remains crucial for practical long-context dialogue systems.

Abstract

Large Language Models (LLMs) often exhibit increased response latency and degraded answer quality as dialogue length grows, making effective context management essential. However, existing methods rely on extra LLM calls to build memory or perform offline memory construction without considering the current user utterance, which can introduce inefficiencies or disrupt conversational continuity. We introduce DyCP, a lightweight context management method that dynamically segment and retrieve relevant memory at query time. It preserves the sequential structure of dialogue without predefined topic boundaries and supports efficient, adaptive context retrieval. Across three long-form dialogue benchmarks, LoCoMo, MT-Bench+, and SCM4LLMs, and multiple LLMs, DyCP consistently improves answer quality while reducing response latency. We also examine the gap between modern LLMs' expanded context windows and their actual long-context processing capacity, highlighting the continued importance of effective context management.

Figures (11)

• Figure 1: For the current user utterance (Turn 96), which requests the RTG3 task defined earlier in the conversation, the agent generates terms like bank and a handful of coins when using the pruned context selected by our method (Turns 7–15), accurately reflecting the task intent. In contrast, using the full context(Turns 1–95) results in outputs like library and book, which are plausible but slightly off-topic for the [bank] and [money] theme. While this example uses top-1 pruning, our method generally selects multiple segments dynamically. Pruning also reduces first-token latency about three times in this example.
• Figure 2: Input context limits (in tokens) of GPT APIs, ranging from GPT-2 to GPT-4 series.
• Figure 3: Evaluation prompt used in GPT4Score
• Figure 4: Comparison of response latency and answer quality across benchmarks using GPT-4o. The x-axis indicates first-token latency in seconds (lower is better), and the y-axis shows answer quality (higher is better). Since quality scores are on different scales across benchmarks, they are normalized as z-scores within each benchmark. DyCP achieves both high quality and low latency across all datasets.
• Figure 5: Comparison of GPT4Score with human correctness labels on one LoCoMo dialogue (50 test QA pairs) across six methods. GPT4Score (0–100) is grouped by human ratings—incorrect ($-1$), partially correct ($0$), and correct ($1$).