Mitigating Conversational Inertia in Multi-Turn Agents

TL;DR

This work identifies conversational inertia as a critical bottleneck in multi-turn LLM-powered agents, driven by diagonal attention to previous assistant outputs that biases decisions and hampers exploration. It couples Context Preference Learning with a clip-style inference context management strategy to mitigate inertia without requiring environment rewards. Empirical results across eight environments and a deep-research scenario show that reducing inertia via CPL and periodic context clipping yields consistent performance gains and substantial computational efficiency, with diagonal attention reductions of over 10% and average improvements around 4%. The methods preserve general capabilities and offer practical guidance for deploying more robust, exploration-friendly agents in long-horizon tasks while maintaining efficiency.

Abstract

Large language models excel as few-shot learners when provided with appropriate demonstrations, yet this strength becomes problematic in multiturn agent scenarios, where LLMs erroneously mimic their own previous responses as few-shot examples. Through attention analysis, we identify conversational inertia, a phenomenon where models exhibit strong diagonal attention to previous responses, which is associated with imitation bias that constrains exploration. This reveals a tension when transforming few-shot LLMs into agents: longer context enriches environmental feedback for exploitation, yet also amplifies conversational inertia that undermines exploration. Our key insight is that for identical states, actions generated with longer contexts exhibit stronger inertia than those with shorter contexts, enabling construction of preference pairs without environment rewards. Based on this, we propose Context Preference Learning to calibrate model preferences to favor low-inertia responses over highinertia ones. We further provide context management strategies at inference time to balance exploration and exploitation. Experimental results across eight agentic environments and one deep research scenario validate that our framework reduces conversational inertia and achieves performance improvements.

Figures (20)

• Figure 1: Attention visualization in maze environment showing conversational inertia. Despite generating diverse responses, the model exhibits strong diagonal attention to previous assistant outputs, revealing token-to-token correspondence associated with conversational inertia. For other environments, see Appendix \ref{['app:heatmaps']}.
• Figure 2: Quantitative analysis of each part of attention as context increases, revealing monotonic growth in attention to previous assistant responses while attention allocation to user inputs remains stable. Detailed analysis is presented in Section \ref{['subsec:attention_ratio']}.
• Figure 3: Left: Context Preference Learning with preference data collection and DPO rafailov2023direct training. Our method does not need environment rewards ($r_5$) to generate preference pairs. Right: Clip context applied during inference context management.
• Figure 4: Three context control methods and their impact on attention scope. (a) Long Context retains complete context but suffers from conversational inertia and theoretically has context capacity limit problems. (b) Window Context maintains recent context but cannot leverage KV cache due to shifting boundaries. (c) Clip Context (our method) periodically clears context while enabling KV cache optimization.
• Figure 5: Diagonal Attention ratio analysis across different context configurations. Lower ratios indicate reduced conversational inertia.