Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agents

Hyunjun Kim

TL;DR

Entropic Context Shaping (ECS) addresses context management for LLM agents by shifting focus from lexical similarity to pragmatic utility. ECS quantifies utility as the signed change in the probability of the correct answer when a candidate update is added, $\mathcal{U}(u; \mathcal{C}, a^*) = \log P_{\mathcal{C} \cup u}(a^*) - \log P_{\mathcal{C}}(a^*) - \lambda |u|$, and extends this with multi-token trajectory divergence $\mathcal{D}_T(u; \mathcal{C}) = \sum_{t=1}^{T} D_{KL}(P^{(t)}_{\mathcal{C} \cup u} \| P^{(t)}_{\mathcal{C}})$. A Red Herring Theorem shows task-irrelevant updates yield near-zero distribution shift, enabling rejection of distractors under mild assumptions. The framework supports two operational settings—ground-truth available or unavailable—with an accompanying algorithm and prefix-caching for efficiency. Empirically, ECS delivers strong gains on fine-grained turn-level evidence selection (e.g., LoCoMo with Llama-3.1-8B achieving $F1=0.265$) but underperforms at coarse session-level granularity where lexical methods excel, highlighting a model- and granularity-dependent landscape for pragmatic utility.

Abstract

Context engineering for large language model (LLM) agents requires distinguishing pragmatically useful information from misleading distractors. We introduce Entropic Context Shaping (ECS), an information-theoretic framework that measures context utility via the shift in the model's answer distribution toward the correct answer. Unlike lexical similarity methods that rely on word overlap, ECS captures pragmatic utility -- whether a passage actually helps answer the question. We formalize utility as the signed change in answer probability and provide theoretical analysis showing that task-irrelevant updates yield near-zero distribution shift. We evaluate on multi-turn context selection tasks using LongMemEval (session-level) and LoCoMo (turn-level) benchmarks. On fine-grained turn selection, ECS with Llama-3.1-8B achieves F1=0.265, a 71.83% relative improvement over TF-IDF (F1=0.154), demonstrating that pragmatic utility outperforms lexical similarity when precise context selection matters. Code and data are available in the supplementary materials.

Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agents

TL;DR

Entropic Context Shaping (ECS) addresses context management for LLM agents by shifting focus from lexical similarity to pragmatic utility. ECS quantifies utility as the signed change in the probability of the correct answer when a candidate update is added, , and extends this with multi-token trajectory divergence . A Red Herring Theorem shows task-irrelevant updates yield near-zero distribution shift, enabling rejection of distractors under mild assumptions. The framework supports two operational settings—ground-truth available or unavailable—with an accompanying algorithm and prefix-caching for efficiency. Empirically, ECS delivers strong gains on fine-grained turn-level evidence selection (e.g., LoCoMo with Llama-3.1-8B achieving ) but underperforms at coarse session-level granularity where lexical methods excel, highlighting a model- and granularity-dependent landscape for pragmatic utility.

Abstract

Context engineering for large language model (LLM) agents requires distinguishing pragmatically useful information from misleading distractors. We introduce Entropic Context Shaping (ECS), an information-theoretic framework that measures context utility via the shift in the model's answer distribution toward the correct answer. Unlike lexical similarity methods that rely on word overlap, ECS captures pragmatic utility -- whether a passage actually helps answer the question. We formalize utility as the signed change in answer probability and provide theoretical analysis showing that task-irrelevant updates yield near-zero distribution shift. We evaluate on multi-turn context selection tasks using LongMemEval (session-level) and LoCoMo (turn-level) benchmarks. On fine-grained turn selection, ECS with Llama-3.1-8B achieves F1=0.265, a 71.83% relative improvement over TF-IDF (F1=0.154), demonstrating that pragmatic utility outperforms lexical similarity when precise context selection matters. Code and data are available in the supplementary materials.
Paper Structure (58 sections, 2 theorems, 6 equations, 5 tables, 1 algorithm)

This paper contains 58 sections, 2 theorems, 6 equations, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Accumulation Fallacy.
  3. Motivating Example.
  4. Our Contribution: Entropic Context Shaping (ECS).
  5. Related Work
  6. Context Engineering for LLM Agents.
  7. Retrieval-Augmented Generation (RAG).
  8. Memory Systems for Agents.
  9. Red Herrings and Inconsequential Noise.
  10. Information-Theoretic Perspectives on LLMs.
  11. Methodology
  12. Problem Formulation
  13. The Direction Problem.
  14. Pragmatic Utility: Theoretical Framework
  15. Relationship to KL-Divergence.
  16. ...and 43 more sections

Key Result

Theorem 1

Let $u$ be task-irrelevant (i.e., $u \perp q$ given $\mathcal{C}$). Under regularity conditions on $\mathcal{M}$: for small $\epsilon > 0$ depending on model capacity.

Theorems & Definitions (6)

  • Definition 1: Pragmatic Utility
  • Theorem 1: Red Herring Rejection
  • proof : Proof Sketch
  • Definition 2: Trajectory Divergence
  • Proposition 2: Amortized Complexity
  • proof