Grounding Agent Memory in Contextual Intent

TL;DR

The paper tackles memory for long-horizon, goal-driven agents by introducing STITCH, a memory system that grounds recall via contextual intent structured as thematic scope, event type, and key entity types. STITCH constructs online cues and disambiguates memory through coreference-aware structural alignment, then retrieves past snippets using intent-compatible filters and label-density ranking. The authors validate STITCH on a new Context-Aware Agent Memory Evaluation Benchmark (CAME-Bench) and LongMemEval, showing state-of-the-art performance and particularly strong gains as trajectories lengthen; ablations show thematic scope and coreference resolution as critical factors. This approach offers improved robustness for robust long-horizon reasoning and better interpretability by grounding memory in explicit intents, with practical impact for multi-domain, goal-directed AI systems.

Abstract

Deploying large language models in long-horizon, goal-oriented interactions remains challenging because similar entities and facts recur under different latent goals and constraints, causing memory systems to retrieve context-mismatched evidence. We propose STITCH (Structured Intent Tracking in Contextual History), an agentic memory system that indexes each trajectory step with a structured retrieval cue, contextual intent, and retrieves history by matching the current step's intent. Contextual intent provides compact signals that disambiguate repeated mentions and reduce interference: (1) the current latent goal defining a thematic segment, (2) the action type, and (3) the salient entity types anchoring which attributes matter. During inference, STITCH filters and prioritizes memory snippets by intent compatibility, suppressing semantically similar but context-incompatible history. For evaluation, we introduce CAME-Bench, a benchmark for context-aware retrieval in realistic, dynamic, goal-oriented trajectories. Across CAME-Bench and LongMemEval, STITCH achieves state-of-the-art performance, outperforming the strongest baseline by 35.6%, with the largest gains as trajectory length increases. Our analysis shows that intent indexing substantially reduces retrieval noise, supporting intent-aware memory for robust long-horizon reasoning.

Figures (15)

• Figure 1: The Challenges of Long-Horizon Agentic Memory. We identify four capabilities required for robust agentic memory: (A) Incremental Memory Revision (tracking state changes over time); (B) Context-Aware Factual Recall (distinguishing semantically similar facts by context); (C) Context-Aware Multi-Hop Reasoning (resolving implicit references across distracting turns); and (D) Context-Aware Information Synthesis.
• Figure 2: Overview of . The framework operates in two phases. Left (§\ref{['subsec:construction']}): Contextual Intent Construction. From a streaming trajectory, the model dynamically induces three structural cues—Thematic Scope ($\sigma_t$), Event Type ($\epsilon_t$), and Key Entity Types ($\kappa_t$)—to form a Contextual Intent tuple $\iota_t$. This structure guides coreference resolution (e.g., resolving "it") and summary generation to create a structured Memory Snippet. Right (§\ref{['sec:retrieval']}): Intent-Aware Retrieval. During inference, the evaluation query $q$ is mapped to a structured filter configuration $F_q$. The retrieval engine applies Label Density Ranking, prioritizing memories that strictly match the intended structure before ranking by semantic content.
• Figure 3: Results broken down by question type in . We compare with the strongest baseline in each category: gpt-5-mini for Long-Context Models, text-embedding-3-large for Embedding RAG Agents, and SeCom secom for Structure-Augmented RAG Agents. The evaluation addresses four distinct capabilities: Incremental Memory Update (Type 1), Context-Aware Factual Recall (Type 2), Context-Aware Multi-Hop Reasoning (Type 3), and Information Synthesis (Type 4). See sec:benchmark_question_types for detailed definitions.
• Figure : Prompts for thematic scope generation, as detailed in sec:thematic_scope. In the prompt text, we refer to thematic scope as "context scope," and "prior structure notes" correspond to the recent history $H_{\text{scope}}$.
• Figure : Prompts for generating a compressed summary $\Sigma_{\sigma}$ of thematic scope, as detailed in sec:thematic_scope.