TeleMem: Building Long-Term and Multimodal Memory for Agentic AI

TL;DR

TeleMem tackles the challenge of sustaining long-term, multimodal interactions by integrating a unified long-term memory with narrative dynamic extraction, paired with a structured writing pipeline that batches, clusters, and consolidates memory entries. It introduces a text memory module (profiles and events) and a multimodal memory module (event and key–value object memories) with ReAct-style reasoning for end-to-end observe–think–act processing over video. Empirical results on the ZH-4O benchmark show TeleMem achieving 19% higher accuracy, 43% fewer tokens, and 2.1x speedup over Mem0, demonstrating improved memory fidelity, storage efficiency, and processing speed. The work suggests significant practical impact for agentic AI requiring robust, scalable long-term memory and multimodal comprehension in real-world applications.

Abstract

Large language models (LLMs) excel at many NLP tasks but struggle to sustain long-term interactions due to limited attention over extended dialogue histories. Retrieval-augmented generation (RAG) mitigates this issue but lacks reliable mechanisms for updating or refining stored memories, leading to schema-driven hallucinations, inefficient write operations, and minimal support for multimodal reasoning.To address these challenges, we propose TeleMem, a unified long-term and multimodal memory system that maintains coherent user profiles through narrative dynamic extraction, ensuring that only dialogue-grounded information is preserved. TeleMem further introduces a structured writing pipeline that batches, retrieves, clusters, and consolidates memory entries, substantially improving storage efficiency, reducing token usage, and accelerating memory operations. Additionally, a multimodal memory module combined with ReAct-style reasoning equips the system with a closed-loop observe, think, and act process that enables accurate understanding of complex video content in long-term contexts. Experimental results show that TeleMem surpasses the state-of-the-art Mem0 baseline with 19% higher accuracy, 43% fewer tokens, and a 2.1x speedup on the ZH-4O long-term role-play gaming benchmark.

Figures (5)

• Figure 1: Overview of the text memory writing pipeline in TeleMem. The system maintains three types of long-term memory: user profile memory $\mathcal{M}_{\mathrm{user}}$, bot profile memory $\mathcal{M}_{\mathrm{bot}}$, and event memory $\mathcal{M}_e$. 1). For each dialogue turn $D_t$, the model generates role-specific profile summaries $p_t^{\mathrm{user}}$ and $p_t^{\mathrm{bot}}$, which are written into their respective profile memory stores in parallel. 2). Event memory is constructed through a four-stage batch pipeline. Each turn $D_t$ is summarized into one or more textual summaries $s_i$, forming the set $\mathcal{S}$. For every summary $s_i$, the system retrieves the most relevant event memories from $\mathcal{M}_e$, yielding retrieval sets $R_i$. All summaries and retrieved items are merged into a unified candidate pool $\mathcal{U}$, which is globally clustered into semantic groups $C_i \in \mathcal{C}$. For each cluster $C_i$, entries are chronologically ordered and passed to an LLM that assigns an action $A \in \{\textit{add}, \textit{delete}, \textit{update}, \textit{no-op}\}$, producing consolidated event memory updates that are written back into $\mathcal{M}_e$.
• Figure 2: Example JSON snippet of a text profile memory entry.
• Figure 3: Example JSON snippet of a text event memory entry.
• Figure 4: Example JSON snippet of a video event memory entry.
• Figure 5: Example JSON snippet of an object memory entry.