Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions

Tingyu Wu, Zhisheng Chen, Ziyan Weng, Shuhe Wang, Chenglong Li, Shuo Zhang, Sen Hu, Silin Wu, Qizhen Lan, Huacan Wang, Ronghao Chen

TL;DR

KnowMe-Bench shifts the evaluation of lifelong digital companions from retrieval accuracy to evidence-grounded person understanding by using high-density autobiographical narratives and a flashback-aware, time-aligned lifelog representation. It introduces a four-stage data pipeline (Segmentation, ANU Extraction, Temporal Realignment, Narrative Instantiation) and a three-tier evaluation (Memory, Reasoning, Insight) with an LLM-as-a-Judge scoring protocol and expert verification. Experiments across three public corpora reveal that retrieval-based systems improve factual recall but struggle with temporally grounded explanations and high-level psychodynamic reasoning, highlighting a gap in current memory architectures. The work provides openly available data, timelines, baselines, and evaluation tools to advance truly understanding user motivations, values, and long-horizon preferences in digital companions.

Abstract

Existing long-horizon memory benchmarks mostly use multi-turn dialogues or synthetic user histories, which makes retrieval performance an imperfect proxy for person understanding. We present \BenchName, a publicly releasable benchmark built from long-form autobiographical narratives, where actions, context, and inner thoughts provide dense evidence for inferring stable motivations and decision principles. \BenchName~reconstructs each narrative into a flashback-aware, time-anchored stream and evaluates models with evidence-linked questions spanning factual recall, subjective state attribution, and principle-level reasoning. Across diverse narrative sources, retrieval-augmented systems mainly improve factual accuracy, while errors persist on temporally grounded explanations and higher-level inferences, highlighting the need for memory mechanisms beyond retrieval. Our data is in \href{KnowMeBench}{https://github.com/QuantaAlpha/KnowMeBench}.

KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions

TL;DR

KnowMe-Bench shifts the evaluation of lifelong digital companions from retrieval accuracy to evidence-grounded person understanding by using high-density autobiographical narratives and a flashback-aware, time-aligned lifelog representation. It introduces a four-stage data pipeline (Segmentation, ANU Extraction, Temporal Realignment, Narrative Instantiation) and a three-tier evaluation (Memory, Reasoning, Insight) with an LLM-as-a-Judge scoring protocol and expert verification. Experiments across three public corpora reveal that retrieval-based systems improve factual recall but struggle with temporally grounded explanations and high-level psychodynamic reasoning, highlighting a gap in current memory architectures. The work provides openly available data, timelines, baselines, and evaluation tools to advance truly understanding user motivations, values, and long-horizon preferences in digital companions.

Abstract

Existing long-horizon memory benchmarks mostly use multi-turn dialogues or synthetic user histories, which makes retrieval performance an imperfect proxy for person understanding. We present \BenchName, a publicly releasable benchmark built from long-form autobiographical narratives, where actions, context, and inner thoughts provide dense evidence for inferring stable motivations and decision principles. \BenchName~reconstructs each narrative into a flashback-aware, time-anchored stream and evaluates models with evidence-linked questions spanning factual recall, subjective state attribution, and principle-level reasoning. Across diverse narrative sources, retrieval-augmented systems mainly improve factual accuracy, while errors persist on temporally grounded explanations and higher-level inferences, highlighting the need for memory mechanisms beyond retrieval. Our data is in \href{KnowMeBench}{https://github.com/QuantaAlpha/KnowMeBench}.
Paper Structure (45 sections, 2 equations, 5 figures, 2 tables)

This paper contains 45 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. (G1) Evaluation misalignment: retrieval proxies $\neq$ person understanding.
  3. (G2) Data Substrate Misalignment: Low-Density and Decontextualized Experience Traces
  4. (M1) Autobiographical narrative substrate (addresses G2a).
  5. (M2) Cognitive-stream reconstruction with mnestic realignment (addresses G2b; supports G2a).
  6. (M3) Evidence-grounded hierarchical evaluation with expert verification (addresses G1; leverages M2).
  7. Baselines and diagnostics.
  8. Related Work
  9. Evaluation of Long-Term Memory Agents.
  10. Benchmarks for Person Modeling and Psychology.
  11. Timeline Construction and Narrative Processing.
  12. Methodology
  13. Overview
  14. Stage I: Context-Aware Segmentation (Module A)
  15. Stage II: Atomic Narrative Unit (ANU) Extraction (Module B)
  16. ...and 30 more sections

Figures (5)

  • Figure 1: Comparison of information density and reasoning capabilities between existing benchmarks and KnowMe-Bench. The left panel illustrates the limitations of existing benchmarks, which rely on low-density traces (sparse dialogues) and suffer from undifferentiated textual flattening (lacking real-time inner thoughts), often leading to reasoning errors in complex queries. In contrast, KnowMe-Bench (right panel) utilizes an autobiographical narrative substrate rich in situational detail and inner monologue. By employing cognitive-stream construction and evidence-grounded hierarchical evaluation, it effectively models multi-dimensional life experiences, enabling the model to deeply research long-term impacts.
  • Figure 1: Performance breakdown across different datasets. (a) Aggregate results; (b) Dataset 1 (Knausgård); (c) Dataset 2 (Ferrante); (d) Dataset 3 (Proust).
  • Figure 2: Overview of the multi-agent dataset generation pipeline. The framework transforms unstructured raw narratives into the structured KnowMe-Bench benchmark through four sequential stages: (A) Segmentation, (B) Atomic Unit (ANU) Extraction, (C) Timeline Generation, and (D) Narrative Generation. To ensure data fidelity, each generative module is paired with a specific Check Agent that enforces a "Verify-and-Revise" loop, culminating in final validation by human literary experts.
  • Figure : Qwen3-32B Performance Profile
  • Figure : GPT-5-mini Performance Profile