Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

Jiawei Chen, Ruoxi Xu, Boxi Cao, Ruotong Pan, Yunfei Zhang, Yifei Hu, Yong Du, Tingting Gao, Yaojie Lu, Yingfei Sun, Xianpei Han, Le Sun, Xiangyu Wu, Hongyu Lin

Abstract

The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain constrained to isolated scenarios, narrow action spaces, or synthetic data, failing to capture the holistic nature of authentic human behavior. To bridge this gap, we introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data, integrating long-horizon, cross-scenario, and heterogeneous behavioral patterns into a unified framework. Based on this benchmark, we first provide empirical evidence that previous datasets with isolated scenarios suffer from tunnel vision, whereas real-world decision-making relies on long-term, cross-scenario causal chains. Extensive evaluations of state-of-the-art LLMs reveal that current models struggle to accurately simulate these complex behaviors, with performance plateauing even as context windows expand. Crucially, a systematic comparison between simulated and authentic behaviors uncovers a fundamental structural bias: LLMs tend to converge toward a positive average person, exhibiting hyper-activity, persona homogenization, and a Utopian bias. This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.

Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

Abstract

The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain constrained to isolated scenarios, narrow action spaces, or synthetic data, failing to capture the holistic nature of authentic human behavior. To bridge this gap, we introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data, integrating long-horizon, cross-scenario, and heterogeneous behavioral patterns into a unified framework. Based on this benchmark, we first provide empirical evidence that previous datasets with isolated scenarios suffer from tunnel vision, whereas real-world decision-making relies on long-term, cross-scenario causal chains. Extensive evaluations of state-of-the-art LLMs reveal that current models struggle to accurately simulate these complex behaviors, with performance plateauing even as context windows expand. Crucially, a systematic comparison between simulated and authentic behaviors uncovers a fundamental structural bias: LLMs tend to converge toward a positive average person, exhibiting hyper-activity, persona homogenization, and a Utopian bias. This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.

Paper Structure

This paper contains 49 sections, 26 figures, 6 tables.

Table of Contents

  1. Introduction
  2. OmniBehavior Benchmark
  3. Overview
  4. Taxonomy
  5. Construction Pipeline
  6. Data Collection
  7. Data Cleaning
  8. Data Sampling
  9. Data Anonymization
  10. Data Statistics
  11. Authentic User Behavior Analysis
  12. Multi-scenario Traces Enable Comprehensive User Modeling
  13. Causal Chains of User Behavior Span Across Scenarios and Long Sequences
  14. Distributional Divergence in Preference Evolution between Synthetic and Real User
  15. Experiments
  16. ...and 34 more sections

Figures (26)

  • Figure 1: Overview of OmniBehavior, a real-world comprehensive benchmark for evaluating LLM-based user simulators. The benchmark is constructed in three stages: (1) Data Collection: aggregation of real-world logs from the Kuaishou platform across several major scenarios, with Customer Service treated as part of the E-commerce scenario, yielding five scenarios in total after aggregation. (2) Data Processing: multi-modal fusion, two-level cleaning, representative sampling, and anonymization. (3) Benchmark Construction: the resulting dataset captures long-horizon, cross-scenario behavior traces, providing a high-fidelity testbed for evaluating LLM-based user simulators in real-world industrial settings.
  • Figure 2: User profile reconstruction based on Single-scenario vs. Multi-scenario data.
  • Figure 3: Cumulative interest coverage with increasing scenarios.
  • Figure 4: Distributions of causal chain spans. (a) Time and (b) scenario counts demonstrate that user decisions naturally extend over long periods and across multiple scenarios.
  • Figure 5: Case study of a cross-scenario causal chain, in which a search-initiated interest in “Xiaomi” accumulates across diverse interactions over 12 days and culminates in a purchase, highlighting the long-term and cross-scenario nature of user behavior.
  • ...and 21 more figures