HugAgent: Benchmarking LLMs for Simulation of Individualized Human Reasoning

TL;DR

HugAgent introduces an extensible benchmark to shift AI reasoning from population-level averages toward individualized human reasoning by combining a human-grounded track with a scalable synthetic track. The benchmark formalizes average-to-individual reasoning adaptation through two tasks—Belief State Inference and Belief Dynamics Update—under open-ended contexts and counterfactual interventions. Across three domains (healthcare, surveillance, zoning), HugAgent reveals persistent adaptation gaps in state-of-the-art LLMs, especially in updating beliefs, and highlights how cross-domain transfer erodes performance despite strong in-domain results. The work provides a principled diagnostic framework for error analysis and mitigation strategies, publishes an open data/trace-pipeline, and establishes a reproducible testbed for advancing individualized human-like reasoning in language models. Overall, HugAgent lays the groundwork for identity-consistent, context-aware reasoning in AI, with implications for social simulations, digital twins, and ethical evaluation of model alignment with human thought trajectories.

Abstract

Simulating human reasoning in open-ended tasks has long been a central aspiration in AI and cognitive science. While large language models now approximate human responses at scale, they remain tuned to population-level consensus, often erasing the individuality of reasoning styles and belief trajectories. To advance the vision of more human-like reasoning in machines, we introduce HugAgent (Human-Grounded Agent Benchmark), which rethinks human reasoning simulation along three dimensions: (i) from averaged to individualized reasoning, (ii) from behavioral mimicry to cognitive alignment, and (iii) from vignette-based to open-ended data. The benchmark evaluates whether a model can predict a specific person's behavioral responses and the underlying reasoning dynamics in out-of-distribution scenarios, given partial evidence of their prior views. HugAgent adopts a dual-track design: a human track that automates and scales the think-aloud method to collect ecologically valid human reasoning data, and a synthetic track for further scalability and systematic stress testing. This architecture enables low-cost, extensible expansion to new tasks and populations. Experiments with state-of-the-art language models reveal persistent adaptation gaps, positioning HugAgent as the first extensible benchmark for aligning machine reasoning with the individuality of human thought. The benchmark, along with its complete data collection pipeline and companion chatbot, is open-sourced as HugAgent (https://anonymous.4open.science/r/HugAgent) and TraceYourThinking (https://anonymous.4open.science/r/trace-your-thinking).

Figures (22)

• Figure 1: Illustration of HugAgent operationalizing "average-to-individual" reasoning adaptation: a gray robot repeats population consensus, then observes an individual’s nuanced reasoning, gradually aligns with that individual (turning colorful), and finally adapts under counterfactual updates (e.g., with eco-lamps). This illustrates the shift from consensus mimicry to individualized reasoning.
• Figure 2: Two benchmark tasks. Task 1 (Belief State Inference) infers stance and reasons from prior context; Task 2 (Belief Dynamics Update) predicts stance shifts and its reasoning under new evidence.
• Figure 3: HugAgent benchmark pipeline. Inputs (demographics, questionnaires, and transcripts) flow through two components: a questionnaire that provides demographic anchors, stance baselines, and counterfactual updates, and a semi-structured chatbot that elicits individualized reasoning. Together these elements define two benchmark tasks: (i) belief state inference—recovering stance and factor polarity from context, and (ii) belief dynamics update—predicting stance shifts and reweighting under new evidence. (A) The chatbot maintains a causal belief network of factors, used to identify the most critical nodes and edges for follow-up. (B) A question generator derives targeted, context-specific probes from this network to structure the dialogue.
• Figure 4: Overview of the QA loop and data structures. The system integrates the Causal Belief Network (CBN), Node Queue, and Question List to guide interaction. Stages regulate question priorities, with an irreversible transition from Stage 1 to Stage 2 once anchor nodes $\geq 3$.
• Figure 5: Illustration of the semi-structured interview process and causal belief network construction. The chatbot begins with open-ended questions and extracts candidate concepts from user responses (Anchor Node Discovery). Once three or more anchors are identified, it transitions to targeted follow-ups to expand causal relations (Anchor Expansion). Edges represent directional influences with polarity, forming the evolving CBN.