Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, Samy Bengio

TL;DR

ASTRA-bench is presented, a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents, and exposes critical limitations in current agents'ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans.

Abstract

Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.

ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

TL;DR

ASTRA-bench is presented, a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents, and exposes critical limitations in current agents'ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans.

Abstract

Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.
Paper Structure (59 sections, 2 equations, 6 figures, 10 tables)

This paper contains 59 sections, 2 equations, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Summary of Contributions
  3. Related Work
  4. Tool-Use Agents and Benchmarks
  5. Personal Context in Agent Evaluation
  6. Synthetic Personas and Storylines
  7. Personal Data Generation
  8. The Protagonist: Real-Life Event-Driven Storyline
  9. The Tenet: Event-Driven Context Data Generation
  10. Agentic Generation Workflow
  11. Protagonist-Centered Scenarios Authoring
  12. User-Centric Scenario Set Up
  13. User Queries and Complexity Tiers
  14. Informational complexity.
  15. Referential complexity.
  16. ...and 44 more sections

Figures (6)

  • Figure 1: The ASTRA-bench Evaluation Lifecycle. Scenarios are grounded in a protagonist’s behavioral prior and time-evolving storyline. Events are projected into multi-source personal data (Panel 1 & 2), forming the basis for human-authored queries annotated by complexity tiers. Evaluation is grounded in intermediate reasoning monologues, related entities, and verifiable success conditions (Panel 3 & 4).
  • Figure 2: Example of an Event and its Projected Artifacts from our Generation Workflow
  • Figure S1: Annotation Flowchart End-to-End
  • Figure S2: Evaluation Strategy Review
  • Figure S3: LLM-judge evaluation across complexity regimes
  • ...and 1 more figures