Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

TL;DR

Gaia2 tackles the problem of evaluating LLM agents in dynamic, asynchronous environments with verifiable rewards, addressing the sim2real gap in long-horizon, noisy settings. It introduces the Agents Research Environments (ARE) framework and Gaia2, a large, verifiable benchmark of 1,120 scenarios that stress execution, timing, ambiguity resolution, adaptability, and collaboration, all under a write-action verifier suitable for RLVR. The empirical study shows no model dominates across capabilities: GPT-5 (high) yields the best overall performance, Claude-4 Sonnet offers favorable latency, and Kimi-K2 provides strong open-source performance, with clear cost–time–accuracy trade-offs and meaningful insights into multi-agent collaboration. The work provides a practical, extensible platform for developing and benchmarking practical agent systems, enabling finer-grained analysis, reproducible evaluation, and RL data generation for future research and deployment.

Abstract

We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.

Figures (19)

• Figure 1: Gaia2 budget scaling curve: for each $\text{max\_budget}$, we plot $\sum \mathbbm{1}\{\text{scenario\_result} = \text{True} \land \text{scenario\_cost} < \text{max\_budget}\}$. Equipped with a simple ReAct-like scaffold (see Section \ref{['sec:are_environment']}), no model evaluated here dominates across the intelligence spectrum—each trades off capability, efficiency, and budget. At equal cost, some models fare better, yet all curves plateau, suggesting that standard scaffolds and/or models miss ingredients for sustained progress. Cost estimates from https://artificialanalysis.ai/models model pricing data (accessed September 10, 2025).
• Figure 2: ARE environments are event-based, time-driven simulations, that run asynchronously from the agent and the user. ARE environments allows playing scenarios, which typically contain tasks for the agent and verification logic. Whether initiated by agent or user, interactions happen through the same interfaces and can be either tool calls, or tool output/notification observations. Extensive simulation control and logging allow precise study of agents behavior.
• Figure 3: App usage distribution across the 12 Mobile apps in Gaia2 for Llama 4 Maverick.
• Figure 4: The seven core agent capabilities evaluated by the splits of Gaia2.
• Figure 5: Gaia2 scores per capability split. Models are reranked independently for each capability, highlighting where they excel or struggle.