ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation

TL;DR

ARTIS addresses the challenge of improving reliability for agentic LLMs that interact with external environments by enabling test-time computation through iterative simulation of action trajectories before real execution. It introduces a risk-aware tool simulator trained with failure-driven data and rebalanced objectives, and a three-stage framework (iterative simulation, summarization, final execution) that separates exploration from commitment. Empirical results on BFCL-v3 and ACEBench show substantial gains over direct execution and standard TTS baselines, with ablations confirming the necessity of evaluation and summarization, and analysis indicating gains generalize across backbones and scale with the number of simulated attempts. The work advances safe, robust, and scalable agentic decision-making by prioritizing high-impact failure prediction and task-level planning during test time, enabling more reliable real-world tool use.

Abstract

Current test-time scaling (TTS) techniques enhance large language model (LLM) performance by allocating additional computation at inference time, yet they remain insufficient for agentic settings, where actions directly interact with external environments and their effects can be irreversible and costly. We propose \emph{\name}, \emph{\underline{A}gentic \underline{R}isk-Aware \underline{T}est-Time Scaling via \underline{I}terative \underline{S}imulation}, a framework that decouples exploration from commitment by enabling test-time exploration through simulated interactions prior to real-world execution. This design allows extending inference-time computation to improve action-level reliability and robustness without incurring environmental risk. We further show that naive LLM-based simulators struggle to capture rare but high-impact failure modes, substantially limiting their effectiveness for agentic decision making. To address this limitation, we introduce a \emph{risk-aware tool simulator} that emphasizes fidelity on failure-inducing actions via targeted data generation and rebalanced training. Experiments on multi-turn and multi-step agentic benchmarks demonstrate that iterative simulation substantially improves agent reliability, and that risk-aware simulation is essential for consistently realizing these gains across models and tasks.

Figures (7)

• Figure 1: Preliminary results on Qwen3-8B, where "N" denotes the maximum number of allowed attempts.
• Figure 2: The overall framework for ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation.
• Figure 3: Risk-aware tool simulator building procedure.
• Figure 4: Abaltion results on Qwen3-8B and Qwen3-32B, where "$N$" denotes the maximum number of allowed attempts and "Full" indicates the full ATRIS method.
• Figure 5: The scaling effects on the accuracy performance and completion token consumption.