MAXS: Meta-Adaptive Exploration with LLM Agents

TL;DR

MAXS tackles two persistent issues in LLM-agent reasoning: local myopia in tool usage and trajectory instability across multi-tool reasoning. It introduces a meta-adaptive framework that performs lookahead rollouts to forecast future rewards, and uses a composite value function that combines an advantage score with step- and slope-based variance terms to steer stable, high-value reasoning steps. A trajectory-convergence mechanism halts rollout when path consistency is achieved, reducing computational cost without sacrificing global effectiveness. Empirical results on five benchmarks and three backbones show MAXS yields state-of-the-art accuracy with substantially lower token costs, and ablations validate the critical roles of lookahead and the proposed value-estimation components. The work demonstrates strong generalization to larger models and highlights practical gains in inference-time efficiency for multi-tool LLM reasoning.

Abstract

Large Language Model (LLM) Agents exhibit inherent reasoning abilities through the collaboration of multiple tools. However, during agent inference, existing methods often suffer from (i) locally myopic generation, due to the absence of lookahead, and (ii) trajectory instability, where minor early errors can escalate into divergent reasoning paths. These issues make it difficult to balance global effectiveness and computational efficiency. To address these two issues, we propose meta-adaptive exploration with LLM agents https://github.com/exoskeletonzj/MAXS, a meta-adaptive reasoning framework based on LLM Agents that flexibly integrates tool execution and reasoning planning. MAXS employs a lookahead strategy to extend reasoning paths a few steps ahead, estimating the advantage value of tool usage, and combines step consistency variance and inter-step trend slopes to jointly select stable, consistent, and high-value reasoning steps. Additionally, we introduce a trajectory convergence mechanism that controls computational cost by halting further rollouts once path consistency is achieved, enabling a balance between resource efficiency and global effectiveness in multi-tool reasoning. We conduct extensive empirical studies across three base models (MiMo-VL-7B, Qwen2.5-VL-7B, Qwen2.5-VL-32B) and five datasets, demonstrating that MAXS consistently outperforms existing methods in both performance and inference efficiency. Further analysis confirms the effectiveness of our lookahead strategy and tool usage.

Figures (13)

• Figure 1: An example of LLM Agents solving a task via multi-step reasoning, dynamically leveraging search and code tools to obtain the final answer.
• Figure 2: Comparison of test time reasoning strategies. CoT and ToT follow step by step generation with limited foresight, while MCTS conducts global simulation at a higher computational cost. On the right, MAXS uses MiMo-VL-7B-SFT as the backbone and consistently outperforms baseline methods across benchmarks.
• Figure 3: Illustration of the MAXS framework. Left: LLM Agents generates reasoning steps from input $s_0$ to final answer $s_n$. Right: At each step, MAXS performs (a) rollout & lookahead, (b) value estimation via advantage and two variance scores, and (c) integration. A trajectory convergence mechanism halts rollouts early to improve efficiency.
• Figure 4: Inference-time scaling law: Accuracy vs. Token usage for different models during decoding.
• Figure 5: Accuracy–cost trade-off under varying lookahead steps across datasets.