Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents

TL;DR

This work uncovers a robust link between entropy reduction and high-quality tool-use in LLM agents, proposing TEPO, which uses delta segment entropy as a supervision signal. It introduces two reward designs—TEPO_{sparse} for efficiency and TEPO_{dense} for performance—and demonstrates substantial reductions in tool calls and notable performance gains across diverse domains. The results show TEPO scales with model size and can adapt tool-use behavior to real-world tasks, highlighting entropy-based supervision as a practical mechanism for training adaptive, tool-aware agents. Limitations include evaluating on up to 14B-scale models and reliance on wiki-18; future work targets larger models and real-time search APIs to extend applicability.

Abstract

Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.

Figures (7)

• Figure 1: Changes in entropy reflect shifts in uncertainty within the agent. High-quality tool calls help the model reduce uncertainty, as indicated by a decrease in entropy.
• Figure 2: The overall framework of $\text{TEPO}_{\text{sparse}}$ and $\text{TEPO}_{\text{dense}}$. In the sparse reward design, the reward and advantage are calculated and then uniformly assigned to each token within the trajectory (same $A_{i,t}$ for all tokens). In contrast, the dense reward design assigns fine-grained tool rewards and advantages, resulting in different $A_{i,t}$ values for different tokens within the same trajectory.
• Figure 3: Results on GAIA dataset evaluated with Qwen3-8B using Bing Search API as search tool, including avg@5, pass@5, and average tool calls.
• Figure 4: Results on Deep Search Tasks evaluated with different sizes of Qwen3 models using wiki-18 as search corpus, including avg@5 and average tool calls.
• Figure 5: Visualization of training dynamics, showing (a) $n$ curve, (b) $m$ curve and (c) $m/n$ ratio curve. Here, $n$ is the total number of tool calls, and $m$ is the number of tool calls inducing entropy reduction.