The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents

TL;DR

Calibration for autonomous tool-using LLM agents is challenged by misalignment between expressed confidence and actual performance. The authors identify a confidence dichotomy: evidence tools induce overconfidence due to noisy retrieval, while verification tools provide grounding through deterministic feedback. They propose Calibration Agentic RL (CAR) with Margin-Separated Calibration Reward (MSCR) to jointly optimize task accuracy and calibration, validating across evidence and verification tasks including API-based retrieval and math reasoning. CAR achieves significant calibration gains (up to $68%$ reduction in $ECE$) with preserved accuracy and demonstrates transfer to noisy real-world settings, establishing a foundation for self-aware, uncertainty-aware agents.

Abstract

Autonomous agents based on large language models (LLMs) are rapidly evolving to handle multi-turn tasks, but ensuring their trustworthiness remains a critical challenge. A fundamental pillar of this trustworthiness is calibration, which refers to an agent's ability to express confidence that reliably reflects its actual performance. While calibration is well-established for static models, its dynamics in tool-integrated agentic workflows remain underexplored. In this work, we systematically investigate verbalized calibration in tool-use agents, revealing a fundamental confidence dichotomy driven by tool type. Specifically, our pilot study identifies that evidence tools (e.g., web search) systematically induce severe overconfidence due to inherent noise in retrieved information, while verification tools (e.g., code interpreters) can ground reasoning through deterministic feedback and mitigate miscalibration. To robustly improve calibration across tool types, we propose a reinforcement learning (RL) fine-tuning framework that jointly optimizes task accuracy and calibration, supported by a holistic benchmark of reward designs. We demonstrate that our trained agents not only achieve superior calibration but also exhibit robust generalization from local training environments to noisy web settings and to distinct domains such as mathematical reasoning. Our results highlight the necessity of domain-specific calibration strategies for tool-use agents. More broadly, this work establishes a foundation for building self-aware agents that can reliably communicate uncertainty in high-stakes, real-world deployments.

Figures (1)

• Figure 1: The confidence dichotomy and the proposed RL framework. (a) Our pilot study reveals a tool-dependent divergence in calibration: evidence tools (e.g., web search), which operate in noisy retrieval environments, systematically induce overconfidence. In contrast, verification tools (e.g., code interpreters), which provide deterministic execution feedback, exhibit better alignment between confidence and accuracy. (b) To address this miscalibration, we fine-tune agents with a joint RL objective that combines task accuracy and calibration rewards, producing robust agents with reliable uncertainty expression.