Agentic Confidence Calibration

TL;DR

The paper tackles agentic confidence calibration by introducing Holistic Trajectory Calibration (HTC), a trajectory-centric framework that extracts a 48-dimensional set of process-level features from an agent’s full log-probability trajectory and maps them through a simple, interpretable calibrator to produce calibrated confidence scores. HTC addresses compounding and multi-source uncertainty, enabling robust calibration with strong discrimination while maintaining data efficiency and interpretability. A General Agent Calibrator (GAC) trained on diverse tasks demonstrates cross-domain generalization, achieving state-of-the-art zero-shot calibration on GAIA. Collectively, HTC advances a process-centric paradigm that supports interpretable diagnostics, transferability across domains, and scalable reliability for autonomous agents in high-stakes settings. These contributions offer practical pathways to real-time reliability monitoring and guidance for design improvements in agentic AI systems, with broad implications for safe deployment and human-AI collaboration.

Abstract

AI agents are rapidly advancing from passive language models to autonomous systems executing complex, multi-step tasks. Yet their overconfidence in failure remains a fundamental barrier to deployment in high-stakes settings. Existing calibration methods, built for static single-turn outputs, cannot address the unique challenges of agentic systems, such as compounding errors along trajectories, uncertainty from external tools, and opaque failure modes. To address these challenges, we introduce, for the first time, the problem of Agentic Confidence Calibration and propose Holistic Trajectory Calibration (HTC), a novel diagnostic framework that extracts rich process-level features ranging from macro dynamics to micro stability across an agent's entire trajectory. Powered by a simple, interpretable model, HTC consistently surpasses strong baselines in both calibration and discrimination, across eight benchmarks, multiple LLMs, and diverse agent frameworks. Beyond performance, HTC delivers three essential advances: it provides interpretability by revealing the signals behind failure, enables transferability by applying across domains without retraining, and achieves generalization through a General Agent Calibrator (GAC) that achieves the best calibration (lowest ECE) on the out-of-domain GAIA benchmark. Together, these contributions establish a new process-centric paradigm for confidence calibration, providing a framework for diagnosing and enhancing the reliability of AI agents.

Figures (15)

• Figure 1: Overview of Holistic Trajectory Calibration (HTC). The framework first collects confidence signals along the agent’s trajectory, then derives rich process-level diagnostic features, which are used to train a simple yet interpretable calibrator. This process not only improves calibration accuracy but also yields the three pillars of reliable agentic AI: interpretability, transferability, and generalization.
• Figure 2: Learning Curve Comparison: HTCvs. Learning-Based Baselines on SimpleQA dataset, showing HTCconsistently outperforms and exhibits much lower variance under small-data regimes.
• Figure 3: The Impact of Base LLM on Calibration Performance on the SimpleQA dataset.
• Figure 4: (Left) Distribution of feature importance across different task domains. (Right) Frequency of feature category across different levels, including Top 1, Top 3, Top 5 and all selected features.
• Figure 5: Low-dimensional t-SNE visualization of the feature spaces for different datasets.