How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domains

TL;DR

This paper introduces the Reasoning Model Confidence estimation Benchmark (RMCB), a public, large-scale resource comprising 347,496 reasoning traces from six open-weight Large Reasoning Models across diverse high-stakes domains, labeled with ground-truth correctness at both sample and chunk levels. It rigorously evaluates over ten representation-based confidence-estimation methods, revealing a durable trade-off between discriminative power (AUROC) and calibration (ECE); text-based encoders excel at discrimination, while structure-aware models optimize calibration, with no single method achieving peak performance on both metrics. The study also finds that increased architectural complexity does not consistently outperform simple sequential baselines, highlighting the limits of chunk-level hidden-state signals and underscoring the need for signal sources beyond static representations, such as token-level or generative signals. By delivering a comprehensive benchmark, rigorous baselines, and extensive analyses across domains and models, the work establishes a foundation for improved reliability of LRMs in high-stakes settings and points to future directions in signal integration and calibration methods. The benchmark and code enable reproducible experimentation and encourage exploration of alternative signals to address current performance plateaus.

Abstract

The miscalibration of Large Reasoning Models (LRMs) undermines their reliability in high-stakes domains, necessitating methods to accurately estimate the confidence of their long-form, multi-step outputs. To address this gap, we introduce the Reasoning Model Confidence estimation Benchmark (RMCB), a public resource of 347,496 reasoning traces from six popular LRMs across different architectural families. The benchmark is constructed from a diverse suite of datasets spanning high-stakes domains, including clinical, financial, legal, and mathematical reasoning, alongside complex general reasoning benchmarks, with correctness annotations provided for all samples. Using RMCB, we conduct a large-scale empirical evaluation of over ten distinct representation-based methods, spanning sequential, graph-based, and text-based architectures. Our central finding is a persistent trade-off between discrimination (AUROC) and calibration (ECE): text-based encoders achieve the best AUROC (0.672), while structurally-aware models yield the best ECE (0.148), with no single method dominating both. Furthermore, we find that increased architectural complexity does not reliably outperform simpler sequential baselines, suggesting a performance ceiling for methods relying solely on chunk-level hidden states. This work provides the most comprehensive benchmark for this task to date, establishing rigorous baselines and demonstrating the limitations of current representation-based paradigms.

Figures (14)

• Figure 1: Performance trade-off between calibration (1-ECE) and discrimination (AUROC). Each ellipse denotes a method, with its center showing the unweighted doubly-averaged mean performance (first across datasets per LRM, then across LRMs). Ellipse width and height represent the standard deviation of these LRM-specific means, reflecting consistency across model architectures. The top-right corner marks ideal performance.
• Figure 2: Calibration curves for each test dataset, aggregated across all LRMs. Each subplot shows one dataset's reliability diagram with methods averaged across all six LRM families. The ECE values in the legend represent each method's average performance across LRMs for that specific dataset. Points closer to the diagonal (dashed line) indicate better calibration.
• Figure 5: Performance trade-off between calibration (1-ECE) and discrimination focused on the positive class (AUCPR). This plot confirms that methods with the best calibration are not necessarily the best at identifying correct answers.
• Figure 6: Performance trade-off between calibration (1-ECE) and F1 Score. This view highlights the relationship between probabilistic accuracy and the harmonic mean of precision and recall.
• Figure 7: Performance trade-off between the Brier Score (plotted as 1-Brier), which combines calibration and discrimination, and pure discrimination (AUROC).