Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis

Abhishek More, Anthony Zhang, Nicole Bonilla, Ashvik Vivekan, Kevin Zhu, Parham Sharafoleslami, Maheep Chaudhary

TL;DR

This work tackles the persistent problem of confidence calibration in chain-of-thought prompting for large language models. It introduces EDTR, which fuses geometry-driven topological analysis of multiple CoT embeddings with Dirichlet-based uncertainty to produce calibrated confidence scores without requiring training-time tuning. Eight interpretable topological features capture coherence and clustering structure across CoT trajectories, and these are combined with Dirichlet-derived uncertainty to form a robust confidence predictor; a calibrated fusion step blends these signals for final scores. Across four diverse reasoning benchmarks and multiple model scales, EDTR yields substantially better calibration (average $\text{ECE}$ improvements around 41%) and competitive or superior task performance, demonstrating a practical, geometry-informed approach to safe, reliable LLM deployment in high-stakes contexts.

Abstract

Chain-of-thought (CoT) prompting enables Large Language Models to solve complex problems, but deploying these models safely requires reliable confidence estimates, a capability where existing methods suffer from poor calibration and severe overconfidence on incorrect predictions. We propose Enhanced Dirichlet and Topology Risk (EDTR), a novel decoding strategy that combines topological analysis with Dirichlet-based uncertainty quantification to measure LLM confidence across multiple reasoning paths. EDTR treats each CoT as a vector in high-dimensional space and extracts eight topological risk features capturing the geometric structure of reasoning distributions: tighter, more coherent clusters indicate higher confidence while dispersed, inconsistent paths signal uncertainty. We evaluate EDTR against three state-of-the-art calibration methods across four diverse reasoning benchmarks spanning olympiad-level mathematics (AIME), grade school math (GSM8K), commonsense reasoning, and stock price prediction \cite{zhang2025aime, cobbe2021training, talmor-etal-2019-commonsenseqa, yahoo_finance}. EDTR achieves 41\% better calibration than competing methods with an average ECE of 0.287 and the best overall composite score of 0.672, while notably achieving perfect accuracy on AIME and exceptional calibration on GSM8K with an ECE of 0.107, domains where baselines exhibit severe overconfidence. Our work provides a geometric framework for understanding and quantifying uncertainty in multi-step LLM reasoning, enabling more reliable deployment where calibrated confidence estimates are essential.

Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis

TL;DR

This work tackles the persistent problem of confidence calibration in chain-of-thought prompting for large language models. It introduces EDTR, which fuses geometry-driven topological analysis of multiple CoT embeddings with Dirichlet-based uncertainty to produce calibrated confidence scores without requiring training-time tuning. Eight interpretable topological features capture coherence and clustering structure across CoT trajectories, and these are combined with Dirichlet-derived uncertainty to form a robust confidence predictor; a calibrated fusion step blends these signals for final scores. Across four diverse reasoning benchmarks and multiple model scales, EDTR yields substantially better calibration (average improvements around 41%) and competitive or superior task performance, demonstrating a practical, geometry-informed approach to safe, reliable LLM deployment in high-stakes contexts.

Abstract

Chain-of-thought (CoT) prompting enables Large Language Models to solve complex problems, but deploying these models safely requires reliable confidence estimates, a capability where existing methods suffer from poor calibration and severe overconfidence on incorrect predictions. We propose Enhanced Dirichlet and Topology Risk (EDTR), a novel decoding strategy that combines topological analysis with Dirichlet-based uncertainty quantification to measure LLM confidence across multiple reasoning paths. EDTR treats each CoT as a vector in high-dimensional space and extracts eight topological risk features capturing the geometric structure of reasoning distributions: tighter, more coherent clusters indicate higher confidence while dispersed, inconsistent paths signal uncertainty. We evaluate EDTR against three state-of-the-art calibration methods across four diverse reasoning benchmarks spanning olympiad-level mathematics (AIME), grade school math (GSM8K), commonsense reasoning, and stock price prediction \cite{zhang2025aime, cobbe2021training, talmor-etal-2019-commonsenseqa, yahoo_finance}. EDTR achieves 41\% better calibration than competing methods with an average ECE of 0.287 and the best overall composite score of 0.672, while notably achieving perfect accuracy on AIME and exceptional calibration on GSM8K with an ECE of 0.107, domains where baselines exhibit severe overconfidence. Our work provides a geometric framework for understanding and quantifying uncertainty in multi-step LLM reasoning, enabling more reliable deployment where calibrated confidence estimates are essential.

Paper Structure

This paper contains 34 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Framework Overview
  6. Geometric Feature Extraction from Reasoning Embeddings
  7. Dirichlet-Based Uncertainty Quantification
  8. Confidence Fusion
  9. Experimentation and Results
  10. Datasets and Tasks
  11. Baselines
  12. Evaluation Metrics
  13. Implementation Details
  14. Base Model:
  15. CoT Generation:
  16. ...and 19 more sections

Figures (2)

  • Figure 2: Calibration quality results comparing EDTR against other methods. Points located in the lowest left region are optimal results. Each point correlates one of 4 different datasets.
  • Figure 3: Heatmap showing the results for 4 different metrics. The results are provided across each method for the four different datasets. The darker the blue, the closer it is to the ideal result.