A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning

TL;DR

This work provides a theoretical framework for analyzing sampling-based test-time scaling in LLM reasoning by decomposing reasoning error into Estimation Error and Model Error. It analyzes two representative methods—self-consistency and perplexity—highlighting linear versus exponential convergence and their distinct limitations. The authors introduce RPC, a hybrid method comprising Perplexity Consistency and Reasoning Pruning, which achieves exponential-like estimation error convergence with low model error and prunes low-probability paths to prevent degradation, backed by theoretical guarantees. Empirical results across seven benchmarks show that RPC can significantly reduce the required sampling budget (by at least 50%) while delivering competitive or superior reasoning accuracy and more reliable confidence estimates, suggesting substantial practical benefits for efficient and reliable LLM reasoning.

Abstract

Test-time scaling seeks to improve the reasoning performance of large language models (LLMs) by adding computational resources. A prevalent approach within the field is sampling-based test-time scaling methods, which enhance reasoning by generating multiple reasoning paths for a given input during inference. However, despite its practical success, the theoretical foundations remain underexplored. In this paper, we provide the first theoretical framework for analyzing sampling-based test-time scaling methods, grounded in the perspective of confidence estimation. Based on the framework, we analyze two dominant paradigms: self-consistency and perplexity, and reveal key limitations: self-consistency suffers from high estimation error while perplexity exhibits substantial modeling error and possible degradation of the estimation error convergence. To address these limitations, we introduce RPC, a hybrid method that leverages our theoretical insights through two key components: Perplexity Consistency and Reasoning Pruning. Perplexity Consistency combines the strengths of self-consistency and perplexity, boosting the convergence rate of estimation error from linear to exponential while preserving model error. Reasoning Pruning prevents degradation by eliminating low-probability reasoning paths. Both theoretical analysis and empirical results across seven benchmark datasets demonstrate that RPC has a strong potential for reducing reasoning error. Notably, RPC achieves reasoning performance comparable to self-consistency while not only enhancing confidence reliability but also reducing sampling costs by 50%. The code and resources are available at https://wnjxyk.github.io/RPC.

Figures (7)

• Figure 1: Illustration of the Rpc approach. The Reasoning Pruning filters out low-probability answers, while the Perplexity Consistency incorporates LLM probabilities into the self-consistency framework, resulting in faster convergence of estimation error.
• Figure 2: The accuracy of the InternLM-2-MATH-Plus 7B model on four math reasoning datasets with different sample sizes $n$. Our proposed Rpc achieves the best performance across all datasets, validating our theoretical analysis. We also show the performance of a single perplexity consistency module (denoted as Pc), which further supports our theoretical findings.
• Figure 3: The reliability diagrams of Sc and Rpc on MathOdyssey dataset using InternLM-2-MATH-Plus 7B model.
• Figure 4: Performance on three code generation tasks using Deepseek-Coder 33B model. The experimental results show that our Rpc achieves the best performance.
• Figure 5: The accuracy of InternLM-2-MATH-Plus 7B model on four mathematical reasoning datasets with different sample sizes $n$. The sampling temperature is set to 1.1.

Theorems & Definitions (17)

• Proposition 1: Error Decomposition
• Remark 1
• Proposition 2: Sc Reasoning Error Decomposition
• Remark 2
• Proposition 3: Ppl Reasoning Error Decomposition
• Remark 3
• Theorem 4: Pc Reasoning Error Decomposition
• Remark 5
• Remark 6
• Theorem 7: Effectiveness of Reasoning Path Pruning