Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning

TL;DR

The paper addresses reliable multi-path LLM reasoning by decomposing error into Estimation Error and Model Error, revealing complementary strengths of perplexity-based confidence (fast estimation but higher model error) and self-consistency (lower model error but slower estimation). It introduces Reasoning-pruning Perplexity Consistency (Rpc), which fuses Perplexity Consistency with Reasoning Pruning to achieve exponential convergence in estimation error while maintaining low model error. Theoretical analyses provide explicit convergence rates and pruning guarantees, and extensive experiments on seven benchmarks in math and code generation show Rpc improves reasoning accuracy, sample efficiency, and confidence calibration across model scales. The approach promises practical gains for robust and efficient LLM reasoning in complex tasks by leveraging internal probabilities and principled path pruning.

Abstract

Recent advancements in large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, single-shot inference often yields unreliable results for complex reasoning tasks, leading researchers to explore multiple reasoning paths through methods such as perplexity and self-consistency. In this paper, we present the first theoretical error decomposition analysis of these techniques, breaking down their error into estimation error and model error. Our analysis reveals a fundamental trade-off: perplexity methods suffer from substantial model error due to the absence of a proper consistency function, while self-consistency exhibits high estimation error due to a slow error convergence rate. To overcome these limitations, we propose Reasoning-Pruning Perplexity Consistency (RPC). This approach combines Perplexity Consistency, which seamlessly integrates LLM perplexity with self-consistency, and Reasoning Pruning, which eliminates low-probability reasoning paths to effectively prevent the degeneration of estimation error reduction. Theoretical analysis demonstrates that RPC not only accelerates the convergence rate of estimation error to an exponential level but also holds strong potential for further reducing model error. Extensive empirical evaluations on seven benchmark datasets confirm that RPC can significantly improve reasoning performance, sample efficiency, and confidence reliability.

Figures (9)

• Figure 1: The comparison between self-consistency and perplexity methods. Self-consistency uses Monte-Carlo estimation while perplexity methods directly use LLM prediction probability (i.e., computing probabilities using perplexity of LLM).
• Figure 2: The performance of InternLM-MATH-Plus 7B model on GSM8K dataset. (1) Estimation error of Ppl converges faster than Sc and Naïve-Sc; (2) Proper consistency function is the key for Sc to achieve the small model error.
• Figure 3: Illustration of the Rpc approach. The Reasoning Pruning filters out low-probability answers, while the Perplexity Consistency incorporates LLM probabilities into the self-consistency framework, resulting in faster convergence of estimation error.
• Figure 4: The accuracy of the InternLM-2-MATH-Plus 7B model on four math reasoning datasets with different sample sizes $n$. Our proposed Rpc achieves the best performance across all datasets, validating our theoretical analysis. We also show the performance of a single perplexity consistency module (denoted as Pc), which further supports our theoretical findings.
• Figure 5: The reliability diagrams of Sc and Rpc on MathOdyssey dataset using InternLM-2-MATH-Plus 7B model.

Theorems & Definitions (14)

• Proposition 1: Sc Reasoning Error Decomposition
• Remark 1
• Proposition 2: Ppl Reasoning Error Decomposition
• Remark 2
• Theorem 3: Pc Reasoning Error Decomposition
• Remark 4
• Theorem 5: Effectiveness of Reasoning Path Pruning
• Remark 6
• proof
• Proposition 3: Comparison of Model Errors