Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs

TL;DR

This study investigates whether inference scaling in self-consistency improves reasoning faithfulness across four frontier models on GSM8K. Using rigorous statistics, including bootstrap confidence intervals, McNemar's tests, and effect sizes, it reveals model-dependent outcomes: GPT-5.2 gains accuracy with stable faithfulness, Claude Opus 4.5 experiences accuracy loss but dramatic faithfulness increases, DeepSeek-v3.2 hits ceiling accuracy with modest faithfulness gains, and Gemini-3-flash shows modest changes. The findings challenge the universal self-consistency narrative and demonstrate that faithfulness and accuracy can move in opposite directions depending on the model and problem difficulty. The paper provides practical recommendations for practitioners, highlights limitations of a single faithfulness probe, and releases open-source tooling to enable broader evaluation and replication. Overall, the work emphasizes the need to test self-consistency effects per model and domain to balance accuracy, interpretability, and safety in real-world deployment.

Abstract

Self-consistency has emerged as a popular technique for improving large language model accuracy on reasoning tasks. The approach is straightforward: generate multiple reasoning paths and select the most common answer through majority voting. While this reliably boosts accuracy, it remains unclear whether these gains reflect genuine improvements in reasoning quality. We investigate a fundamental question that has not been studied before: does inference scaling improve reasoning faithfulness? We conduct a comprehensive empirical study across four frontier models (GPT-5.2, Claude Opus 4.5, Gemini-3-flash-preview, and DeepSeek-v3.2) on 100 GSM8K mathematical reasoning problems. Our analysis employs bootstrap confidence intervals, McNemar's tests for paired comparisons, and Cohen's d effect sizes to quantify the effects rigorously. The results reveal striking differences across models that challenge common assumptions about self-consistency. GPT-5.2 shows the expected pattern: accuracy improves from 78% to 90% at N=5, with faithfulness remaining relatively stable (0.540 to 0.510). Claude Opus 4.5 tells a completely different story. Its accuracy actually drops from 78% to 74.3% while faithfulness jumps dramatically from 0.270 to 0.891 at N=5. DeepSeek-v3.2, already at 98% accuracy, shows ceiling effects with modest faithfulness gains (0.440 to 0.541). Gemini-3-flash improves from 81% to 86% accuracy with a slight faithfulness decrease (0.260 to 0.212). Problem difficulty analysis reveals that GPT-5.2 solves 82% of hard problems while breaking only 13% of easy ones. Claude, in contrast, breaks 23% of easy problems, explaining its accuracy decrease. These findings matter for practitioners: self-consistency is not universally beneficial, and teams should test their specific models before deployment. We release our code and provide practical recommendations for navigating these tradeoffs.

Figures (4)

• Figure 1: Overview of experimental pipeline.
• Figure 2: Accuracy vs. Inference Scaling across all models. GPT-5.2 shows the largest gains (78% to 90% at N=5). Claude Opus 4.5 uniquely shows accuracy decrease. DeepSeek-v3.2 exhibits ceiling effects at 98%.
• Figure 3: Faithfulness vs. Inference Scaling across all models. Claude Opus 4.5 shows a dramatic spike from 0.27 to 0.89 at N=5. Other models show relatively stable or slightly decreasing faithfulness.
• Figure 4: The Accuracy-Faithfulness Tradeoff (bubble size = N). Claude Opus 4.5 (blue) moves up and left with scaling: lower accuracy, higher faithfulness. GPT-5.2 (red) moves right with stable faithfulness. DeepSeek (orange) stays in the high-accuracy region.