Self-Training Large Language Models with Confident Reasoning

TL;DR

This paper tackles the limitation of existing confidence-based self-training for LLMs, which relies on answer-level confidence and can mislead reasoning quality. It introduces CORE-PO, a method that incorporates reasoning-level confidence $C_{ heta}(r|x)$ alongside answer-level confidence to bias training toward high-confidence reasoning paths using online Direct Preference Optimization. Empirical results across GSM8K, ARC-Challenge, GPQA, MATH, CRUXEval, and Game of 24 show improved reasoning accuracy and robustness, with gains observed on both in-distribution and out-of-distribution tasks for two base models. The work advances practical self-training of LLMs by emphasizing reasoning quality, offering a more reliable route to improved generalization in complex problem-solving tasks.

Abstract

Large language models (LLMs) have shown impressive performance by generating reasoning paths before final answers, but learning such a reasoning path requires costly human supervision. To address this issue, recent studies have explored self-training methods that improve reasoning capabilities using pseudo-labels generated by the LLMs themselves. Among these, confidence-based self-training fine-tunes LLMs to prefer reasoning paths with high-confidence answers, where confidence is estimated via majority voting. However, such methods exclusively focus on the quality of the final answer and may ignore the quality of the reasoning paths, as even an incorrect reasoning path leads to a correct answer by chance. Instead, we advocate the use of reasoning-level confidence to identify high-quality reasoning paths for self-training, supported by our empirical observations. We then propose a new self-training method, CORE-PO, that fine-tunes LLMs to prefer high-COnfidence REasoning paths through Policy Optimization. Our experiments show that CORE-PO improves the accuracy of outputs on four in-distribution and two out-of-distribution benchmarks, compared to existing self-training methods.

Figures (3)

• Figure 1: Limitations in existing confidence-based self-training methods. Existing self-training methods prefer reasoning paths associated with a high-confidence answer (a), estimated via majority voting. However, they fail to capture the errors in their third reasoning path, even though the answer is correct. As a result, they can degrade the reasoning capabilities of the LLM, e.g., preferring "(b)-(d) are boiling points" can lead to "$32\,^{\circ}\mathrm{C}$ is the boiling point", as shown in below. In contrast, our method measures reasoning-level confidence (as depicted by the dashed line) and fine-tunes LLMs to prefer high-confidence reasoning paths that yield correct statements.
• Figure 2: Answer vs. reasoning accuracies. We use Llama3.1-8B-Instruct meta2024llama3. Reasoning-level accuracy coincides more closely with reasoning-level confidence than with answer-level confidence.
• Figure 3: Overview of CORE-PO. The LLM $M_{\theta}$ generates multiple outputs, each consisting of a reasoning and an answer $s = [r, a]$ for a given question. Next, we measure the reasoning-level confidence $C_{\theta}(r|x)=\text{P(True)}$ for each reasoning path. Then, we fine-tune the LLM to prefer high-confidence reasoning paths.