Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs

Hadi Mohammadi, Tamas Kozak, Anastasia Giachanou

TL;DR

The paper addresses the faithfulness of chain-of-thought reasoning in large language models and compares two optimization approaches, GRPO and DPO, across a scale range of 1.5B–14B on the GSM8K benchmark. Using LoRA-adapted fine-tuning on Qwen2.5-Instruct models and a fixed evaluation suite of five metrics, the study finds that model size generally improves both accuracy and faithfulness, with GRPO achieving the strongest faithfulness gains at the largest scales (notably 14B) while showing less stability at smaller scales. DPO provides steadier improvements across model sizes but delivers comparatively smaller faithfulness gains. The results imply that GRPO is a promising direction for more transparent reasoning in very large models, albeit requiring careful hyperparameter tuning and more resources, whereas DPO offers a more practical option for mid-sized models. The study highlights distinct faithfulness dimensions captured by NLI-based metrics and human-like judgments, informing future work on scalable, interpretable alignment of CoT reasoning.

Abstract

Chain-of-thought (CoT) reasoning has emerged as a powerful technique for improving the problem-solving capabilities of large language models (LLMs), particularly for tasks requiring multi-step reasoning. However, recent studies show that CoT explanations often fail to reflect the model's actual reasoning process, as models may produce coherent yet misleading justifications or modify answers without acknowledging external cues. Such discrepancies undermine the reliability of CoT-based methods for safety supervision and alignment monitoring, as models can generate plausible but deceptive rationales for incorrect answers. To better understand this limitation, we evaluate two optimization methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), in their ability to improve CoT faithfulness. Our experiments show that GRPO achieves higher performance than DPO in larger models, with the Qwen2.5-14B-Instruct model attaining the best results across all evaluation metrics. Both approaches exhibit positive correlations between model size and performance, but GRPO shows greater potential for improving faithfulness metrics, albeit with less stable behavior at smaller scales. These results suggest that GRPO offers a promising direction for developing more transparent and trustworthy reasoning in LLMs.

Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs

TL;DR

The paper addresses the faithfulness of chain-of-thought reasoning in large language models and compares two optimization approaches, GRPO and DPO, across a scale range of 1.5B–14B on the GSM8K benchmark. Using LoRA-adapted fine-tuning on Qwen2.5-Instruct models and a fixed evaluation suite of five metrics, the study finds that model size generally improves both accuracy and faithfulness, with GRPO achieving the strongest faithfulness gains at the largest scales (notably 14B) while showing less stability at smaller scales. DPO provides steadier improvements across model sizes but delivers comparatively smaller faithfulness gains. The results imply that GRPO is a promising direction for more transparent reasoning in very large models, albeit requiring careful hyperparameter tuning and more resources, whereas DPO offers a more practical option for mid-sized models. The study highlights distinct faithfulness dimensions captured by NLI-based metrics and human-like judgments, informing future work on scalable, interpretable alignment of CoT reasoning.

Abstract

Chain-of-thought (CoT) reasoning has emerged as a powerful technique for improving the problem-solving capabilities of large language models (LLMs), particularly for tasks requiring multi-step reasoning. However, recent studies show that CoT explanations often fail to reflect the model's actual reasoning process, as models may produce coherent yet misleading justifications or modify answers without acknowledging external cues. Such discrepancies undermine the reliability of CoT-based methods for safety supervision and alignment monitoring, as models can generate plausible but deceptive rationales for incorrect answers. To better understand this limitation, we evaluate two optimization methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), in their ability to improve CoT faithfulness. Our experiments show that GRPO achieves higher performance than DPO in larger models, with the Qwen2.5-14B-Instruct model attaining the best results across all evaluation metrics. Both approaches exhibit positive correlations between model size and performance, but GRPO shows greater potential for improving faithfulness metrics, albeit with less stable behavior at smaller scales. These results suggest that GRPO offers a promising direction for developing more transparent and trustworthy reasoning in LLMs.
Paper Structure (9 sections, 2 figures, 1 table)

This paper contains 9 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Datasets
  4. Models
  5. Evaluation Metrics
  6. Results
  7. Performance Analysis
  8. Faithfulness and Reliability Analysis
  9. Conclusion

Figures (2)

  • Figure 1: Overall performance: (a) Average across scales showing 3B collapse and recovery; (b) 14B peak performance demonstrating GRPO's faithfulness advantages.
  • Figure 2: Model-scale comparison: (a) 1.5B baseline similarity; (b) 3B performance collapse; (c) 7B GRPO advantages emerge in faithfulness metrics.