Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching LLMs to Refine with Tools

Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song, Zhaopeng Tu, Haitao Mi, Dong Yu

TL;DR

CaP introduces a framework that uses external tools to refine chain-of-thought (CoT) responses, enabling cross-reasoning refinement across different LLMs. It employs a two-stage training process—supervised fine-tuning (SFT) and preference optimization via DPO variants—to align refinement behavior, and it explores Best-of-N sampling and BoNBoN inference strategies to balance quality and compute. The approach demonstrates that preference optimization is essential for effective refinement, achieving improved accuracy on Chinese mathematical benchmarks and enabling efficient inference even when refining weaker CoT sources. The work highlights potential for cross-model and cross-reasoning refinement, with implications for multilingual deployment, adaptive compute, and more expressive error-aware critics that focus on processes and outcomes.

Abstract

Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP's potential for effective cross-reasoning refinement and efficient inference.

Teaching LLMs to Refine with Tools

TL;DR

CaP introduces a framework that uses external tools to refine chain-of-thought (CoT) responses, enabling cross-reasoning refinement across different LLMs. It employs a two-stage training process—supervised fine-tuning (SFT) and preference optimization via DPO variants—to align refinement behavior, and it explores Best-of-N sampling and BoNBoN inference strategies to balance quality and compute. The approach demonstrates that preference optimization is essential for effective refinement, achieving improved accuracy on Chinese mathematical benchmarks and enabling efficient inference even when refining weaker CoT sources. The work highlights potential for cross-model and cross-reasoning refinement, with implications for multilingual deployment, adaptive compute, and more expressive error-aware critics that focus on processes and outcomes.

Abstract

Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP's potential for effective cross-reasoning refinement and efficient inference.

Paper Structure

This paper contains 14 sections, 5 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Best-of-N Sampling from Teacher Models
  4. Training a Reference Policy Model
  5. Preference Optimization
  6. Experiments
  7. Data Construction and Implementation
  8. Evaluation Datasets
  9. Main Results
  10. Scaling Test-Time Compute
  11. Generalizability Across Other Backbone Models
  12. Discussions and Remaining Challenges
  13. Conclusions and Future Work
  14. Appendices

Figures (4)

  • Figure 1: Overview of refining CoT solutions with PoT solutions during alignment and inference.
  • Figure 2: Left: CaP performance using greedy decoding based on different sources of CoT responses. Right: Average accuracy of BoN and BoNBoN sampling strategies on out-of-distribution math tasks.
  • Figure 3: Average accuracy comparison of CaP SFT/DPO models trained with different backbone models, using greedy decoding during inference.
  • Figure 4: Refining off-policy responses using CaP$_\text{DPO}$ with three backbone models.