Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought

Jooyoung Lee, Fan Yang, Thanh Tran, Qian Hu, Emre Barut, Kai-Wei Chang, Chengwei Su

TL;DR

This paper introduces LM-Guided CoT, a resource-efficient two-LM framework where a lightweight model generates rationales that guide a frozen large LM in reasoning. It combines knowledge distillation to transfer reasoning from the large to the small model with reinforcement learning to refine rationale quality using multi-faceted rewards and a task-focused objective. On multi-hop QA benchmarks HotpotQA and 2WikiMultiHopQA, the approach outperforms standard prompting and vanilla CoT baselines, with KD yielding notable gains and RL providing additional improvements in rationale and QA performance. The method offers a practical path to enhancing reasoning in large LMs without expensive fine-tuning, highlighting a meaningful trade-off between rationale quality and task performance that warrants further exploration across tasks and models.

Abstract

We introduce a novel framework, LM-Guided CoT, that leverages a lightweight (i.e., <1B) language model (LM) for guiding a black-box large (i.e., >10B) LM in reasoning tasks. Specifically, the lightweight LM first generates a rationale for each input instance. The Frozen large LM is then prompted to predict a task output based on the rationale generated by the lightweight LM. Our approach is resource-efficient in the sense that it only requires training the lightweight LM. We optimize the model through 1) knowledge distillation and 2) reinforcement learning from rationale-oriented and task-oriented reward signals. We assess our method with multi-hop extractive question answering (QA) benchmarks, HotpotQA, and 2WikiMultiHopQA. Experimental results show that our approach outperforms all baselines regarding answer prediction accuracy. We also find that reinforcement learning helps the model to produce higher-quality rationales with improved QA performance.

Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought

TL;DR

This paper introduces LM-Guided CoT, a resource-efficient two-LM framework where a lightweight model generates rationales that guide a frozen large LM in reasoning. It combines knowledge distillation to transfer reasoning from the large to the small model with reinforcement learning to refine rationale quality using multi-faceted rewards and a task-focused objective. On multi-hop QA benchmarks HotpotQA and 2WikiMultiHopQA, the approach outperforms standard prompting and vanilla CoT baselines, with KD yielding notable gains and RL providing additional improvements in rationale and QA performance. The method offers a practical path to enhancing reasoning in large LMs without expensive fine-tuning, highlighting a meaningful trade-off between rationale quality and task performance that warrants further exploration across tasks and models.

Abstract

We introduce a novel framework, LM-Guided CoT, that leverages a lightweight (i.e., <1B) language model (LM) for guiding a black-box large (i.e., >10B) LM in reasoning tasks. Specifically, the lightweight LM first generates a rationale for each input instance. The Frozen large LM is then prompted to predict a task output based on the rationale generated by the lightweight LM. Our approach is resource-efficient in the sense that it only requires training the lightweight LM. We optimize the model through 1) knowledge distillation and 2) reinforcement learning from rationale-oriented and task-oriented reward signals. We assess our method with multi-hop extractive question answering (QA) benchmarks, HotpotQA, and 2WikiMultiHopQA. Experimental results show that our approach outperforms all baselines regarding answer prediction accuracy. We also find that reinforcement learning helps the model to produce higher-quality rationales with improved QA performance.
Paper Structure (18 sections, 1 equation, 4 figures, 5 tables)

This paper contains 18 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LM-guided Chain-of-Thought
  4. Rationale Distillation
  5. Rationale Refinement
  6. Experiments and Results
  7. Experimental Setup
  8. Results
  9. Conclusion
  10. Limitations
  11. Ethical Considerations
  12. Acknowledgments
  13. Bibliographical References
  14. Language Resource References
  15. Appendices
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Illustration of our proposed method.
  • Figure 2: Average answer prediction performance (across three evaluation metrics) and average rationale quality scores (i.e., $R_{aspect}$) for HotpotQA (left) and 2WikiMultiHopQA (right). The right y-axis represents the mean answer prediction scores, and the left y-axis represents the mean rationale quality scores.
  • Figure 3: Demonstration example for "logicality" annotation.
  • Figure 4: Mean scores of human annotation results by answer prediction correctness.