Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition

Tyler McDonald, Ali Emami

TL;DR

This work addresses the challenge of enabling smaller, open-source language models to emulate the reasoning capabilities of larger models without extensive fine-tuning. It introduces Trace-of-Thought Prompting, a prompt-based distillation framework that decomposes problems into steps and transfers reasoning from high-resource teachers to low-resource students via in-context learning. The authors formalize the approach, provide a practical application example, and evaluate on GSM8K and MATH across multiple prompting strategies and teacher–student configurations, reporting substantial absolute gains, especially for weaker models. Significantly, Trace-of-Thought enhances the transparency of the reasoning process, enabling more effective human-in-the-loop interventions and potentially broadening access to capable interpretable reasoning in open-source models. Collectively, the results suggest a viable path for leveraging open-source LLMs as both students and teachers, diminishing reliance on proprietary, high-resource systems.

Abstract

Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.

Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition

TL;DR

This work addresses the challenge of enabling smaller, open-source language models to emulate the reasoning capabilities of larger models without extensive fine-tuning. It introduces Trace-of-Thought Prompting, a prompt-based distillation framework that decomposes problems into steps and transfers reasoning from high-resource teachers to low-resource students via in-context learning. The authors formalize the approach, provide a practical application example, and evaluate on GSM8K and MATH across multiple prompting strategies and teacher–student configurations, reporting substantial absolute gains, especially for weaker models. Significantly, Trace-of-Thought enhances the transparency of the reasoning process, enabling more effective human-in-the-loop interventions and potentially broadening access to capable interpretable reasoning in open-source models. Collectively, the results suggest a viable path for leveraging open-source LLMs as both students and teachers, diminishing reliance on proprietary, high-resource systems.

Abstract

Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.
Paper Structure (20 sections, 6 equations, 4 figures, 10 tables)

This paper contains 20 sections, 6 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt-Based Knowledge Distillation
  4. Trace-of-Thought Prompting
  5. Formalization
  6. Practical Application Example
  7. Experimental Setup
  8. Benchmarks
  9. Prompting Approaches
  10. Evaluation
  11. Results
  12. High-Resource Teachers - GPT-4
  13. Low-Resource Teachers - Llama 3
  14. Relative Accuracy Changes
  15. Distillation Ability vs. Absolute Performance
  16. ...and 5 more sections

Figures (4)

  • Figure 1: A Visual Depiction of our Trace-of-Thought prompting strategy on a GSM8K problem instance cobbe2021training.
  • Figure 2: Visual depiction of the methods employed during experimentation. Trace-of-Thought provides a novel decomposition framework in a linear manner.
  • Figure 3: Relative accuracy changes with Trace-of-Thought (high-resource) visualized, in order of absolute performance.
  • Figure 4: Relative accuracy changes with Trace-of-Thought (low-resource) visualized, in order of absolute performance.