Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation

Siddhant Bhambri, Upasana Biswas, Subbarao Kambhampati

TL;DR

This paper investigates the faithfulness of intermediate reasoning traces used in distance-based knowledge distillation for Open Book QA. It introduces a rule-based problem decomposition into Classification and Information Retrieval to generate auditable traces, enabling simultaneous evaluation of both intermediate traces and final solutions. Through supervised fine-tuning of small language models on datasets CoTemp QA, MARCO QA, and bAbI QA, the study reveals a surprising lack of consistent correlation between trace correctness and final solution accuracy—correct traces do not guarantee correct answers and vice versa. These findings challenge the premise that reasoning traces reliably improve final performance in KD and underscore the need for verifiable evaluations to maintain trust in interactive AI systems.

Abstract

Question Answering (QA) poses a challenging and critical problem, particularly in today's age of interactive dialogue systems such as ChatGPT, Perplexity, Microsoft Copilot, etc. where users demand both accuracy and transparency in the model's outputs. Since smaller language models (SLMs) are computationally more efficient but often under-perform compared to larger models, Knowledge Distillation (KD) methods allow for finetuning these smaller models to improve their final performance. Lately, the intermediate tokens or the so called `reasoning' traces produced by Chain-of-Thought (CoT) or by reasoning models such as DeepSeek R1 are used as a training signal for KD. However, these reasoning traces are often verbose and difficult to interpret or evaluate. In this work, we aim to address the challenge of evaluating the faithfulness of these reasoning traces and their correlation with the final performance. To this end, we employ a KD method leveraging rule-based problem decomposition. This approach allows us to break down complex queries into structured sub-problems, generating interpretable traces whose correctness can be readily evaluated, even at inference time. Specifically, we demonstrate this approach on Open Book QA, decomposing the problem into a Classification step and an Information Retrieval step, thereby simplifying trace evaluation. Our SFT experiments with correct and incorrect traces on the CoTemp QA, Microsoft Machine Reading Comprehension QA, and Facebook bAbI QA datasets reveal the striking finding that correct traces do not necessarily imply that the model outputs the correct final solution. Similarly, we find a low correlation between correct final solutions and intermediate trace correctness. These results challenge the implicit assumption behind utilizing reasoning traces for improving SLMs' final performance via KD.

Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation

TL;DR

This paper investigates the faithfulness of intermediate reasoning traces used in distance-based knowledge distillation for Open Book QA. It introduces a rule-based problem decomposition into Classification and Information Retrieval to generate auditable traces, enabling simultaneous evaluation of both intermediate traces and final solutions. Through supervised fine-tuning of small language models on datasets CoTemp QA, MARCO QA, and bAbI QA, the study reveals a surprising lack of consistent correlation between trace correctness and final solution accuracy—correct traces do not guarantee correct answers and vice versa. These findings challenge the premise that reasoning traces reliably improve final performance in KD and underscore the need for verifiable evaluations to maintain trust in interactive AI systems.

Abstract

Question Answering (QA) poses a challenging and critical problem, particularly in today's age of interactive dialogue systems such as ChatGPT, Perplexity, Microsoft Copilot, etc. where users demand both accuracy and transparency in the model's outputs. Since smaller language models (SLMs) are computationally more efficient but often under-perform compared to larger models, Knowledge Distillation (KD) methods allow for finetuning these smaller models to improve their final performance. Lately, the intermediate tokens or the so called `reasoning' traces produced by Chain-of-Thought (CoT) or by reasoning models such as DeepSeek R1 are used as a training signal for KD. However, these reasoning traces are often verbose and difficult to interpret or evaluate. In this work, we aim to address the challenge of evaluating the faithfulness of these reasoning traces and their correlation with the final performance. To this end, we employ a KD method leveraging rule-based problem decomposition. This approach allows us to break down complex queries into structured sub-problems, generating interpretable traces whose correctness can be readily evaluated, even at inference time. Specifically, we demonstrate this approach on Open Book QA, decomposing the problem into a Classification step and an Information Retrieval step, thereby simplifying trace evaluation. Our SFT experiments with correct and incorrect traces on the CoTemp QA, Microsoft Machine Reading Comprehension QA, and Facebook bAbI QA datasets reveal the striking finding that correct traces do not necessarily imply that the model outputs the correct final solution. Similarly, we find a low correlation between correct final solutions and intermediate trace correctness. These results challenge the implicit assumption behind utilizing reasoning traces for improving SLMs' final performance via KD.

Paper Structure

This paper contains 25 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language & Reasoning Models
  4. Knowledge Distillation for Small Language Models
  5. Question-Answering and Problem Decomposition
  6. Knowledge Distillation using Problem Decomposition
  7. Rule-based Problem Decomposition
  8. Intermediate Trace Generation for SFT
  9. Experimental Setup
  10. Datasets
  11. CoTemp QA:
  12. Microsoft MARCO QA:
  13. Facebook bAbI QA:
  14. Direct Prompting & SFT Evaluation
  15. Direct Prompting SLMs:
  16. ...and 10 more sections

Figures (3)

  • Figure 1: The construction of SFT dataset w/ verifiable intermediate traces using rule-based problem decomposition on an example from the CoTemp QA dataset.
  • Figure 2: Confusion Matrices for SFT w/ Correct Traces on Llama-3.2-1B-It model (top) and Qwen3-1.7B (bottom) model, showing Final Solution Accuracy (X-axis) vs Trace Accuracy (Y-axis) for the CoTemp QA, Microsoft MARCO QA, and, the Facebook bAbI QA datasets.
  • Figure 3: Confusion Matrices for SFT w/ Inorrect Traces on Llama-3.2-1B-It model (top) and Qwen3-1.7B (bottom) model, showing Final Solution Accuracy (X-axis) vs Trace Accuracy (Y-axis) for the CoTemp QA, Microsoft MARCO QA, and, the Facebook bAbI QA datasets.