Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents

TL;DR

The paper tackles the challenge of incorporating robust commonsense and multi-hop reasoning into dialogue by distilling chain-of-thought rationales from unreliable LLMs into a dedicated CoT reasoner. It introduces DONUT, a large-scale dataset of dialogue CoT rationales, and DOCTOR, a rationale-based dialogue reasoner trained on DONUT, with alignment filters to ensure context and response relevance. Empirical results show that DOCTOR-grounded dialogue models outperform single-hop baselines and Self-CoT prompts across in-domain and out-of-domain datasets, with strong human preference for DOCTOR-generated responses. The work demonstrates cost-effective, scalable data collection and provides insights into the importance of iterative QA and rationale alignment in improving conversational coherence and informativeness. Collectively, these contributions advance practical, commonsense-aware dialogue agents and open pathways for more reliable reasoning in open-domain conversations.

Abstract

Human-like chatbots necessitate the use of commonsense reasoning in order to effectively comprehend and respond to implicit information present within conversations. Achieving such coherence and informativeness in responses, however, is a non-trivial task. Even for large language models (LLMs), the task of identifying and aggregating key evidence within a single hop presents a substantial challenge. This complexity arises because such evidence is scattered across multiple turns in a conversation, thus necessitating integration over multiple hops. Hence, our focus is to facilitate such multi-hop reasoning over a dialogue context, namely dialogue chain-of-thought (CoT) reasoning. To this end, we propose a knowledge distillation framework that leverages LLMs as unreliable teachers and selectively distills consistent and helpful rationales via alignment filters. We further present DOCTOR, a DialOgue Chain-of-ThOught Reasoner that provides reliable CoT rationales for response generation. We conduct extensive experiments to show that enhancing dialogue agents with high-quality rationales from DOCTOR significantly improves the quality of their responses.

Figures (9)

• Figure 1: Comparison between responses generated via single-hop reasoning and multi-hop reasoning.
• Figure 2: Overview of our framework. We leverage an LLM to collect CoT rationales and apply filters to selectively annotate them. The same dialogue from Figure \ref{['fig.motivating_example']} is used to showcase rationale generation (left) and alignment filtering (middle). The dotted square shows the training of the critic model with counterfactual rationales.
• Figure 3: Results of head-to-head comparison between rationales from DONUT, commonsense annotation from CICERO ghosal-etal-2022-cicero, and Reflect zhou-etal-2022-reflect via human judgment. The y-axis represents the win percentage against other datasets. The differences in all of the categories are statistically significant ($p < 0.05$).
• Figure 4: Results of qualitative analysis. Left shows the proportions of rationales that are aligned with context (r-to-c) or response (r-to-r). Right shows the percentage of coherent responses by rationale alignment.
• Figure 5: Distribution of the helpfulness ratio.