QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models

TL;DR

The paper tackles the challenge of deploying resource-efficient language models without sacrificing reasoning quality by distilling contrastive knowledge from large LLMs. It introduces QCRD, which combines temperature-driven generation of diverse positive rationales with a self-adversarial mechanism for negative rationales, all guided by an online discriminator that weights rationale quality in a many-to-one contrastive loss. The approach is validated on four reasoning benchmarks with two student sizes (T5-base and T5-small), showing consistent gains over strong baselines and enabling higher-quality rationale generation in smaller models. The work highlights practical implications for efficient reasoning systems, while noting training cost and prompt quality as important factors for future optimization.

Abstract

The deployment of large language models (LLMs) faces considerable challenges concerning resource constraints and inference efficiency. Recent research has increasingly focused on smaller, task-specific models enhanced by distilling knowledge from LLMs. However, prior studies have often overlooked the diversity and quality of knowledge, especially the untapped potential of negative knowledge. Constructing effective negative knowledge remains severely understudied. In this paper, we introduce a novel framework called quality-guided contrastive rationale distillation aimed at enhancing reasoning capabilities through contrastive knowledge learning. For positive knowledge, we enrich its diversity through temperature sampling and employ self-consistency for further denoising and refinement. For negative knowledge, we propose an innovative self-adversarial approach that generates low-quality rationales by sampling previous iterations of smaller language models, embracing the idea that one can learn from one's own weaknesses. A contrastive loss is developed to distill both positive and negative knowledge into smaller language models, where an online-updating discriminator is integrated to assess qualities of rationales and assign them appropriate weights, optimizing the training process. Through extensive experiments across multiple reasoning tasks, we demonstrate that our method consistently outperforms existing distillation techniques, yielding higher-quality rationales.

Figures (7)

• Figure 1: Comparison between previous methods and our proposed method, where circle points denote rationales, and colors of the circle points correspond to rationale types, and shades of darker indicate higher qualities. The "align" means minimizing the distance between rationales, while the "repel" means maximizing the distance.
• Figure 1: A case of the j-iteration-before-model for the negative rationale generator.
• Figure 2: Illustration of the proposed quality-guided contrastive rationale distillation for distilling contrastive knowledge from teacher models into the student model. Fig.a represents our multi-task framework, i.e., the main prediction label task and additional rationale task. Fig.b represents generation of contrastive rationales for distillation. Fig.c represents details about the quality-guided contrastive rationale loss, and CE denotes the cross-entropy.
• Figure 3: A case of the prompt and rationale output.
• Figure 4: Comparisons with varying sizes of training datasets on the T5-base model for four benchmarks.