Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation

TL;DR

This work introduces Distillation Contrastive Decoding (DCD), a method to improve LLM reasoning during inference by fusing Contrastive Chain-of-thought Prompting with distillation techniques. DCD removes the need for an external amateur model by using Dropout and Quantization to distill amateur-like reasoning signals and contrasts them with an expert's valid CoT, formalized as $s=(1+β)s_e-β s_a$. Empirically, DCD achieves significant gains on GSM8K and StrategyQA across multiple base models, outperforming Contrastive Decoding and Contrastive Prompting baselines while achieving efficiency benefits. The paper also analyzes distillation variants, identifying optimal dropout ranges and highlighting that quantization alone may not always help, and discusses limitations and directions for scaling to larger, more capable models and more complex reasoning tasks.

Abstract

We propose a straightforward approach called Distillation Contrastive Decoding (DCD) to enhance the reasoning capabilities of Large Language Models (LLMs) during inference. In contrast to previous approaches that relied on smaller amateur models or analysis of hidden state differences, DCD employs Contrastive Chain-of-thought Prompting and advanced distillation techniques, including Dropout and Quantization. This approach effectively addresses the limitations of Contrastive Decoding (CD), which typically requires both an expert and an amateur model, thus increasing computational resource demands. By integrating contrastive prompts with distillation, DCD obviates the need for an amateur model and reduces memory usage. Our evaluations demonstrate that DCD significantly enhances LLM performance across a range of reasoning benchmarks, surpassing both CD and existing methods in the GSM8K and StrategyQA datasets.

Figures (11)

• Figure 1: An overview of Distillation Contrastive Decoding method. Valid CoT demonstrations as well as the query will be sent to an LLM, while invalid CoT demonstrations and the query will be sent into a distilled version of the model. We will then use this logit information to enhance the reasoning decoding process.
• Figure 2: Comparison between 3 methods: (1) Contrastive Chain-of-thought Prompting, which relies on extensive prefixes incorporating Contrastive Chain-of-thought examples; (2) Contrastive Decoding, which necessitates the availability of a smaller amateur version of the LLM; and (3) Distillation Contrastive Decoding (Ours), conceived to overcome the constraints of the previous methods by incorporating the fundamental principles of both (1) and (2)
• Figure 3: Performance of different Contrastive Chain-of-thought settings discussed in Section \ref{['sec:invalid_cot_design']}. Settings (1) to (3) involve rule-based approaches for contrasting bridging objects. Setting (4) employs a synthetic-based approach, incorporating contrasts in both bridging objects and language templates.
• Figure 4: Illustration of discrepancies among invalid CoT prompts. For more details, see Appendix \ref{['sec:full_prompts']}.
• Figure 5: Relationship between MMLU Score and Improvement on GSM8K. Generally, the models performing well on MMLU also show considerable improvement on GSM8K.