Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, Jungwook Choi

TL;DR

The paper tackles the gap between cost-efficient quantized LLMs and their conversational performance by identifying token-flipping as a key degradation factor. It introduces Quantization-aware Direct Preference Optimization (QDPO), which constructs a preference-based objective from full-precision and quantized model outputs to align the two regimes without heavy human labeling. Through experiments on Vicuna and Mi:dm across MT-Bench, Vicuna-Eval, and FLASK, QDPO consistently outperforms PTQ baselines, AWQ, and KD, recovering conversation quality and maintaining task performance. The results demonstrate a practical route to deploy efficient quantized LLMs without sacrificing nuanced conversational behavior, with strong multilingual support and compatibility with RLHF-based workflows.

Abstract

The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

TL;DR

The paper tackles the gap between cost-efficient quantized LLMs and their conversational performance by identifying token-flipping as a key degradation factor. It introduces Quantization-aware Direct Preference Optimization (QDPO), which constructs a preference-based objective from full-precision and quantized model outputs to align the two regimes without heavy human labeling. Through experiments on Vicuna and Mi:dm across MT-Bench, Vicuna-Eval, and FLASK, QDPO consistently outperforms PTQ baselines, AWQ, and KD, recovering conversation quality and maintaining task performance. The results demonstrate a practical route to deploy efficient quantized LLMs without sacrificing nuanced conversational behavior, with strong multilingual support and compatibility with RLHF-based workflows.

Abstract

The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.
Paper Structure (29 sections, 2 theorems, 6 equations, 17 figures, 9 tables, 1 algorithm)

This paper contains 29 sections, 2 theorems, 6 equations, 17 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Conversational Ability of LLM
  4. LLM Quantization
  5. Alignment with Human Preferences
  6. Conversational Abilities of Quantized LLMs
  7. Observations
  8. Breakdown Analysis
  9. Why Token-Flipping Happens?
  10. QDPO: Quantization-aware Direct Preference Optimization
  11. Method
  12. Implementation
  13. Experiments
  14. Experimental Settings
  15. Experimental Results: MT-Bench
  16. ...and 14 more sections

Key Result

theorem 1

For any response $y$ in the set of all possible responses $Y$, if $y_1 = \arg\max_{y \in Y} \pi_\text{fp}(y|x)$ and $y_2 = \arg\max_{y \in Y} \pi_\text{q}(y|x)$, then it is guaranteed that $p^*(y_1 \succ y_2) \geq p^*(y_2 \succ y_1)$.

Figures (17)

  • Figure 1: Example responses generated by Mi:dm-7B on 16-bit and 4-bit quantized inference.
  • Figure 2: (a) Breakdown of factors influencing sentence generation in quantized models. (b) Case study on the impact of each factor. The ROUGE-L score is used to measure changes in sentences. More results for ROUGE-1/2 are in Fig. \ref{['fig:rouge-1-2']}. (c-d) Case-wise ROUGE scores in models where W4A16 PTQ is applied with (c) RTN and (d) AWQ.
  • Figure 3: (a) Auto-regressive inference probabilities for baseline and quantized models, token by token. (b) Difference in average probability between top-1 and top-2 tokens per sample (Mi:dm, from MT-Bench). See Fig. \ref{['fig:prob_gap_awq']} for more on the AWQ case.
  • Figure 4: Training dynamics of QDPO showing chosen and rejected rewards (left), and loss (right) across steps.
  • Figure 5: Vicuna-Eval results on Mi:dm.
  • ...and 12 more figures

Theorems & Definitions (3)

  • theorem 1
  • theorem 1
  • proof