Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?

Liang Lin, Feng Xiong, Zengbin Wang, Kun Wang, Junhao Dong, Xuecai Hu, Yong Wang, Xiangxiang Chu

TL;DR

AR-MAP investigates whether alignment knowledge from preference-aligned autoregressive LLMs can be transferred to diffusion LLMs to improve preference alignment without incurring ELBO-based variance. The method derives alignment and diffusion task vectors from AR-LLM fine-tuning differences and applies a scaling factor to absorb this knowledge into DLLMs, guided by a reward-accuracy objective. The authors demonstrate a weight scaling law based on spectral analysis, show a reward-driven search finds near-optimal scaling in most tasks, and report competitive or superior results across six benchmarks, with robust generalization to other model-merge methods. The work provides a practical, data-efficient path to align DLLMs by leveraging existing AR-LLM alignment knowledge, potentially reducing compute compared to DLLM-specific alignment training.

Abstract

Diffusion Large Language Models (DLLMs) have emerged as a powerful alternative to autoregressive models, enabling parallel token generation across multiple positions. However, preference alignment of DLLMs remains challenging due to high variance introduced by Evidence Lower Bound (ELBO)-based likelihood estimation. In this work, we propose AR-MAP, a novel transfer learning framework that leverages preference-aligned autoregressive LLMs (AR-LLMs) as implicit teachers for DLLM alignment. We reveal that DLLMs can effectively absorb alignment knowledge from AR-LLMs through simple weight scaling, exploiting the shared architectural structure between these divergent generation paradigms. Crucially, our approach circumvents the high variance and computational overhead of direct DLLM alignment and comprehensive experiments across diverse preference alignment tasks demonstrate that AR-MAP achieves competitive or superior performance compared to existing DLLM-specific alignment methods, achieving 69.08\% average score across all tasks and models. Our Code is available at https://github.com/AMAP-ML/AR-MAP.

AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?

TL;DR

AR-MAP investigates whether alignment knowledge from preference-aligned autoregressive LLMs can be transferred to diffusion LLMs to improve preference alignment without incurring ELBO-based variance. The method derives alignment and diffusion task vectors from AR-LLM fine-tuning differences and applies a scaling factor to absorb this knowledge into DLLMs, guided by a reward-accuracy objective. The authors demonstrate a weight scaling law based on spectral analysis, show a reward-driven search finds near-optimal scaling in most tasks, and report competitive or superior results across six benchmarks, with robust generalization to other model-merge methods. The work provides a practical, data-efficient path to align DLLMs by leveraging existing AR-LLM alignment knowledge, potentially reducing compute compared to DLLM-specific alignment training.

Abstract

Diffusion Large Language Models (DLLMs) have emerged as a powerful alternative to autoregressive models, enabling parallel token generation across multiple positions. However, preference alignment of DLLMs remains challenging due to high variance introduced by Evidence Lower Bound (ELBO)-based likelihood estimation. In this work, we propose AR-MAP, a novel transfer learning framework that leverages preference-aligned autoregressive LLMs (AR-LLMs) as implicit teachers for DLLM alignment. We reveal that DLLMs can effectively absorb alignment knowledge from AR-LLMs through simple weight scaling, exploiting the shared architectural structure between these divergent generation paradigms. Crucially, our approach circumvents the high variance and computational overhead of direct DLLM alignment and comprehensive experiments across diverse preference alignment tasks demonstrate that AR-MAP achieves competitive or superior performance compared to existing DLLM-specific alignment methods, achieving 69.08\% average score across all tasks and models. Our Code is available at https://github.com/AMAP-ML/AR-MAP.
Paper Structure (20 sections, 1 theorem, 12 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 20 sections, 1 theorem, 12 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. RQ1: Can Weight Transfer from AR-LLMs to DLLMs?
  5. RQ2: Discovery of Weight Scaling Law
  6. RQ3: How to Effectively Scale Task Vectors
  7. Experiments
  8. Experiments Setup
  9. Main Experimental Results
  10. Ablation Study
  11. Generalization Experiment
  12. Related Work
  13. Diffusion Large Language Models
  14. Model Merging.
  15. Conclusion
  16. ...and 5 more sections

Key Result

Proposition 3.1

Let $\tau_{\mathrm{diffusion}}, \tau_{\mathrm{pref}}$ be the task vectors defined in Eq. (eq:6). Given the dominance of the diffusion adaptation magnitude, we assume the condition $\|\tau_{\mathrm{pref}}\|_2 \le \epsilon \|\tau_{\mathrm{diffusion}}\|_2$ holds for a scalar $\epsilon \ll 1$. Under wei

Figures (7)

  • Figure 1: The average performance of AR-MAP and baseline method on all tasks and models.
  • Figure 2: Singular value analysis of task vectors. (a) Singular value spectrum in one MLP layer. (b) Layer-wise maximum singular values where solid/dashed lines denote means and shaded areas represent the min-max range.
  • Figure 3: An overview of AR-MAP. AR-MAP has implemented a new preference alignment method through training AR LLMs and simple but effective weight transfer from AR-LLMs to DLLMs.
  • Figure 4: Analysis of Domain-Specific Alignment Transfer. We visualize the performance gains of the AR teacher compared to the DLLM student across distinct domains. The results highlight that AR-MAP adapts its transfer mechanism.
  • Figure 5: Examples of different models that perform reward accuracy calculation on 4096 training set samples.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Proposition 3.1: Spectral Shadowing in Heterogeneous Task Merging