Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation

TL;DR

This work tackles the challenge of aligning large language models without costly human-annotated data. It introduces Direct Large Model Alignment (DLMA), a three-step method that generates contrastive-response data, scores it with a self-rewarding probability-based metric, and optimizes with a modified Direct Preference Optimization (DPO) objective. Empirical results on Llama2-7B/13B demonstrate DLMA surpassing baselines, including RLHF with human data, while maintaining text quality; GPT-4-based and limited human evaluations corroborate alignment quality. The approach generalizes across multiple datasets and models, offering a practical, data-efficient path toward safer, more helpful LLM behavior.

Abstract

Aligning large language models (LLMs) with human expectations without human-annotated preference data is an important problem. In this paper, we propose a method to evaluate the response preference by using the output probabilities of response pairs under contrastive prompt pairs, which could achieve better performance on LLaMA2-7B and LLaMA2-13B compared to RLAIF. Based on this, we propose an automatic alignment method, Direct Large Model Alignment (DLMA). First, we use contrastive prompt pairs to automatically generate preference data. Then, we continue to evaluate the generated preference data using contrastive prompt pairs and calculate a self-rewarding score. Finally, we use the DPO algorithm to effectively align LLMs by combining this self-rewarding score. In the experimental stage, our DLMA method could surpass the \texttt{RLHF} method without relying on human-annotated preference data.

Figures (5)

• Figure 1: In contrast to the RLAIF method bai2022constitutional for evaluating preference data with text-generation based method, our approach evaluates preference data through the comparison of output probabilities between responses under two contrastive prompts.
• Figure 2: The overall process of the DLMA method contains three critical steps. The initial step involves generating response pairs through contrastive prompts (\ref{['sec:pdg']}). Subsequently, a self-reward mechanism is introduced using contrastive prompts (\ref{['sec:rescore']}). Finally, direct preference optimization is employed using the self-reward data (\ref{['sec:dpo']}).
• Figure 3: A quantitative analysis of preference data generated by Llama2-7B with contrastive prompt pairs across various self-rewarding score ranges and the win rate of outputs from positive prompts relative to negative prompts evaluated with GPT-4 in PKU-SafeRLHF and HH-Harmless datasets.
• Figure 4: On the original PKU-SafeRLHF and HH-Harmless datasets, the distribution of our self-rewarding score and the relationship between GPT-4's preference annotation win rate are different from those of text generated by the model itself (Figure \ref{['fig:self']}). On the original dataset, the self-rewarding score does not effectively evaluate the quality of responses. For a theoretical explanation, please refer to Appendix \ref{['sec:theoretical-analysis']}.
• Figure 5: In the case of using the same prompt to generate preference data with a LLM, evaluate the results using a self-rewarding score. Compared to the results generated using contrastive prompt pairs (Figure \ref{['fig:self']}), the data generated by the LLM using the same prompt also shows a similar trend, but the number distribution trends within different intervals are distinct.