Hybrid Alignment Training for Large Language Models

TL;DR

A Hybrid Alignment Training (Hbat) approach, based on alternating alignment and modified elastic weight consolidation methods, that yields consistent performance gains over the traditional two-stage alignment training when using both proximal policy optimization and direct preference optimization.

Abstract

Alignment training is crucial for enabling large language models (LLMs) to cater to human intentions and preferences. It is typically performed based on two stages with different objectives: instruction-following alignment and human-preference alignment. However, aligning LLMs with these objectives in sequence suffers from an inherent problem: the objectives may conflict, and the LLMs cannot guarantee to simultaneously align with the instructions and human preferences well. To response to these, in this work, we propose a Hybrid Alignment Training (Hbat) approach, based on alternating alignment and modified elastic weight consolidation methods. The basic idea is to alternate between different objectives during alignment training, so that better collaboration can be achieved between the two alignment tasks.We experiment with Hbat on summarization and dialogue tasks. Experimental results show that the proposed \textsc{Hbat} can significantly outperform all baselines. Notably, Hbat yields consistent performance gains over the traditional two-stage alignment training when using both proximal policy optimization and direct preference optimization.

Figures (5)

• Figure 1: Architecture of Hbat. We introduce the alternating alignment and the modified EWC methods to design Hbat, which enables it to address optimization conflict problem in the process of LLM alignment training. Here, black solid arrows () denote learning from the subsets $\mathcal{D}_\mathrm{IFA}^{n}$ and $\mathcal{D}_\mathrm{HPA}^{n}$ via Eq. \ref{['eq:ifa']} and Eq. \ref{['eq:hpa']}, respectively. Black dashed arrows () denote computing the amount of parameter changes before and after training and blue dashed arrows () denote accumulating the parameter changes resulting from learning all previous subsets (see Section \ref{['sec:alternationAlignmnet']}). IFA: instruction-following alignment; HPA: human-preference alignment.
• Figure 2: Performance of Hbat with different number of dataset splits (i.e.,$N$) and the maximum values of $F$ (i.e.,$F_{max}$) on the dialogue validation set.
• Figure 3: PandaLM score for different sampling temperatures on the LLaMA2-7B model. For each dialogue model, we conduct the generation three times and report the mean score of these generated responses.
• Figure 4: Prompt templates of computing GPT-4 win rates for summarization and dialogue tasks.
• Figure 5: PandaLM score over training steps for the Hbat and traditional two-stage alignment training.