Xwin-LM: Strong and Scalable Alignment Practice for LLMs

TL;DR

Xwin-LM presents a scalable RLHF alignment pipeline for LLMs built from Llama-2, combining supervised finetuning, reward modeling, rejection sampling, and direct preference optimization. The study introduces large multi-turn preference datasets (Xwin-Pair, Xwin-Set) annotated by GPT-4 and trains reward models at 7B, 13B, and 70B, culminating in RS and DPO steps that yield state-of-the-art results among Llama-2–based models on AlpacaEval and MT-bench. Key findings show data quality and RM size drive gains, RS improves ground-truth grounding through top-ranked responses, and DPO, with carefully curated dispreferred samples, stabilizes output without inflating the potential upper bound. The work highlights practical insights for scalable alignment, including sweet spots in candidate counts and the importance of aligning dispreferred data with the policy distribution, while acknowledging limitations in multi-turn capabilities and annotation stability.

Abstract

In this work, we present Xwin-LM, a comprehensive suite of alignment methodologies for large language models (LLMs). This suite encompasses several key techniques, including supervised finetuning (SFT), reward modeling (RM), rejection sampling finetuning (RS), and direct preference optimization (DPO). The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt is linked to 64 unique responses generated by Xwin-LM-SFT and scored by Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate consistent and significant improvements across the pipeline, demonstrating the strength and scalability of Xwin-LM. The repository https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster community research.

Figures (9)

• Figure 1: Performance evolution. The performance evolution on the AlpacaEval (left) and MT-bench (right) benchmarks suggests the strength and scalability for Xwin-LM, which can continuously improve the instruction-following ability on the 7B, 13B, and 70B scales. 'SFT', 'RSFT', and 'DPO' denote supervised finetuning, rejection sampling finetuning, and direct preference optimization, respectively.
• Figure 2: Data scaling in SFT. The performance is exponentially related to the data scale and gradually tends to saturate. The model trained on response from gpt-4 are significantly better than those from gpt-3.5-turbo.
• Figure 3: Comparing RMs on best-of-64 evaluation protocol. Compared to sampling a single response, employing RM enables the selection of high-quality responses from a pool of 64 candidates, and larger RMs can select responses with superior performance, indicating that the capabilities of RM increase with size.
• Figure 4: Best-of-n evaluation. We select the best response from different numbers of samples to evaluate the alignment between Xwin-RMs and the off-the-shelf AI judge. Left y-axis: AlpacaEval winrate judged by gpt-4. Right y-axis: RM score predicted by Xwin-RM-70B.
• Figure 5: Distribution of RM scores of responses at each rank. Responses are sorted by score from highest to lowest, with the horizontal axis indicating the rank of the responses; a smaller rank signifies a higher score. The scores are averaged across all prompts.