FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion

TL;DR

FuseRL tackles the inefficiency of relying on a single best output from source LLMs by introducing a dense, two-stage fusion framework. FuseSFT uses weighted supervision from multiple sources to initialize the target model, while FusePO leverages weighted, multi-source preference signals to optimize alignment with human preferences, compatible with RLOO, DPO, and SimPO. Empirical results across AlpacaEval-2 and Arena-Hard show state-of-the-art performance for 8B LLMs and reduced bias/variance in fusion signals, with strong robustness to model size and source count. The approach highlights the practical value of dense, diverse signals from heterogeneous sources for more reliable and scalable LLM alignment. Overall, FuseRL provides a principled, scalable path to harness the collective strengths of multiple LLMs for improved generalization and user-aligned behavior.

Abstract

Heterogeneous model fusion enhances the performance of LLMs by integrating the knowledge and capabilities of multiple structurally diverse models. However, existing approaches often rely solely on selecting the best output for each prompt from source models, which underutilizes their full potential due to limited source knowledge and results in sparse optimization signals. To address this limitation, we propose FuseRL, a novel two-stage framework comprising FuseSFT and FusePO to maximize the utilization of source LLMs. FuseSFT establishes a robust initialization by integrating the strengths of heterogeneous source models through weighted supervised fine-tuning (SFT) on diverse outputs for each prompt. FusePO optimizes weighted preferences based on the outputs of multiple source models to enable superior alignment performance. Extensive experiments demonstrate the effectiveness of our framework across various preference alignment methods, including RLOO, DPO, and SimPO. Using Llama-3.1-8B-Instruct as the target model, our approach achieves state-of-the-art performance among 8B LLMs on the AlpacaEval-2 and Arena-Hard benchmarks. Further analysis suggests that FuseSFT regularizes the training process to reduce overfitting, while FusePO introduces dense and diverse signals for preference optimization.

Figures (8)

• Figure 1: Effect of using a single (SFT+PO) vs. multiple (FuseRL) source LLMs for each prompt for heterogeneous model fusion on UltraFeedback Cui2024UltraFeedbackBL. Accuracymeng2024simpo measures the ability to accurately distinguish between preferred and dispreferred responses by comparing the average log-probabilities assigned by different fused models. Left: Accuracy for multiple responses generated from a single source model. Right: Accuracy across responses generated by different source models. Compared to directly applying SFT followed by preference optimization (SFT+PO), FuseRL shows superior performance in distinguishing responses, indicating improved alignment with human preferences. More details are provided in Appendix \ref{['app_sec:rw_acc']}.
• Figure 2: Overview of the proposed FuseRL framework. The framework comprises two stages: FuseSFT, which fine-tunes the target model using high-quality responses from diverse source models via a reward-based mechanism to prioritize informative and relevant outputs; and FusePO, which dynamically adjusts weighted preference pair contributions to align the target model with human preferences.
• Figure 3: A t-SNE visualization of prompts from UltraFeedback, AlpacaEval-2, and Arena-Hard. The prompt embeddings are generated using the all-mpnet-base-v2 model and then reduced in dimensionality using t-SNE. While AlpacaEval-2 prompts are distributed relatively evenly across the UltraFeedback distribution, the Arena-Hard prompts show a more pronounced deviation.
• Figure 4: Ablation studies for FuseRL across various preference learning methods. SFT refers to applying supervised fine-tuning on the target model, while FuseSFT operates similarly but utilizes multiple responses. FuseRL combines FuseSFT and FusePO.
• Figure 5: Statistical results of FuseRL compared to the baseline (SFT+DPO) on AlpacaEval-2, with preference scores provided by GPT-4-Preview-1106 using GPT4-Top1 as the reference. Left: Absolute bias and variance. $\text{FuseRL}_\text{DPO}$ achieves reductions in both absolute bias and variance compared to SFT+DPO. Right: Absolute error distribution. $\text{FuseRL}_\text{DPO}$ exhibits a consistently lower and more concentrated error distribution. These results show the effectiveness of FuseRL in reducing bias and variance.