Weighted-Reward Preference Optimization for Implicit Model Fusion

TL;DR

This work tackles the challenge of leveraging heterogeneous open-source LLMs to enhance a target LLM without the heavy overhead of vocabulary alignment or distribution merging. It introduces implicit model fusion via Weighted-Reward Preference Optimization, a progressive adaptation strategy that blends internal rewards from source and target responses through a fusion coefficient $\alpha$ and a weighted reward $r(x, \mathtt{y}_w)$. The method builds on Direct Preference Optimization and derives a WRPO objective that mitigates distributional shifts while enabling scalable fusion across multiple source models. Empirical results on MT-Bench, AlpacaEval-2, and Arena-Hard show WRPO consistently outperforming explicit fusion and other preference baselines, with notable gains on AlpacaEval-2 and competitive performance relative to much larger collective systems. The approach offers a practical, efficient pathway to amplify an instruction-following model by assimilating diverse source capabilities.

Abstract

While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.

Figures (6)

• Figure 1: Distribution deviations between responses from heterogeneous source LLMs and the LLaMA3-8B-Instruct target LLM before (a) and after (b) DPO fine-tuning, with the prompts from Ultrafeedback Cui2024UltraFeedbackBL as input. Subfigure (c) shows the results ($\pi_\text{DPO-off}$) of preference optimization with this deviated preference dataset, compared to the results ($\pi_\theta$) from directly applying the target model and those ($\pi_\text{DPO-on}$) from DPO fine-tuning on un-deviated preference data sampled from the target model.
• Figure 2: Overview of our proposed WRPO for implicit model fusion.
• Figure 3: Internal reward dynamics on Target-SFT model under different preference optimization setups. (a) DPO-on: DPO training on on-policy preference pairs $(x, y_{w_t}, y_l)$. (b) DPO-hybrid: DPO training on hybrid-policy preference pairs $(x, y_{w_s}, y_l)$. (c) WRPO $\alpha=0.5$: WRPO training with $\alpha$ increasing from 0 to 0.5.
• Figure 4: Results of ablation studies for our WRPO method on AlpacaEval-2, utilizing the length-controlled win rate metric.
• Figure 5: AlpacaEval-2 length-controlled win rate and hybrid-policy internal reward accuracy under different fusion coefficient $\alpha$ settings.