Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

TL;DR

This work tackles the challenge of balancing helpfulness, honesty, and harmlessness (3H) in large language models by contrasting data-mix strategies with parameter-level model merging. It reveals that data mixtures incur practical costs and conflicting optimization signals, while merging can better negotiate multi-objective trade-offs. The authors introduce RESM, a Reweighting Enhanced Task Singular Vector Merging method that uses outlier-aware weighting and sparsity-aware rank selection to address preference noise and layer sparsity during merging. Across two backbones and multiple datasets, RESM delivers robust improvements in balanced 3H alignment and is released for public use, marking a significant step toward scalable, ethically constrained LLMs.

Abstract

Achieving balanced alignment of large language models (LLMs) in terms of Helpfulness, Honesty, and Harmlessness (3H optimization) constitutes a cornerstone of responsible AI. Existing methods like data mixture strategies face limitations, including heavy reliance on expert knowledge and conflicting optimization signals. While model merging offers parameter-level conflict-resolution strategies through integrating specialized models' parameters, its potential for 3H optimization remains underexplored. This paper systematically compares the effectiveness of model merging and data mixture methods in constructing 3H-aligned LLMs for the first time, revealing previously overlooked collaborative and conflict relationships among the 3H dimensions and discussing the advantages and drawbacks of data mixture (\textit{data-level}) and model merging (\textit{parameter-level}) methods in mitigating the conflict for balanced 3H optimization. Specially, we propose a novel \textbf{R}eweighting \textbf{E}nhanced task \textbf{S}ingular \textbf{M}erging method, \textbf{RESM}, through outlier weighting and sparsity-aware rank selection strategies to address the challenges of preference noise accumulation and layer sparsity adaptation inherent in 3H-aligned LLM merging. Extensive evaluations can verify the effectiveness and robustness of RESM compared to previous data mixture (2\%-5\% gain) and model merging (1\%-3\% gain) methods in achieving balanced LLM alignment. We release our models through \href{https://huggingface.co/Jinluan}{3H\_Merging} for further investigations.

Figures (10)

• Figure 1: Trade-offs in optimizing LLMs across the 3H objectives.
• Figure 2: (a) illustrates the impact of outlier weighting in model merging when incorporating multi-aligned models. The performance discrepancy before and after processing quantifies the degree of preference noise accumulation; (b) depicts the effective rank required to capture 95% of the total energy in singular values for task vectors derived from 3H training. Larger ranks indicate the necessity of retaining a higher proportion of singular values to preserve task-relevant information.
• Figure 3: Reweighting-Enhanced Task Singular Vector Merging
• Figure 4: Illustration of the 3H Trade-Off, where Helpful Training benefits 3H performance simultaneously, but Honest and Harmless Training weaken each other. The dashed line (Instruct Model) serves as the reference, where rectangles on the left represent the weakening effect, and vice versa.
• Figure 5: Illustration of Training Stage of 3H Optimization, which aims to further enhance LLMs' alignment from three perspectives based on the existing Initially Aligned LLMs.