Improving Model Alignment Through Collective Intelligence of Open-Source LLMS

TL;DR

The paper addresses the data bottleneck in aligning large language models by introducing Mixture of Agents Alignment (MoAA), which pools diverse open-source LLMs to generate high-quality synthetic data for both supervised fine-tuning and preference optimization. MoAA operates in two stages: MoAA-SFT uses a layered proposer-aggregator ensemble to synthesize training data, and MoAA-DPO uses the ensemble as a reward model to label preference data and drive direct preference optimization. Across AlpacaEval2, Arena-Hard, and MT-Bench, MoAA improves win rates and outperforms single-teacher baselines, demonstrating the power of collective intelligence and data-efficient alignment. The work also shows a self-improvement opportunity, where fine-tuning on MoAA data enhances the strongest model in the mix, and it reports favorable cost-efficiency compared with using a strong closed-source model for data generation. The approach emphasizes open-source scalability and reproducibility while delivering strong practical gains for open-source LLMs.

Abstract

Building helpful and harmless large language models (LLMs) requires effective model alignment approach based on human instructions and feedback, which necessitates high-quality human-labeled data. Constructing such datasets is often expensive and hard to scale, and may face potential limitations on diversity and generalization. To address these challenges, we introduce Mixture of Agents Alignment (MoAA), that leverages the collective strengths of various language models to provide high-quality data for model alignment. By employing MoAA, we enhance both supervised fine-tuning and preference optimization, leading to improved performance compared to using a single model alone to generate alignment data (e.g. using GPT-4o alone). Evaluation results show that our approach can improve win rate of LLaMA-3.1-8B-Instruct from 19.5 to 48.3 on Arena-Hard and from 22.33 to 57.23 on AlpacaEval2, highlighting a promising direction for model alignment through this new scalable and diverse synthetic data recipe. Furthermore, we demonstrate that MoAA enables a self-improvement pipeline, where models finetuned on MoA-generated data surpass their own initial capabilities, providing evidence that our approach can push the frontier of open-source LLMs without reliance on stronger external supervision. Data and code will be released.

Figures (5)

• Figure 1: SFT results using different models to generate synthetic data. Baseline is the original LLaMA-3.1-8B-Instruct model. Random Teacher means we distill from datasets labeled by one of the five LLMs randomly used in our MoA setup. Combined Teacher means we distill from datasets labeled by five LLMs combined used in our MoA setup (five times data). More details in \ref{['subsec:moaa_sft']}.
• Figure 2: Two-stage Mixture of Agents Alignment to enhance the target model performance.
• Figure 3: The architecture of Mixture-of-Agents wang2024mixtureofagentsenhanceslargelanguage. This example showcases 3 MoA layers where the first layer has three proposers, the second layer has three aggregators that also serve as proposers in the next layer, and the last layer has one aggregator.
• Figure 4: Model performances comparison by SFT on the data generated by single models and MoA. All models are tuned on the original Llama-3.1-8B-Instruct. The x-axis shows the teacher's original performance for each benchmark, whereas the y-axis presents the performance of the Llama-3.1-8B-Instruct model fine-tuned by the corresponding teacher model. We use UF + UC as the dataset for all experiments. The dashed red line indicates the original Llama-3.1-8B-Instruct performance.
• Figure 5: Performance comparison of models using different DPO settings. MoA-OnPolicy uses the SFT model to generate on-policy responses in a MoA style, with the SFT model as the aggregator and unchanged proposers. MoA-OffPolicy uses the MoA architecture in stage 1 to generate responses.