MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

TL;DR

MACPO tackles weak-to-strong alignment by enabling iterative, mutual learning between weak teachers and a strong student through contrastive preference optimization. It introduces mutual positive behavior augmentation and hard negative behavior construction to continually improve positive behaviors and penalize familiar negatives, respectively. Across HH-RLHF and PKU-SafeRLHF, MACPO improves both strong student and weak teacher performance, with gains amplified as more weak teachers participate. The approach leverages DPO with a two-stage training regime and demonstrates robustness against collapse seen in some self-alignment methods, offering a scalable path for aligning strong LLMs with human values.

Abstract

As large language models (LLMs) are rapidly advancing and achieving near-human capabilities on specific tasks, aligning them with human values is becoming more urgent. In scenarios where LLMs outperform humans, we face a weak-to-strong alignment problem where we need to effectively align strong student LLMs through weak supervision generated by weak teachers. Existing alignment methods mainly focus on strong-to-weak alignment and self-alignment settings, and it is impractical to adapt them to the much harder weak-to-strong alignment setting. To fill this gap, we propose a multi-agent contrastive preference optimization (MACPO) framework. MACPO facilitates weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors while penalizing familiar negative ones. To get this, we devise a mutual positive behavior augmentation strategy to encourage weak teachers and strong students to learn from each other's positive behavior and further provide higher quality positive behavior for the next iteration. Additionally, we propose a hard negative behavior construction strategy to induce weak teachers and strong students to generate familiar negative behavior by fine-tuning on negative behavioral data. Experimental results on the HH-RLHF and PKU-SafeRLHF datasets, evaluated using both automatic metrics and human judgments, demonstrate that MACPO simultaneously improves the alignment performance of strong students and weak teachers. Moreover, as the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds.

Figures (10)

• Figure 1: (a) Naive weak-to-strong alignment reinforces strong students on weak labels generated by weak teachers, but ignores the benefit of iteratively improving the quality of positive behavior and penalizing negative behavior. (b) Self-alignment methods iteratively train strong students on self-generated labels, but may collapse. (c) MACPO facilitates weak teachers and strong students to learn from each other by iteratively reinforcing unfamiliar positive behaviors and penalizing familiar negative ones.
• Figure 2: Effectiveness of MACPO with different numbers of weak teachers. As the number of weak teachers increases, MACPO achieves better weak-to-strong alignment performance through more iteration optimization rounds. Different plots use different data ranges.
• Figure 3: Alignment performance of weak teachers during the iterative optimization process. Different plots use different data ranges.
• Figure 4: Ablation study with different strategies. Different plots use different data ranges.
• Figure 5: Prompts for GPT-4 helpfulness evaluation.