Gradual Vigilance and Interval Communication: Enhancing Value Alignment in Multi-Agent Debates

TL;DR

This work targets value alignment for large language models by framing Multi-Agent Debate (MAD) as a more efficient alternative to dense supervision methods. It introduces Gradual Vigilance and Interval Communication (GVIC), a MAD-based framework where agents with increasing vigilance discuss a question in sparsely connected rounds to balance usefulness $H(r)$ and harmlessness $S(r)$ while reducing communication overhead. The authors provide a theoretical link showing debate outcomes are bounded by the best-performing individual responses, and demonstrate via extensive experiments that GVIC consistently outperforms a single agent and a classical Debate setup across diverse datasets and base-model sizes, with pronounced gains in harmlessness and fraud prevention. The results indicate GVIC's broad applicability and potential for safer, scalable value alignment, and point to future work in multi-modal extensions and reward-based refinements.

Abstract

In recent years, large language models have shown exceptional performance in fulfilling diverse human needs. However, their training data can introduce harmful content, underscoring the necessity for robust value alignment. Mainstream methods, which depend on feedback learning and supervised training, are resource-intensive and may constrain the full potential of the models. Multi-Agent Debate (MAD) offers a more efficient and innovative solution by enabling the generation of reliable answers through agent interactions. To apply MAD to value alignment, we examine the relationship between the helpfulness and harmlessness of debate outcomes and individual responses, and propose a MAD based framework Gradual Vigilance and Interval Communication (GVIC). GVIC allows agents to assess risks with varying levels of vigilance and to exchange diverse information through interval communication. We theoretically prove that GVIC optimizes debate efficiency while reducing communication overhead. Experimental results demonstrate that GVIC consistently outperforms baseline methods across various tasks and datasets, particularly excelling in harmfulness mitigation and fraud prevention. Additionally, GVIC exhibits strong adaptability across different base model sizes, including both unaligned and aligned models, and across various task types.

Figures (5)

• Figure 1: Comparison between the classical Debate framework and the GVIC framework.
• Figure 2: Performance comparison between a single agent, the classical Debate framework, and GVIC across various value alignment tasks. The datasets SAFE-RLHF and Harmless are used to evaluate model harmlessness, Helpful assesses helpfulness, and Red Team measures susceptibility to adversarial attacks. The vertical axis $1-D_\text{WL}^\text{(GVIC)}$ indicates the relative performance of other frameworks compared to GVIC. For example, on the Harmless dataset, if GVIC scores 1, the single agent and Debate framework achieve 53% and 64% of GVIC's performance, respectively. The Win-Loss Differential Index ($D_\text{(WL)}$) is defined in Eq \ref{['eq:DWL']}.
• Figure 3: The overall framework of GVIC. As agents' vigilance increases, their perception of potential harm intensifies, prompting more cautious responses. Through multiple rounds of debate, interspersed with intervals of communication, agents with lower vigilance levels gradually align with the high-vigilance agents' assessments of harmlessness, while high-vigilance agents begin to integrate the low-vigilance agents' evaluations of usefulness. The resulting decision is one that maximizes usefulness while ensuring minimal harm.
• Figure 4: Three MAD communication frameworks: (1) Fully connected communication; (2) Adjacent communication; (3) Interval communication.
• Figure 5: Performance variation of GVIC and the classical Debate framework relative to a single agent on the SAFE-RLHF dataset, as the debate progresses across different base models. $D_{\text{WL}}^{\text{(Single)}}$ represents the Win-Loss Differential Index of the compared frameworks relative to the base model.