AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs

TL;DR

This study investigates AI debate as a scalable oversight mechanism for subjective questions by measuring models' prior beliefs and testing how judge personas influence persuasion. It compares sequential and simultaneous debate protocols and a consultancy setting to assess whether debaters align with their own priors or with the judge's persona. The results show a clear tendency for models to defend positions aligned with judge personas, with sequential formats biasing toward the second debater, and that persuasiveness improves when arguing in line with prior beliefs, though pairwise evaluations often rate misaligned arguments as higher quality. These findings inform training signal design for human judges and highlight nuanced persuasion dynamics in human-AI interactions, suggesting both opportunities and risks for real-world alignment and safety.

Abstract

The core premise of AI debate as a scalable oversight technique is that it is harder to lie convincingly than to refute a lie, enabling the judge to identify the correct position. Yet, existing debate experiments have relied on datasets with ground truth, where lying is reduced to defending an incorrect proposition. This overlooks a subjective dimension: lying also requires the belief that the claim defended is false. In this work, we apply debate to subjective questions and explicitly measure large language models' prior beliefs before experiments. Debaters were asked to select their preferred position, then presented with a judge persona deliberately designed to conflict with their identified priors. This setup tested whether models would adopt sycophantic strategies, aligning with the judge's presumed perspective to maximize persuasiveness, or remain faithful to their prior beliefs. We implemented and compared two debate protocols, sequential and simultaneous, to evaluate potential systematic biases. Finally, we assessed whether models were more persuasive and produced higher-quality arguments when defending positions consistent with their prior beliefs versus when arguing against them. Our main findings show that models tend to prefer defending stances aligned with the judge persona rather than their prior beliefs, sequential debate introduces significant bias favoring the second debater, models are more persuasive when defending positions aligned with their prior beliefs, and paradoxically, arguments misaligned with prior beliefs are rated as higher quality in pairwise comparison. These results can inform human judges to provide higher-quality training signals and contribute to more aligned AI systems, while revealing important aspects of human-AI interaction regarding persuasion dynamics in language models.

Figures (22)

• Figure 1: Example of the experimental pipeline. In this case, the model prioritizes the moral norm of ‘do your duty’ over retributive justice. The judge is characterized as vindictive person, inclined toward vigilante justice.
• Figure 2: Differences in the opportunity for argumentative attacks between sequential and simultaneous debates.
• Figure 3: Elo ratings in simultaneous debates, split by aligned vs. misaligned stances.
• Figure 4: Empirical win counts for Debater 2 against the Binomial$(n=290, p=0.5)$ null, showing strong positional bias.
• Figure 5: Proportion of arguments aligned or misaligned with prior beliefs selected by GPT-5-chat.