Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategies

TL;DR

This work investigates whether optimizing LLMs for persuasive debate can generalize better than traditional truth-focused training. It introduces DebateQD, a minimal Quality-Diversity evolutionary framework that evolves diverse debate prompts across seven behavioral families, while holding the debate protocol constant and swapping only the fitness objective between persuasion and truth. Across 7B, 32B, and 72B models on the QuALITY HARD subset, persuasion-optimized strategies substantially reduce train–test generalization gaps (up to 13.94%) and achieve competitive or superior test accuracy, indicating more transferable reasoning under competitive pressure. The findings provide controlled evidence that persuasive signaling, rather than collaborative truth-seeking alone, can foster more robust generalization in LLMs, with implications for scalable evaluation, safety, and future hybrid objective designs.

Abstract

Large Language Models (LLMs) optimized to output truthful answers often overfit, producing brittle reasoning that fails to generalize. While persuasion-based optimization has shown promise in debate settings, it has not been systematically compared against mainstream truth-based approaches. We introduce DebateQD, a minimal Quality-Diversity (QD) evolutionary algorithm that evolves diverse debate strategies across different categories (rationality, authority, emotional appeal, etc.) through tournament-style competitions where two LLMs debate while a third judges. Unlike previously proposed methods that require a population of LLMs, our approach maintains diversity of opponents through prompt-based strategies within a single LLM architecture, making it more accessible for experiments while preserving the key benefits of population-based optimization. In contrast to prior work, we explicitly isolate the role of the optimization objective by fixing the debate protocol and swapping only the fitness function: persuasion rewards strategies that convince the judge irrespective of truth, whereas truth rewards collaborative correctness. Across three model scales (7B, 32B, 72B parameters) and multiple dataset sizes from the QuALITY benchmark, persuasion-optimized strategies achieve up to 13.94% smaller train-test generalization gaps, while matching or exceeding truth optimization's test performance. These results provide the first controlled evidence that competitive pressure to persuade, rather than seek the truth collaboratively, fosters more transferable reasoning skills, offering a promising path for improving LLM generalization.

Figures (7)

• Figure 1: An illustration of DebateQD, our evolutionary debate pipeline. We initialize 35 prompts (7 strategy families $\times$ 5 prompts). In each generation, prompts compete in information‑asymmetric debates to obtain Elo ratings; the bottom 50% are discarded and the top 50% seed a mutator LLM that produces improved variants. Parents and offspring re‑enter the tournament for 20 generations. From the final pool, we select the 15 highest‑Elo strategies for held‑out evaluation and generalization tests.
• Figure 2: Elo rating analysis for persuasion optimization: (a) Example Elo progression across categories over 20 generations for persuasion optimization with 7B parameter model on 100 questions. (b) ELO vs Test Accuracy (Model size: 7B, Questions: 10, Generations: 20) for persuasion-optimized strategies across all categories. The results show a strong positive correlation (r = 0.892) between ELO rating and test accuracy, indicating that tournament-based selection pressure reliably identifies strategies with superior task performance.
• Figure 3: (a) Elo comparison between StaticGen and our method DebateQD (b) Embedding diversity measured with the average pairwise distance. (c) Generalization gap difference (Persuasion minus Truth) across different question set sizes and model scales. Negative values indicate persuasion optimization advantage. Error bars show 95% confidence intervals. (d) Training vs test accuracy comparison between persuasion and truth optimization across experimental conditions. Points above the diagonal line indicate better test performance than training performance.
• Figure 4: Example of multi-turn information-asymmetric debate. Two debaters, each with access to a comprehension text document, defend opposite answer choices to a given question (grey box in the figure). Debater 1 follows Strategy 1 (blue) to defend answer 1, and Debater 2 follows Strategy 2 (red) to defend answer 2. The debate runs for $n$ rounds; in each round both debaters present one argument, so each contributes $n$ arguments in total. The judge never sees the passage and decides based only on the debate transcript. In our implementation, the winner is determined by comparing the judge model's probabilities for the two answer choices (based on its log-probs for answer 1 or 2). In this case, the judge is correct because it chose the true answer (1).
• Figure 5: Example of strategy evolution across generations. We show five strategy prompts for the Deception (Manipulation) strategy sampled from generations 0, 5, 15, and 20 out of the 20 total generations. Prompts are truncated for space, but illustrate the trend that strategies become progressively longer and more elaborate over time.