Scalable Oversight for Superhuman AI via Recursive Self-Critiquing

TL;DR

The paper tackles the problem of aligning superhuman AI when direct human supervision becomes infeasible by proposing recursive self-critiquing as a scalable oversight paradigm. It hypothesizes that critique of critique is easier than critique itself, and that this difficulty relation can be extended recursively, enabling multi-level meta-evaluations to supervise AI outputs. Through comprehensive Human-Human, Human-AI, and AI-AI experiments across diverse tasks, the authors demonstrate that higher-order critiques can improve accuracy, confidence, and efficiency relative to direct evaluation or simple voting baselines, though effectiveness can depend on relative model capabilities. The work suggests a feasible pathway toward scalable oversight as AI systems continue to surpass human abilities, while acknowledging current limitations and the need for further improvements in automatic critique capabilities and supervision reliability.

Abstract

As AI capabilities increasingly surpass human proficiency in complex tasks, current alignment techniques, including SFT and RLHF, face fundamental challenges in ensuring reliable oversight. These methods rely on direct human assessment and become impractical when AI outputs exceed human cognitive thresholds. In response to this challenge, we explore two hypotheses: (1) \textit{Critique of critique can be easier than critique itself}, extending the widely-accepted observation that verification is easier than generation to the critique domain, as critique itself is a specialized form of generation; (2) \textit{This difficulty relationship holds recursively}, suggesting that when direct evaluation is infeasible, performing higher-order critiques (e.g., critique of critique of critique) offers a more tractable supervision pathway. We conduct Human-Human, Human-AI, and AI-AI experiments to investigate the potential of recursive self-critiquing for AI supervision. Our results highlight recursive critique as a promising approach for scalable AI oversight.

Figures (14)

• Figure 1: Overview of the recursive critique framework. Starting from response generation for a given question, each subsequent level performs pair-wise evaluation of outputs from the previous level, forming a recursive critique chain. $C^1$ denotes Critique, $C^2$ denotes Critique of Critique, $C^3$ denotes Critique of Critique of Critique.
• Figure 2: PR scores with a fixed 7B critic model and supervised models of varying sizes.
• Figure 3: PR scores with a fixed 7B supervised model and critic models of varying sizes.
• Figure 4: The relative accuracy improvement of critique and recursive critique stages compared to the response stage. Scores are averaged across all datasets. The improvement is calculated as $\exp(\text{Acc}_\text{stage} - \text{Acc}_\text{response})$, where samples are grouped according to their response accuracy levels.
• Figure 5: AI generartion template in Response Stage