Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalization

TL;DR

Alice reformulates weak-to-strong generalization as proactive learning by eliciting teacher uncertainty and enabling student-generated demonstrations to guide supervision. The framework combines uncertainty-aware teacher guidance with the student’s superior reasoning to produce higher-quality supervision data, and extends this with cascade generalization for large capability gaps. Empirical results across four datasets and two model families show notable gains in knowledge-based, mathematical, and logical reasoning, with mathematical reasoning benefiting particularly (22.62%). The work demonstrates a practical, scalable approach to supervising superhuman LLMs with limited human oversight, enabling more robust knowledge transfer and improved oversight efficiency.

Abstract

The growing capabilities of large language models (LLMs) present a key challenge of maintaining effective human oversight. Weak-to-strong generalization (W2SG) offers a promising framework for supervising increasingly capable LLMs using weaker ones. Traditional W2SG methods rely on passive learning, where a weak teacher provides noisy demonstrations to train a strong student. This hinders students from employing their knowledge during training and reaching their full potential. In this work, we introduce Alice (pro{A}ctive {l}earning w{i}th tea{c}her's D{e}monstrations), a framework that leverages complementary knowledge between teacher and student to enhance the learning process. We probe the knowledge base of the teacher model by eliciting their uncertainty, and then use these insights together with teachers' responses as demonstrations to guide student models in self-generating improved responses for supervision. In addition, for situations with significant capability gaps between teacher and student models, we introduce cascade Alice, which employs a hierarchical training approach where weak teachers initially supervise intermediate models, who then guide stronger models in sequence. Experimental results demonstrate that our method significantly enhances the W2SG performance, yielding substantial improvements in three key tasks compared to the original W2SG: knowledge-based reasoning (+4.0%), mathematical reasoning (+22.62%), and logical reasoning (+12.11%). This highlights the effectiveness of our new W2SG paradigm that enables more robust knowledge transfer and supervision outcome.

Figures (4)

• Figure 1: The comparison between the typical W2SG approach and Alice. While the typical W2SG approach utilizes noisy demonstrations that may contain misleading information to supervise the strong student directly, we probe the knowledge base of the teacher model and take advantage of the strong student model's capabilities to bridge the knowledge gap and generate higher-quality demonstrations for supervision.
• Figure 2: The overview of Alice. We first train weak teachers using self-generated CoT to provide them with task-specific knowledge. Next, we probe the teacher models’ knowledge base by eliciting their uncertainty for each question. Finally, we implement proactive learning, where the student model combines teacher guidance with its existing knowledge base and reasoning capabilities to generate final responses. These responses are subsequently used to supervise and refine the student model itself.
• Figure 3: Our main experimental settings. For Weak/Strong Performance, we directly fine-tune models on ground-truth labels using the last half of training set before evaluation. For both original W2SG and Alice, we first fine-tune teacher models on ground-truth labels using the first half of training set to to equip them with basic task-relevant knowledge, then perform corresponding student supervision using only questions from last half of the training set.
• Figure 4: Case studies of Alice's effectiveness in generating higher-quality supervision signals by eliciting teacher's uncertainty and taking advantages of student's superior capabilities to bridge the potential knowledge gap.