Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weak-Driven Learning: How Weak Agents make Strong Agents Stronger

Zehao Chen, Gongxun Li, Tianxiang Ai, Yifei Li, Zixuan Huang, Wang Zhou, Fuzhen Zhuang, Xianglong Liu, Jianxin Li, Deqing Wang, Yikun Ban

TL;DR

Weak-Driven Learning reframes post-training optimization by using weaker historical checkpoints as corrective signals to extend learning beyond saturation. The WMSS framework jointly trains a strong agent with a weak reference via logit mixing, guided by a curriculum that emphasizes informative, hard regions and correction of forgetting. Theoretical analysis shows gradient amplification for hard negatives and sustained learning pressure in saturated regimes, while experiments on math reasoning and code generation demonstrate consistent, cost-free inference gains across model scales. This approach offers a data-efficient, self-reinforcing path to stronger agents without increasing inference cost or relying on external teachers.

Abstract

As post-training optimization becomes central to improving large language models, we observe a persistent saturation bottleneck: once models grow highly confident, further training yields diminishing returns. While existing methods continue to reinforce target predictions, we find that informative supervision signals remain latent in models' own historical weak states. Motivated by this observation, we propose WMSS (Weak Agents Can Make Strong Agents Stronger), a post-training paradigm that leverages weak checkpoints to guide continued optimization. By identifying recoverable learning gaps via entropy dynamics and reinforcing them through compensatory learning, WMSS enables strong agents to improve beyond conventional post-training saturation. Experiments on mathematical reasoning and code generation datasets show that agents trained with our approach achieve effective performance improvements, while incurring zero additional inference cost.

Weak-Driven Learning: How Weak Agents make Strong Agents Stronger

TL;DR

Weak-Driven Learning reframes post-training optimization by using weaker historical checkpoints as corrective signals to extend learning beyond saturation. The WMSS framework jointly trains a strong agent with a weak reference via logit mixing, guided by a curriculum that emphasizes informative, hard regions and correction of forgetting. Theoretical analysis shows gradient amplification for hard negatives and sustained learning pressure in saturated regimes, while experiments on math reasoning and code generation demonstrate consistent, cost-free inference gains across model scales. This approach offers a data-efficient, self-reinforcing path to stronger agents without increasing inference cost or relying on external teachers.

Abstract

As post-training optimization becomes central to improving large language models, we observe a persistent saturation bottleneck: once models grow highly confident, further training yields diminishing returns. While existing methods continue to reinforce target predictions, we find that informative supervision signals remain latent in models' own historical weak states. Motivated by this observation, we propose WMSS (Weak Agents Can Make Strong Agents Stronger), a post-training paradigm that leverages weak checkpoints to guide continued optimization. By identifying recoverable learning gaps via entropy dynamics and reinforcing them through compensatory learning, WMSS enables strong agents to improve beyond conventional post-training saturation. Experiments on mathematical reasoning and code generation datasets show that agents trained with our approach achieve effective performance improvements, while incurring zero additional inference cost.
Paper Structure (53 sections, 11 theorems, 81 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 53 sections, 11 theorems, 81 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Post-training Paradigms: From SFT to Knowledge Distillation.
  4. Weak-to-Strong Generalization & Supervision.
  5. Preliminaries
  6. How Can Weak Make Strong Stronger
  7. Weak and strong agent initilization.
  8. Curriculum-Enhanced Data Activation
  9. Entropy dynamics.
  10. Curriculum construction.
  11. Weak-Driven Learning: Joint Training of Weak and Strong
  12. Joint training via logit mixing.
  13. Training Pipeline
  14. Why Can Weak Make Strong Stronger
  15. Why Joint Training Amplifies Gradients in Saturated Regions
  16. ...and 38 more sections

Key Result

Theorem 5.1

If $m_k(z_{\mathrm{weak}}(x)) \le m_k(z_{\mathrm{strong}}(x))$ for all $k \neq y$, then

Figures (4)

  • Figure 1: Paradigm Comparison: Distillation-Based Learning vs. Weak-Driven Learning.
  • Figure 2: Weak-Driven Learning. Overview of WMSS. The framework has three phases: (1) initialization, (2) activate SFT data via curriculum learning, and (3) jointly train weak and strong models to obtain a stronger model; The right panel visualizes the joint-training principle through logit mixing and gradient amplification.
  • Figure 3: Convergence Analysis across Tasks. The training trajectory of WMSS(Qwen3-4B-Base) over 4 epochs.
  • Figure 4: Limits of SFT Logit Growth on Qwen3-4B-Base. After an initial growth phase, both correct and incorrect token logits saturate, preventing standard SFT from further enlarging their margin.

Theorems & Definitions (25)

  • Theorem 5.1: Total negative-mass increase under uniform margin shrinkage
  • Corollary 5.2: Per-token amplification on hard negatives
  • Proposition 5.3: Logit updates on negative and target tokens
  • Definition 2.1: Target margin
  • Lemma 2.2: Softmax log-odds
  • proof
  • Lemma 2.3: Margin mixing
  • proof
  • Definition 2.4: Hard-negative set
  • Lemma 2.5: Relative probability increase on hard negatives
  • ...and 15 more