Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contrastive Weak-to-strong Generalization

Houcheng Jiang, Junfeng Fang, Jiaxin Wu, Tianyu Zhang, Chen Gao, Yong Li, Xiang Wang, Xiangnan He, Yang Deng

TL;DR

This work tackles the robustness and generalization limitations of weak-to-strong generalization (W2SG) by linking implicit rewards with Contrastive Decoding (CD). It introduces Contrastive Weak-to-Strong Generalization (ConG), a two-stage framework that first uses CD between pre- and post-alignment weak models to generate high-quality training signals (ConG-S), then refines the strong model with Direct Preference Optimization (DPO) using CD-derived preferences. The authors establish a CD–Implicit Reward Equivalence, showing that CD approximately maximizes an implicit log-likelihood ratio reward, and provide empirical evidence that CD samples carry higher implicit rewards than naive sampling. Across two major model families and evaluation benchmarks, ConG consistently improves weak-to-strong alignment and, in self-alignment, surpasses existing baselines while preserving downstream capabilities, suggesting a scalable and robust path toward generalizable AI systems.

Abstract

Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.

Contrastive Weak-to-strong Generalization

TL;DR

This work tackles the robustness and generalization limitations of weak-to-strong generalization (W2SG) by linking implicit rewards with Contrastive Decoding (CD). It introduces Contrastive Weak-to-Strong Generalization (ConG), a two-stage framework that first uses CD between pre- and post-alignment weak models to generate high-quality training signals (ConG-S), then refines the strong model with Direct Preference Optimization (DPO) using CD-derived preferences. The authors establish a CD–Implicit Reward Equivalence, showing that CD approximately maximizes an implicit log-likelihood ratio reward, and provide empirical evidence that CD samples carry higher implicit rewards than naive sampling. Across two major model families and evaluation benchmarks, ConG consistently improves weak-to-strong alignment and, in self-alignment, surpasses existing baselines while preserving downstream capabilities, suggesting a scalable and robust path toward generalizable AI systems.

Abstract

Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.

Paper Structure

This paper contains 33 sections, 20 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Reinforcement Learning from Human Feedback (RLHF)
  4. Direct Preference Optimization (DPO)
  5. CD--Implicit Reward Equivalence
  6. Language Models as Reward Functions
  7. Connection between Contrastive Decoding and Implicit Reward
  8. Empirical Analysis
  9. Contrastive Weak-to-Strong Generalization (ConG)
  10. Experiments
  11. Experimental Setup
  12. Weak-to-Strong Alignment Results (RQ1)
  13. Effect of Model Capability Gap (RQ2)
  14. Impact of $\alpha$ on Weak-to-Strong Alignment (RQ3)
  15. Downstream Task Evaluation (RQ4)
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Overview of our proposed ConG. (a) Paradigm illustration comparing traditional weak-to-strong methods with ConG. (b) Scatter plot showing the correlation between implicit and explicit rewards, together with a comparison of sample rewards from naive sampling and contrastive decoding. (c) Radar chart comparing weak-to-strong methods on AlpacaEval2 (AE) and Arena-Hard (AH); metrics with underlines denote Qwen2.5-7B-Instruct, while those without underlines correspond to Llama3-8B-Instruct. Best viewed in color.
  • Figure 2: Performance comparison of contrastive decoding with different contrastive coefficients $\alpha$ for Llama3.2-3B-Instruct (top row) and Qwen2.5-3B-Instruct (bottom row). (a,d) Relationship between $\alpha$ and implicit/explicit rewards. (b,e) Relationship between $\alpha$ and response length. (c,f) Win-rate matrices showing the proportion of cases where row $\alpha$ outperforms column $\alpha$.
  • Figure 3: Performance of contrastive weak-to-strong generalization. (a) and (b) Results across different weak–strong model combinations for Qwen2.5 (top) and Llama3 (bottom). (c) and (d) Effect of contrastive coefficient $\alpha$ on alignment performance, with error bars indicating standard errors. “Base” refers to the unaligned reference model.