Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning

Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, Lingpeng Kong

TL;DR

SunGen tackles the critical bottleneck of low-quality PLM-generated data in data-generation-based zero-shot learning by introducing a self-guided, bilevel re-weighting framework that learns per-sample weights without gold labels. The inner loop trains a tiny task model with weighted cross-entropy, while the outer loop uses a noise-robust loss (e.g., reversed cross-entropy) to guide weight learning from a synthetic validation set. The authors provide theoretical guarantees for recovering a noise-free distribution and generalization bounds, and demonstrate substantial empirical gains across eight text classification tasks, with SunGen-LSTM achieving up to 9.8% relative improvement over the ZeroGen baseline. These results show that automatic data cleaning via SunGen can markedly improve the efficiency and reliability of zero-shot learning pipelines without additional human labeling, and the framework can be integrated with other data-generation optimizations.

Abstract

There is a rising interest in further exploring the zero-shot learning potential of large pre-trained language models (PLMs). A new paradigm called data-generation-based zero-shot learning has achieved impressive success. In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs. The main hurdle of this approach is that the synthesized data from PLM usually contains a significant portion of low-quality samples. Fitting on such data will greatly hamper the performance of the task-specific model, making it unreliable for deployment. Previous methods remedy this issue mainly by filtering synthetic data using heuristic metrics(e.g., output confidence), or refining the data with the help of a human expert, which comes with excessive manual tuning or expensive costs. In this paper, we propose a novel noise-robust re-weighting framework SunGen to automatically construct high-quality data for zero-shot classification problems. Our framework features the ability to learn the sample weights indicating data quality without requiring any human annotation. We theoretically and empirically verify the ability of our method to help construct good-quality synthetic datasets. Notably, SunGen-LSTM yields a 9.8% relative improvement than the baseline on average accuracy across eight different established text classification tasks.

Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning

TL;DR

SunGen tackles the critical bottleneck of low-quality PLM-generated data in data-generation-based zero-shot learning by introducing a self-guided, bilevel re-weighting framework that learns per-sample weights without gold labels. The inner loop trains a tiny task model with weighted cross-entropy, while the outer loop uses a noise-robust loss (e.g., reversed cross-entropy) to guide weight learning from a synthetic validation set. The authors provide theoretical guarantees for recovering a noise-free distribution and generalization bounds, and demonstrate substantial empirical gains across eight text classification tasks, with SunGen-LSTM achieving up to 9.8% relative improvement over the ZeroGen baseline. These results show that automatic data cleaning via SunGen can markedly improve the efficiency and reliability of zero-shot learning pipelines without additional human labeling, and the framework can be integrated with other data-generation optimizations.

Abstract

There is a rising interest in further exploring the zero-shot learning potential of large pre-trained language models (PLMs). A new paradigm called data-generation-based zero-shot learning has achieved impressive success. In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs. The main hurdle of this approach is that the synthesized data from PLM usually contains a significant portion of low-quality samples. Fitting on such data will greatly hamper the performance of the task-specific model, making it unreliable for deployment. Previous methods remedy this issue mainly by filtering synthetic data using heuristic metrics(e.g., output confidence), or refining the data with the help of a human expert, which comes with excessive manual tuning or expensive costs. In this paper, we propose a novel noise-robust re-weighting framework SunGen to automatically construct high-quality data for zero-shot classification problems. Our framework features the ability to learn the sample weights indicating data quality without requiring any human annotation. We theoretically and empirically verify the ability of our method to help construct good-quality synthetic datasets. Notably, SunGen-LSTM yields a 9.8% relative improvement than the baseline on average accuracy across eight different established text classification tasks.
Paper Structure (52 sections, 3 theorems, 24 equations, 7 figures, 11 tables, 1 algorithm)

This paper contains 52 sections, 3 theorems, 24 equations, 7 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Prompt-based Zero-Shot Learning
  4. Efficient Zero-Shot Learning via Data Generation
  5. Synthetic Data Generation.
  6. Efficient Training and Inference.
  7. Method
  8. Notations.
  9. Overall Framework.
  10. Noise-robust Loss Functions.
  11. Remark.
  12. Clean Subset Sampling.
  13. Theoretical Analysis
  14. Experiments
  15. Setup
  16. ...and 37 more sections

Key Result

Theorem 1

If Assumption ass:syn_and_clean holds, there exists a $\boldsymbol{w}^*$ such that $\hat{\boldsymbol{\theta}}(\boldsymbol{w}^*) = \boldsymbol{\theta}^*.$ Further with Assumption ass:theta_unique_rob and Property property:noise_tolerance, our method can uniquely return $\boldsymbol{w}^*$ and the resu

Figures (7)

  • Figure 1: Training and testing accuracy of LSTM model trained on synthetic dataset. After training for more epochs, the testing performance of ZeroGen starts to deteriorate significantly, indicating that the model starts to fit the erroneous data.
  • Figure 2: The framework of SunGen. Our bi-level framework learns sample weights $\boldsymbol{w}$ measuring data quality without relying on any human-annotated data. In the inner loop, we train a tiny task model (TAM) with weighted CE loss based on current sample weights, and produce trained TAM parameters $\hat{\boldsymbol{\theta}}(\boldsymbol{w})$; in the outer loop, we adopt a noise-robust loss to guide the learning of $\boldsymbol{w}$ by evaluating $\hat{\boldsymbol{\theta}}(\boldsymbol{w})$ on a synthetic validation set.
  • Figure 3: Histogram of learnt weights in IMDb synthetic dataset (1,000k). The weights are gradually separated as optimization proceeds, indicating SunGen can differentiate high-quality data from erroneous ones.
  • Figure 4: Loss surface visualization of RCE and CE losses. We parameterize the surface with $\alpha$, $\beta$ parameters to vary model parameters and calculate its loss values alongside a 2D grid. In each subplot, the left one visualizes the loss surface of RCE loss; and the right one visualizes the CE loss surface. (a)-(c) are the visualization from different angles; (d) is the loss contour of CE and RCE.
  • Figure 5: Loss curves of model training. Experiments run on the IMDb standard training set. The loss used to train the network is in red, and the other loss used to evaluate the network is in grey. We have run experiments using various learning rates from {1e-2, 1e-3, 1e-4, 1e-5} and the curves show similar trends.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2: Finite Sample Generalization
  • proof
  • proof
  • Theorem 3
  • proof