Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding

Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Chonghua Liao, Jian Li, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, Zhilin Yang

TL;DR

Few-shot NLU lacks a unified evaluation protocol, hindering fair progress tracking. The authors propose a Multi-Splits-based evaluation framework that emphasizes dev-test correlation and stability, and they re-evaluate state-of-the-art methods under this framework. Key findings show prior results often overestimate performance, no single method dominates across tasks, and combining complementary approaches nearly closes the gap to fully supervised baselines, especially on smaller models; gains can diminish with larger pretrained models. They release FewNLU, a toolkit implementing the framework and methods, and call for community-wide convergence on evaluation standards and extensions to generation tasks.

Abstract

The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.

FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding

TL;DR

Few-shot NLU lacks a unified evaluation protocol, hindering fair progress tracking. The authors propose a Multi-Splits-based evaluation framework that emphasizes dev-test correlation and stability, and they re-evaluate state-of-the-art methods under this framework. Key findings show prior results often overestimate performance, no single method dominates across tasks, and combining complementary approaches nearly closes the gap to fully supervised baselines, especially on smaller models; gains can diminish with larger pretrained models. They release FewNLU, a toolkit implementing the framework and methods, and call for community-wide convergence on evaluation standards and extensions to generation tasks.

Abstract

The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.

Paper Structure

This paper contains 41 sections, 3 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Summary of Findings
  3. Related Work
  4. Evaluation Framework
  5. Formulation of Evaluation Framework
  6. How to Construct Data Splits
  7. Desiderata: Performance, Correlation, and Stability
  8. Data Split Strategies
  9. Experimental Setup
  10. Main Results and Analysis
  11. Test Performance and Correlation.
  12. Stability w.r.t. the number of runs $K$.
  13. Remark
  14. Re-Evaluation of State-of-the-Art Methods
  15. Few-Shot Methods
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Test performance, correlation and standard deviation along with different $K$ on BoolQ, RTE, and COPA tasks under different strategies. A smooth and stable dot-line indicates the setting is insensitive to the choice of $K$.
  • Figure 2: Architecture of FewNLU.
  • Figure 3: Visualization of few-shot performance over the same hyper-parameter space of ADAPET and PET based on DeBERTa and Multi-Splits. The x-axis is the index of the hyper-parameter combination. We search each task with a learning rate of 1e-5 or 5e-6, max steps of 250 or 500, evaluation ratio of 0.02 or 0.04, and all the available prompt patterns. Therefore, each task has $8N$ hyper-parameter combinations, where $N$ is the number of available prompt patterns, i.e., 6 for BoolQ and RTE, 3 for WiC, and 2 for COPA. The y-axis is the score of each task given a certain hyper-parameter combination.