Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, Dzmitry Bahdanau

TL;DR

This work tackles data scarcity in few-shot intent classification by using a prompt-based augmentation strategy that leverages off-the-shelf language models (notably GPT-3 and GPT-J) to generate labeled data for target intents. By focusing prompts on a single seed intent and varying model scale, the approach creates augmented datasets without task-specific fine-tuning, enabling evaluation across four datasets with varying intent granularity. The results show that GPT-generated data can improve performance on well-separated intents (e.g., CLINC150, SNIPS) and that larger models yield larger gains, while close-intent scenarios (HWU64, Banking77) benefit from filtering via a GPT-3 classifier and, when possible, human relabeling. The findings highlight the potential and limitations of LM-driven data augmentation for intent classification and suggest practical filtering and relabeling steps to enhance data quality and downstream accuracy, with attention to ethical considerations.

Abstract

Data augmentation is a widely employed technique to alleviate the problem of data scarcity. In this work, we propose a prompting-based approach to generate labelled training data for intent classification with off-the-shelf language models (LMs) such as GPT-3. An advantage of this method is that no task-specific LM-fine-tuning for data generation is required; hence the method requires no hyper-parameter tuning and is applicable even when the available training data is very scarce. We evaluate the proposed method in a few-shot setting on four diverse intent classification tasks. We find that GPT-generated data significantly boosts the performance of intent classifiers when intents in consideration are sufficiently distinct from each other. In tasks with semantically close intents, we observe that the generated data is less helpful. Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one. We present preliminary evidence that a prompting-based GPT classifier could be helpful in filtering the generated data to enhance its quality.

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

TL;DR

This work tackles data scarcity in few-shot intent classification by using a prompt-based augmentation strategy that leverages off-the-shelf language models (notably GPT-3 and GPT-J) to generate labeled data for target intents. By focusing prompts on a single seed intent and varying model scale, the approach creates augmented datasets without task-specific fine-tuning, enabling evaluation across four datasets with varying intent granularity. The results show that GPT-generated data can improve performance on well-separated intents (e.g., CLINC150, SNIPS) and that larger models yield larger gains, while close-intent scenarios (HWU64, Banking77) benefit from filtering via a GPT-3 classifier and, when possible, human relabeling. The findings highlight the potential and limitations of LM-driven data augmentation for intent classification and suggest practical filtering and relabeling steps to enhance data quality and downstream accuracy, with attention to ethical considerations.

Abstract

Data augmentation is a widely employed technique to alleviate the problem of data scarcity. In this work, we propose a prompting-based approach to generate labelled training data for intent classification with off-the-shelf language models (LMs) such as GPT-3. An advantage of this method is that no task-specific LM-fine-tuning for data generation is required; hence the method requires no hyper-parameter tuning and is applicable even when the available training data is very scarce. We evaluate the proposed method in a few-shot setting on four diverse intent classification tasks. We find that GPT-generated data significantly boosts the performance of intent classifiers when intents in consideration are sufficiently distinct from each other. In tasks with semantically close intents, we observe that the generated data is less helpful. Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one. We present preliminary evidence that a prompting-based GPT classifier could be helpful in filtering the generated data to enhance its quality.
Paper Structure (19 sections, 7 figures, 5 tables)

This paper contains 19 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experimental Setup
  4. Datasets
  5. Setup
  6. Training and Evaluation
  7. Full few-shot.
  8. Partial few-shot.
  9. Experimental Results
  10. Full few-shot.
  11. Partial few-shot.
  12. Analysis
  13. Data generation for close intents.
  14. Can GPT Models Understand Close Intents?
  15. Comparison with GPT3Mix
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Generation Process. Given a seed intent (here, music_likeness) and $K$(=10) available examples for that intent, we construct a prompt following the shown template. Note that the last line of the prompt is incomplete (there is no new line character.) We then feed this prompt to a GPT-3 engine, which generates some completions of the prompt. In this example, red text denotes unfaithful examples and blue text denotes faithful examples. Note: For brevity, we only show ten generated sentences.
  • Figure 2: Partial few-shot validation performance for different GPT-3 models and temperatures. (a) few-shot accuracy, (b) OOS recall of intent classifiers trained on augmented sets, and (c) fidelity measured as the accuracy of the oracle on the augmented sets.
  • Figure 3: Full few-shot validation performance for different GPT-J temperatures on different datasets. (a) few-shot inscope accuracy of intent classifiers trained on augmented sets, and (b) fidelity (oracle accuracy) of augmented sets generated by GPT-J with different temperatures.
  • Figure 4: Distribution of labels as predicted by the oracle for lowest-fidelity intents in Banking77 and HWU64 datasets ("topping_up_by_card" and "music_likeness," respectively). Green areas denote the portion of generated sentences deemed fit by the oracle for the lowest-fidelity intents in the two datasets. Red and Blue areas respectively correspond to the most common and the second most common alternative intent predicted by the oracle.
  • Figure 5: Using GPT-3 as a classifier. Given a triplet of close intents, we mix and shuffle the multiple seed examples available for each of them. Then, we append an incomplete line to the prompt with just the generated sentence and feed it to GPT-3 multiple times. Among the responses, we choose the most generated in-triplet intent as the predicted intent ("music_likeness" in the above example). Note: For brevity, we don't show all the seed examples and predictions.
  • ...and 2 more figures