How DDAIR you? Disambiguated Data Augmentation for Intent Recognition

TL;DR

This work tackles the problem that LLM-based data augmentation for intent recognition can yield ambiguous synthetic utterances that resemble non-target intents. It introduces DDAIR, an iterative framework that combines sentence-transformer embeddings to detect ambiguity via class-centroids and LLM-driven re-generation prompts to disambiguate samples. Evaluated on BANKING77, CLINC150, and MPGT with Mistral 7B and Llama-3-8B, and using multiple encoders, DDAIR reduces ambiguity and yields improvements in downstream Macro-F1, with 2–3 disambiguation steps often being optimal; MPGT benefits most from disambiguation while PVI-based filtering underperforms. The approach highlights the practical trade-off between additional LLM calls and improved data quality, and suggests embedding-space sensitivity as a key consideration for robust augmentation in loosely defined intent spaces.

Abstract

Large Language Models (LLMs) are effective for data augmentation in classification tasks like intent detection. In some cases, they inadvertently produce examples that are ambiguous with regard to untargeted classes. We present DDAIR (Disambiguated Data Augmentation for Intent Recognition) to mitigate this problem. We use Sentence Transformers to detect ambiguous class-guided augmented examples generated by LLMs for intent recognition in low-resource scenarios. We identify synthetic examples that are semantically more similar to another intent than to their target one. We also provide an iterative re-generation method to mitigate such ambiguities. Our findings show that sentence embeddings effectively help to (re)generate less ambiguous examples, and suggest promising potential to improve classification performance in scenarios where intents are loosely or broadly defined.

Figures (11)

• Figure 1: Top: A LLM-generated utterance for the target label country support from the banking domain. The synthetic utterance is semantically similar to the card acceptance intent. Bottom: After one disambiguation iteration, the re-generated utterance shows no ambiguity with respect to any class from the label space.
• Figure 2: Ambiguity ratios of the original generations and re-generations after multiple iterative disambiguation steps (dis-1, dis-2, dis-3). A lower ratio indicates a lower proportion of ambiguous generated utterances. In all scenarios, ambiguity ratios decrease after disambiguation steps. MPGT is the most ambiguous corpus.
• Figure 3: Ambiguity ratio by number of in-context learning examples on Mistral 7B generations for the CLINC150 corpus on multiple embedding spaces.
• Figure 4: Silhouette coefficients of the original Llama-3 8B generations and re-generations after multiple iterative disambiguation steps (dis-1, dis-2, dis-3). A higher coefficient indicates a better inter-cluster and intra-cluster mean distance relation of the utterance in the embedding space. In all scenarios, the coefficients increase after 3 disambiguation steps.
• Figure 5: Prompt template used on all LLM generation experiments. Highlighted text in blue varies according to the expected intent and in-context learning examples.