Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization

TL;DR

The paper tackles the challenge of few-shot intent detection by showing that supervised pre-training can yield an anisotropic PLM feature space. It introduces two isotropization regularizers, CL-Reg (contrastive-based) and Cor-Reg (correlation-matrix-based), and integrates them into the supervised pre-training objective to improve representation isotropy during fine-tuning. Empirical results across BERT and RoBERTa show that these regularizers, especially when combined, produce consistent gains over strong baselines on BANKING77, HINT3, and HWU64, with Cor-Reg typically providing the strongest signal. The work demonstrates that carefully regularizing isotropy during supervised pre-training can meaningfully boost few-shot performance and has potential implications for broader PLM-based NLU tasks.

Abstract

It is challenging to train a good intent classifier for a task-oriented dialogue system with only a few annotations. Recent studies have shown that fine-tuning pre-trained language models with a small amount of labeled utterances from public benchmarks in a supervised manner is extremely helpful. However, we find that supervised pre-training yields an anisotropic feature space, which may suppress the expressive power of the semantic representations. Inspired by recent research in isotropization, we propose to improve supervised pre-training by regularizing the feature space towards isotropy. We propose two regularizers based on contrastive learning and correlation matrix respectively, and demonstrate their effectiveness through extensive experiments. Our main finding is that it is promising to regularize supervised pre-training with isotropization to further improve the performance of few-shot intent detection. The source code can be found at https://github.com/fanolabs/isoIntentBert-main.

Figures (7)

• Figure 1: Illustration of our proposed regularized supervised pre-training. SPT denotes supervised pre-training (fine-tuning an off-the-shelf PLM on a set of labeled utterances), which makes the feature space more anisotropic. CL-Reg and Cor-Reg are designed to regularize SPT and increase the isotropy of the feature space, which leads to better performance on few-shot intent detection.
• Figure 2: The impact of contrastive learning on IntentBERT with experiments on HWU64 and BANKING77 datasets. The performance (blue) drops while the isotropy (orange) increases.
• Figure 3: The impact of whitening on IntentBERT with experiments on HWU64 and BANKING77 datasets. Whitening transformation leads to perfect isotropy but has inconsistent effects on the performance.
• Figure 4: Illustration of CL-Reg (contrastive-learning-based regularizer) and Cor-Reg (correlation-matrix-based regularizer). $x_i$ is the $i_\text{th}$ utterance in a batch of size $3$. In (a), $x_i$ is fed to the PLM twice with built-in dropout to produce two different representations of $x_i$: $\mathbf{h}_i$ and $\mathbf{h}_i^{+}$. Positive and negative pairs are then constructed for each $x_i$. For example, $\mathbf{h}_1$ and $\mathbf{h}_1^{+}$ form a positive pair for $x_1$, while $\mathbf{h}_1$ and $\mathbf{h}_2^{+}$, and $\mathbf{h}_1$ and $\mathbf{h}_3^{+}$, form negative pairs for $x_1$. In (b), the correlation matrix is estimated from $\textbf{h}_i$, feature vectors generated by the PLM, and is regularized towards the identity matrix.
• Figure 5: Relation between performance and isotropy. The results are obtained with BERT on $5$-way $2$-shot tasks.