Class Distillation with Mahalanobis Contrast: An Efficient Training Paradigm for Pragmatic Language Understanding Tasks

TL;DR

The paper tackles the challenge of extracting a minority target class—pragmatic language phenomena like sarcasm, metaphors, and sexism—from a heterogeneous background. It introduces Class Distillation (ClaD), a geometry-aware training paradigm that combines a Mahalanobis-distance–based contrast loss with a beta-distribution–driven decision rule to exploit the target class manifold. Across sarcasm, metaphor, and sexism, ClaD achieves competitive or superior performance using small models and single-epoch training, often outperforming large language models in low-resource settings. The findings highlight that exploiting data geometry can yield substantial efficiency gains and robust performance, offering a practical path for deploying pragmatic language understanding systems with limited computational resources.

Abstract

Detecting deviant language such as sexism, or nuanced language such as metaphors or sarcasm, is crucial for enhancing the safety, clarity, and interpretation of online social discourse. While existing classifiers deliver strong results on these tasks, they often come with significant computational cost and high data demands. In this work, we propose \textbf{Cla}ss \textbf{D}istillation (ClaD), a novel training paradigm that targets the core challenge: distilling a small, well-defined target class from a highly diverse and heterogeneous background. ClaD integrates two key innovations: (i) a loss function informed by the structural properties of class distributions, based on Mahalanobis distance, and (ii) an interpretable decision algorithm optimized for class separation. Across three benchmark detection tasks -- sexism, metaphor, and sarcasm -- ClaD outperforms competitive baselines, and even with smaller language models and orders of magnitude fewer parameters, achieves performance comparable to several large language models (LLMs). These results demonstrate ClaD as an efficient tool for pragmatic language understanding tasks that require gleaning a small target class from a larger heterogeneous background.

Figures (8)

• Figure 1: The minority target class representing deviant language () versus a highly diverse and heterogeneous non-target class of everything else (). This t-SNE maaten2008tsne visualization (where lighter shades indicate instances located further away in 3-D) displays a representative sample from the "Call me sexist but …" corpus samory2021sexism.
• Figure 2: Comparison of ClaD across three detection tasks (from top to bottom) -- (a) sarcasm, (b) metaphors, and (c) sexism -- against four transfer learning baseline results where Transformer-based models are fine-tuned on task-specific data: (from left to right) ALBERT, DistilBERT, SimCSE, and XLNet.
• Figure 3: Comparison of 5-shot evaluation of a suite of nine large language models (left to right): Llama2, Llama3, Phi2, Phi3, Mistral-7B, Falcon, Qwen2, GPT-2, and OPT, against ClaD's single-epoch training (rightmost).
• Figure 4: Comparison of ClaD across the three detection tasks against nine large language models (LLMs) in a limited data regime. The LLMs are trained on 100 instances over 10 epochs. Results shown for the target class are: (top) the $F_1$ scores; and (bottom) the false positive rates (FPR) for the best-performing LLM.
• Figure 5: Comparison of ClaD across the three detection tasks against LLMs, with identical training data: all models utilize the entire training set for a single epoch. $F_1$ scores (top) show ClaD being competitive with most LLMs, and outperforming a few others, while false positive rates (FPR) (bottom) show ClaD remaining superior to the LLMs.