Synthetic Feature Augmentation Improves Generalization Performance of Language Models
Ashok Choudhary, Cornelius Thiels, Hojjat Salehinejad
TL;DR
The paper addresses the challenge of generalization under class imbalance in text classification by performing synthetic feature augmentation in embedding space. It develops a suite of embedding-space augmentation techniques, including SMOTE variants, Random Oversampling, and VAEs, to generate synthetic minority embeddings that balance training data. Across binary and multiclass benchmarks, SMOTE-based augmentation consistently improves accuracy on balanced test sets, with VAEs offering complementary strengths. The results demonstrate a practical, model-agnostic strategy to enhance robustness and fairness of language models when data is scarce or imbalanced.
Abstract
Training and fine-tuning deep learning models, especially large language models (LLMs), on limited and imbalanced datasets poses substantial challenges. These issues often result in poor generalization, where models overfit to dominant classes and underperform on minority classes, leading to biased predictions and reduced robustness in real-world applications. To overcome these challenges, we propose augmenting features in the embedding space by generating synthetic samples using a range of techniques. By upsampling underrepresented classes, this method improves model performance and alleviates data imbalance. We validate the effectiveness of this approach across multiple open-source text classification benchmarks, demonstrating its potential to enhance model robustness and generalization in imbalanced data scenarios.