Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMOTExT: SMOTE meets Large Language Models

Mateusz Bystroński, Mikołaj Hołysz, Grzegorz Piotrowski, Nitesh V. Chawla, Tomasz Kajdanowicz

TL;DR

SMOTExT adapts SMOTE to NLP by encoding two texts into embeddings and linearly interpolating them in latent space to produce a novel vector $e_{new}$, which is then decoded into text using xRAG: $e_{new}=\lambda e_i+(1-\lambda)e_j$. This approach leverages a fixed retriever and a trainable modality bridge to generate fluent, contextually blended text without retraining the core models, enabling data augmentation in low-resource settings and potential privacy-preserving data synthesis. Preliminary experiments on 20 Newsgroups show that augmenting real data with SMOTExT improves macro and weighted F1, and that synthetic data alone can approach real-data performance, highlighting the method’s utility for few-shot and privacy-constrained scenarios. The work discusses limitations such as out-of-distribution risk and the need for broader quantitative evaluation, while outlining future directions like multi-point interpolation and mechanisms for controlling semantic fidelity.

Abstract

Data scarcity and class imbalance are persistent challenges in training robust NLP models, especially in specialized domains or low-resource settings. We propose a novel technique, SMOTExT, that adapts the idea of Synthetic Minority Over-sampling (SMOTE) to textual data. Our method generates new synthetic examples by interpolating between BERT-based embeddings of two existing examples and then decoding the resulting latent point into text with xRAG architecture. By leveraging xRAG's cross-modal retrieval-generation framework, we can effectively turn interpolated vectors into coherent text. While this is preliminary work supported by qualitative outputs only, the method shows strong potential for knowledge distillation and data augmentation in few-shot settings. Notably, our approach also shows promise for privacy-preserving machine learning: in early experiments, training models solely on generated data achieved comparable performance to models trained on the original dataset. This suggests a viable path toward safe and effective learning under data protection constraints.

SMOTExT: SMOTE meets Large Language Models

TL;DR

SMOTExT adapts SMOTE to NLP by encoding two texts into embeddings and linearly interpolating them in latent space to produce a novel vector , which is then decoded into text using xRAG: . This approach leverages a fixed retriever and a trainable modality bridge to generate fluent, contextually blended text without retraining the core models, enabling data augmentation in low-resource settings and potential privacy-preserving data synthesis. Preliminary experiments on 20 Newsgroups show that augmenting real data with SMOTExT improves macro and weighted F1, and that synthetic data alone can approach real-data performance, highlighting the method’s utility for few-shot and privacy-constrained scenarios. The work discusses limitations such as out-of-distribution risk and the need for broader quantitative evaluation, while outlining future directions like multi-point interpolation and mechanisms for controlling semantic fidelity.

Abstract

Data scarcity and class imbalance are persistent challenges in training robust NLP models, especially in specialized domains or low-resource settings. We propose a novel technique, SMOTExT, that adapts the idea of Synthetic Minority Over-sampling (SMOTE) to textual data. Our method generates new synthetic examples by interpolating between BERT-based embeddings of two existing examples and then decoding the resulting latent point into text with xRAG architecture. By leveraging xRAG's cross-modal retrieval-generation framework, we can effectively turn interpolated vectors into coherent text. While this is preliminary work supported by qualitative outputs only, the method shows strong potential for knowledge distillation and data augmentation in few-shot settings. Notably, our approach also shows promise for privacy-preserving machine learning: in early experiments, training models solely on generated data achieved comparable performance to models trained on the original dataset. This suggests a viable path toward safe and effective learning under data protection constraints.
Paper Structure (15 sections, 1 figure, 3 tables)

This paper contains 15 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Data Augmentation.
  4. SMOTE and Interpolation.
  5. xRAG.
  6. Proposed Method
  7. Embedding Text Examples.
  8. Latent Space Interpolation.
  9. Decoding via xRAG.
  10. Example Workflow.
  11. Experiments
  12. Discussion and Conclusions
  13. Limitations
  14. Appendix
  15. Clinical Text Interpolation Case Study

Figures (1)

  • Figure 1: Two textual examples A and B are encoded into embeddings. An interpolated vector SMOTExT is fed into the xRAG model via its modality bridge (projector). Language model then generates a synthetic text that reflects combined features of the inputs.