inversedMixup: Data Augmentation via Inverting Mixed Embeddings

TL;DR

In NLP data augmentation, a core challenge is balancing the controllability of embedding-level Mixup with the interpretability of token-level text generation. InversedMixup merges Mixup with LLM inversion by learning an adaptor that aligns the task model's embedding space with the LLM's input space, enabling mixed embeddings to be decoded into human-readable sentences. The method introduces a three-stage training process—adaptor alignment on unlabeled data, supervised warming-up on labeled data, and inverting mixed embeddings—to produce high-quality synthetic samples and study manifold intrusion, proposing hard-labeling as a remedy. Empirical results across multiple datasets and settings show notable improvements in both few-shot and fully supervised scenarios, indicating strong generalization and practical impact for data-efficient NLP modeling.

Abstract

Mixup generates augmented samples by linearly interpolating inputs and labels with a controllable ratio. However, since it operates in the latent embedding level, the resulting samples are not human-interpretable. In contrast, LLM-based augmentation methods produce sentences via prompts at the token level, yielding readable outputs but offering limited control over the generation process. Inspired by recent advances in LLM inversion, which reconstructs natural language from embeddings and helps bridge the gap between latent embedding space and discrete token space, we propose inversedMixup, a unified framework that combines the controllability of Mixup with the interpretability of LLM-based generation. Specifically, inversedMixup adopts a three-stage training procedure to align the output embedding space of a task-specific model with the input embedding space of an LLM. Upon successful alignment, inversedMixup can reconstruct mixed embeddings with a controllable mixing ratio into human-interpretable augmented sentences, thereby improving the augmentation performance. Additionally, inversedMixup provides the first empirical evidence of the manifold intrusion phenomenon in text Mixup and introduces a simple yet effective strategy to mitigate it. Extensive experiments demonstrate the effectiveness and generalizability of our approach in both few-shot and fully supervised scenarios.

Figures (8)

• Figure 1: Illustration of inversedMixup. Mixed embedding with a ratio $\lambda$ between a 'Location'-labeled sentence (green) and a 'Numberic'-labeled sentence (orange) is inverted by the LLM. The reconstructed sentence incorporates semantics from both source inputs.
• Figure 2: Overview of the proposed inversedMixup framework. (a) Adaptor alignment with unlabeled data aligns the task-specific model $M_\theta$ with the LLM $M_\psi$ by training the adaptor $A_\phi$ on large-scale unlabeled data. (b) Adaptor refinement further fine-tunes $M_\theta$ and $A_{\phi^*}$ using labeled target-task data to enhance alignment and task adaptability. (c) Inverting mixed embeddings mixes embeddings from the aligned $M_{\theta^*}$, transforms the mixed embedding into the LLM space via $A_{\phi^{**}}$, and leverages the LLM to invert the mixed embedding into augmented samples $\widetilde{x}$.
• Figure 3: 'Inside' refers to cases where the interpolated sample $\widetilde{x}$ is assigned a label that matches either $x_i$ or $x_j$, indicating it still lies within the original class manifold. 'Outside' refers to cases where $\widetilde{x}$ receives a label unrelated to either original class, reflecting a deviation from the class manifold.
• Figure 4: (a) T-SNE visualization of the embedding of the original and reconstructed sentence. (b) The similarity distribution of different combinations of sentence pairs.
• Figure 5: Comparison of different LLM-based augmentation methods with inversedMixup. $\text{Num}(\widetilde{x})$ representes the number of augmented sample.