CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP

TL;DR

CoDa tackles data scarcity in NLP by generating targeted synthetic data through constraint-based prompting of instruction-tuned LLMs. It extracts per-instance constraints (lexical, syntactic, semantic/label, length, and concept) and verbalizes them into a natural-language instruction that guides augmentation. Across 11 datasets spanning classification, NER, and QA, CoDa delivers consistent improvements over diverse baselines without any fine-tuning, while offering explicit, tunable control and domain adaptability. The approach, however, incurs higher inference costs and relies on LLMs' ability to follow constraints, suggesting future work in more efficient prompting and enhanced constraint decomposition. Overall, CoDa provides a scalable, training-free path to improve performance in low-resource settings and supports easy adaptation to new domains.

Abstract

We present CoDa (Constrained Generation based Data Augmentation), a controllable, effective, and training-free data augmentation technique for low-resource (data-scarce) NLP. Our approach is based on prompting off-the-shelf instruction-following Large Language Models (LLMs) for generating text that satisfies a set of constraints. Precisely, we extract a set of simple constraints from every instance in the low-resource dataset and verbalize them to prompt an LLM to generate novel and diverse training instances. Our findings reveal that synthetic data that follows simple constraints in the downstream dataset act as highly effective augmentations, and CoDa can achieve this without intricate decoding-time constrained generation techniques or fine-tuning with complex algorithms that eventually make the model biased toward the small number of training instances. Additionally, CoDa is the first framework that provides users explicit control over the augmentation generation process, thereby also allowing easy adaptation to several domains. We demonstrate the effectiveness of CoDa across 11 datasets spanning 3 tasks and 3 low-resource settings. CoDa outperforms all our baselines, qualitatively and quantitatively, with improvements of 0.12%-7.19%. Code is available here: https://github.com/Sreyan88/CoDa

Figures (5)

• Figure 1: Illustration of CoDa. ① For every document in a low-resource NLU dataset $\mathcal{D}$, we extract a set of simple heuristic-based constraints from and ② verbalize them to generate an instruction. ③ This instruction is then fed to an existing instruction-tuned LLM for generating augmentations, which are then added to $\mathcal{D}$ for training a downstream model.
• Figure 2: Augmentation examples on the OTS dataset. All generations are produced in a low-resource setting (500 training examples). CoDa generates augmentations that are coherent, diverse, and label-consistent.
• Figure 3: Augmentation examples on the ATIS dataset. All generations are produced in a low-resource setting (500 training examples).
• Figure 4: Augmentation examples on the CoNLL-2003 dataset. All generations are produced in a low-resource setting (500 training examples).
• Figure 5: Augmentation examples on the SQUAD dataset. All generations are produced in a low-resource setting (500 training examples).