Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes

TL;DR

The paper tackles data scarcity in tabular ML by introducing Curated LLM (CLLM), a framework that couples LLM-based generation with a learning-dynamics–driven curation step to augment small labeled datasets. It generates a large synthetic set $D_{syn}$ from a tiny $D_{train}$ using a frozen LLM with in-context demonstrations, then selects a high-quality subset $D_{curated}$ via confidence and aleatoric-uncertainty metrics computed over learning dynamics in a downstream model. Across seven real-world datasets, especially for $n<100$, CLLM with GPT-4 (and a curation step) achieves superior downstream AUC compared to traditional tabular generators, approaching the oracle upper bound $D_{oracle}$ and exhibiting notable gains for underrepresented subgroups. The work advances data-centric AI by showing how careful generation plus principled curation improves utility in low-data regimes, while also discussing biases, prompting strategies, and the practical implications for LMICs and sensitive domains.

Abstract

Machine Learning (ML) in low-data settings remains an underappreciated yet crucial problem. Hence, data augmentation methods to increase the sample size of datasets needed for ML are key to unlocking the transformative potential of ML in data-deprived regions and domains. Unfortunately, the limited training set constrains traditional tabular synthetic data generators in their ability to generate a large and diverse augmented dataset needed for ML tasks. To address this challenge, we introduce CLLM, which leverages the prior knowledge of Large Language Models (LLMs) for data augmentation in the low-data regime. However, not all the data generated by LLMs will improve downstream utility, as for any generative model. Consequently, we introduce a principled curation mechanism, leveraging learning dynamics, coupled with confidence and uncertainty metrics, to obtain a high-quality dataset. Empirically, on multiple real-world datasets, we demonstrate the superior performance of CLLM in the low-data regime compared to conventional generators. Additionally, we provide insights into the LLM generation and curation mechanism, shedding light on the features that enable them to output high-quality augmented datasets.

Figures (11)

• Figure 1: CLLM uses a small dataset $D_{\mathrm{train}}$ and a frozen black-box LLM to generate a larger synthetic set $D_{\mathrm{syn}}$. The curator computes the learning dynamics of samples in $D_{\mathrm{syn}}$, assessing samples based on their aleatoric uncertainty and predictive confidence, then curates $D_{\mathrm{syn}}$ with the goal that a downstream model trained on the curated $D_{\mathrm{curated}}$ will have improved performance.
• Figure 2: GPT-4 is able to extrapolate to regions of the oracle (true manifold) even where there is no training data covering them, as can be seen by the overlap with the turquoise dots, with the effect more pronounced when $D_{\mathrm{train}}$ is small
• Figure 3: Subgroups with fewest samples in $D_{\mathrm{train}}$ benefit the most from data augmentation, on average.
• Figure 4: Contextual information in the prompt is important for extrapolation.
• Figure 5: $\hat{\eta}$ aligns more with $D_{\mathrm{curated}}$ than $D_{\mathrm{discarded}}$ for each generative model: the curation step keeps high quality samples tailored to the downstream task.

Theorems & Definitions (2)

• Definition 2.1: Average confidence
• Definition 2.2: Aleatoric uncertainty