Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMs

Yafeng Tang, Xiaoou Ding, Jianzhuo Du, Zishuo Yan, Zhuang Ma, Zheng Liang, Zekai Qian, Hongzhi Wang

TL;DR

DATE tackles the challenge of heterogeneity in tabular data by partitioning data into distributionally distinct, high-quality subsets using Distribution-Guiding Rules (DGR). It then employs LLMs with decision-tree reasoning to iteratively generate distribution-specific data, while a Multi-Arm Bandit-based sampling balances data diversity and quality. The framework combines DGR-based prompt design, model-sharing and top-down DGR discovery, a DT-guided generation loop, and principled data selection to outperform state-of-the-art GAN- and LLM-based methods on classification and regression benchmarks, with reduced synthetic data requirements. This approach advances practical synthetic tabular data generation, improving downstream model robustness and enabling improved reasoning in LLMs when fine-tuning data are scarce. The authors provide open-source code for reproducibility.

Abstract

Tabular data generation has become increasingly essential for enabling robust machine learning applications, which require large-scale, high-quality data. Existing solutions leverage generative models to learn original data distributions. However, real-world data are naturally heterogeneous with diverse distributions, making it challenging to obtain a universally good model for diverse data generation. To address this limitation, we introduce Diversity-Aware Tabular data gEnerator (DATE), a framework that (i) prepares high-quality and distributionally distinct examples for in-context learning by effectively partitioning the original heterogeneous data into multiple diverse subsets; (ii) harnesses Large Language Models (LLMs) to explore the diversity of the partitioned distribution with decision tree reasoning as feedback, generating high-quality labeled data for each subset. However, the massive generated data inherently involves a trade-off between diversity and quality. To integrate this issue, existing solutions greedily select the validation-best data. However, we prove that the selection in heterogeneous settings does not possess the greedy-choice property, and design a Multi-Arm Bandit-based sampling algorithm that balances the diversity and quality of generated data. Extensive experiments on tabular classification and regression benchmarks demonstrate that DATE consistently outperforms state-of-the-art GAN-based and LLM-based methods. On average, DATE achieves a 23.75% reduction in error rate with just 100 generated data. Empirically, we demonstrate that data generated by DATE can improve the accuracy of Direct Preference Optimization (DPO) and enhance the reasoning capability of LLMs on the target data. Code is available at https://github.com/windblow32/DATE.

Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMs

TL;DR

DATE tackles the challenge of heterogeneity in tabular data by partitioning data into distributionally distinct, high-quality subsets using Distribution-Guiding Rules (DGR). It then employs LLMs with decision-tree reasoning to iteratively generate distribution-specific data, while a Multi-Arm Bandit-based sampling balances data diversity and quality. The framework combines DGR-based prompt design, model-sharing and top-down DGR discovery, a DT-guided generation loop, and principled data selection to outperform state-of-the-art GAN- and LLM-based methods on classification and regression benchmarks, with reduced synthetic data requirements. This approach advances practical synthetic tabular data generation, improving downstream model robustness and enabling improved reasoning in LLMs when fine-tuning data are scarce. The authors provide open-source code for reproducibility.

Abstract

Tabular data generation has become increasingly essential for enabling robust machine learning applications, which require large-scale, high-quality data. Existing solutions leverage generative models to learn original data distributions. However, real-world data are naturally heterogeneous with diverse distributions, making it challenging to obtain a universally good model for diverse data generation. To address this limitation, we introduce Diversity-Aware Tabular data gEnerator (DATE), a framework that (i) prepares high-quality and distributionally distinct examples for in-context learning by effectively partitioning the original heterogeneous data into multiple diverse subsets; (ii) harnesses Large Language Models (LLMs) to explore the diversity of the partitioned distribution with decision tree reasoning as feedback, generating high-quality labeled data for each subset. However, the massive generated data inherently involves a trade-off between diversity and quality. To integrate this issue, existing solutions greedily select the validation-best data. However, we prove that the selection in heterogeneous settings does not possess the greedy-choice property, and design a Multi-Arm Bandit-based sampling algorithm that balances the diversity and quality of generated data. Extensive experiments on tabular classification and regression benchmarks demonstrate that DATE consistently outperforms state-of-the-art GAN-based and LLM-based methods. On average, DATE achieves a 23.75% reduction in error rate with just 100 generated data. Empirically, we demonstrate that data generated by DATE can improve the accuracy of Direct Preference Optimization (DPO) and enhance the reasoning capability of LLMs on the target data. Code is available at https://github.com/windblow32/DATE.
Paper Structure (25 sections, 7 theorems, 12 equations, 7 figures, 3 tables, 3 algorithms)

This paper contains 25 sections, 7 theorems, 12 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Challenges
  3. Contributions
  4. Overview
  5. Background
  6. Preliminaries
  7. Problem Definition
  8. Solution Overview
  9. DGR-based Prompt Designing
  10. DGR-based Prompt Template
  11. DGR Discovery
  12. Model Sharing Strategy
  13. Top-Down DGR Construction
  14. Distribution-Specific Generation.
  15. DGR-Guided Iterative Data Generator.
  16. ...and 10 more sections

Key Result

Proposition 1

Given the table $T$, an evaluation model $m$, the error threshold $\rho$, DGR $r_1$, if there exists DGR $r_2\vdash r_1$, then $e_1:(m,\rho, r_1,T_{r_1})$ implies $e_2:(m,\rho, r_2,T_{r_2})$.

Figures (7)

  • Figure 1: Two key requirements for heterogeneous data generation (better viewed in color): ①High-Quality Examples and ②Customized Generation for Diverse Distribution. We present a toy example composed of three distributions.
  • Figure 2: Overall framework of DATE (better viewed in color), which consists of two components with three steps: (1) DGR-based Prompt Designing, which partitions heterogeneous data into diverse data distributions through DGR discovery and constructs prompts with DGR-based examples. We present the prompt template at the bottom center. (2) Distribution-Specific Generation, which focuses on a targeted shared model $m_i$ and iteratively generates data with refined DGRs. To integrate generated data, DATE selects diversity-quality balanced synthesized data as the final result.
  • Figure 3: CD diagram for DATE against ablation baselines in terms of accuracy ranking. The significance level $\alpha=0.05$. The top-ranked method is highlighted in red.
  • Figure 4: Analyses of the model sharing strategy across all datasets. Our runtime is presented on a logarithmic scale.
  • Figure 5: DPO experiments using Q&A pairs constructed from classification datasets. We report the error rate of the LLM's responses.
  • ...and 2 more figures

Theorems & Definitions (13)

  • Definition 1: Tabular Data Generation
  • Definition 2: Predicate Space
  • Definition 3: Distribution-Guiding Rule
  • Definition 4: DGR-based Examples
  • Proposition 1: Induction
  • Example 1
  • Proposition 2: Model Sharing
  • Proposition 3: Fusion
  • Proposition 4: Generalization
  • Theorem 1
  • ...and 3 more