Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TabPFGen -- Tabular Data Generation with TabPFN

Junwei Ma, Apoorv Dankar, George Stein, Guangwei Yu, Anthony Caterini

TL;DR

This work introduces TabPFGen, an energy-based generator that repurposes a pretrained TabPFN to synthesize tabular data without further training. By defining a class-conditional energy E(x|y) = -f(x)[y] and sampling via stochastic gradient Langevin dynamics, TabPFGen leverages TabPFN’s in-context discriminative power for generation. Across 18 OpenML-CC18 datasets, it improves downstream predictive performance through data augmentation, aids class balancing, and enhance imputation quality, outperforming several baselines. The approach demonstrates that leveraging discriminative priors as generative energies can effectively address tabular data generation, with practical benefits and clear avenues for scaling with advancing transformer models. Limitations include current input-size constraints and focus on numerical data, which future transformer advances may alleviate.

Abstract

Advances in deep generative modelling have not translated well to tabular data. We argue that this is caused by a mismatch in structure between popular generative models and discriminative models of tabular data. We thus devise a technique to turn TabPFN -- a highly performant transformer initially designed for in-context discriminative tabular tasks -- into an energy-based generative model, which we dub TabPFGen. This novel framework leverages the pre-trained TabPFN as part of the energy function and does not require any additional training or hyperparameter tuning, thus inheriting TabPFN's in-context learning capability. We can sample from TabPFGen analogously to other energy-based models. We demonstrate strong results on standard generative modelling tasks, including data augmentation, class-balancing, and imputation, unlocking a new frontier of tabular data generation.

TabPFGen -- Tabular Data Generation with TabPFN

TL;DR

This work introduces TabPFGen, an energy-based generator that repurposes a pretrained TabPFN to synthesize tabular data without further training. By defining a class-conditional energy E(x|y) = -f(x)[y] and sampling via stochastic gradient Langevin dynamics, TabPFGen leverages TabPFN’s in-context discriminative power for generation. Across 18 OpenML-CC18 datasets, it improves downstream predictive performance through data augmentation, aids class balancing, and enhance imputation quality, outperforming several baselines. The approach demonstrates that leveraging discriminative priors as generative energies can effectively address tabular data generation, with practical benefits and clear avenues for scaling with advancing transformer models. Limitations include current input-size constraints and focus on numerical data, which future transformer advances may alleviate.

Abstract

Advances in deep generative modelling have not translated well to tabular data. We argue that this is caused by a mismatch in structure between popular generative models and discriminative models of tabular data. We thus devise a technique to turn TabPFN -- a highly performant transformer initially designed for in-context discriminative tabular tasks -- into an energy-based generative model, which we dub TabPFGen. This novel framework leverages the pre-trained TabPFN as part of the energy function and does not require any additional training or hyperparameter tuning, thus inheriting TabPFN's in-context learning capability. We can sample from TabPFGen analogously to other energy-based models. We demonstrate strong results on standard generative modelling tasks, including data augmentation, class-balancing, and imputation, unlocking a new frontier of tabular data generation.
Paper Structure (17 sections, 3 equations, 3 figures, 13 tables, 1 algorithm)

This paper contains 17 sections, 3 equations, 3 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Method
  4. Experiments & Analysis
  5. Results
  6. Conclusion
  7. Limitations and Impact
  8. Datasets Details
  9. Experimental Details
  10. Baseline Details
  11. TabPFGen Details
  12. Additional Results
  13. Imputation with TabPFGen
  14. Ablation Studies
  15. Additional Class Balancing Experiment Details
  16. ...and 2 more sections

Figures (3)

  • Figure 1: TabPFGen overview. We backpropagate from the class-conditional energy to $x_\text{synth}$ for gradient generation. CE denotes cross entropy; blue and red arrows represent attention.
  • Figure 2: Contour and marginal density plots of: (a) original two-moons dataset; (b)-(d) synthetic data generated using baseline methods; (e) synthetic data generated by TabPFGen
  • Figure 3: Imputation results. TabPFGen consistently has lower RMSE than the baseline method and also has decreasing RMSE with a decreasing fraction of missing data.