Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis

TL;DR

Tabby introduces a transformer-compatible, mixture-of-experts modification that assigns a dedicated expert per data column, enabling more faithful synthesis of tabular and other structured data. Paired with Plain, a simple yet effective Fine-tuning approach on tabular inputs, Tabby achieves near-parity with real data on several tabular datasets and parity on a nested JSON dataset, while enabling smaller models to compete with larger baselines. The work demonstrates strong empirical gains over prior LLM-based and diffusion methods, provides per-column training diagnostics, and shows extensions to general structured modalities beyond tables. This approach lowers the barrier to high-fidelity synthetic structured data, with practical implications for privacy-preserving data sharing and data augmentation across diverse domains.

Abstract

While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.

Figures (8)

• Figure 1: Tabby Multi-Head modifications (right side) compared to an original, Non-Tabby LLM on the left.
• Figure 2: Process for calculating our primary metric, Machine Learning Efficacy (MLE). We train a generative model, which produces a synthetic dataset. Two downstream classifiers are trained: one on the generative model's training data and the other on the synthetic data. Each downstream model is evaluated on real test data. MLE is the difference in downstream models' test-time performance. Higher scores indicate better-quality synthetic data.
• Figure 3: Performance profile curves and AUP scores across computed using the MLE scores on our evaluation tasks. The top performing method is Tabby MH DGPT2 with Plain training.
• Figure 4: The House dataset's target Median House Value column as a function of its most-predictive feature, Median Income. Left to right: synthetic data from Tab-DDPM, the prior best LLM-based method and Plain Tabby MH, followed by the original data distribution. Tabby demonstrates a similar distribution to the real data and to GTT, but without GTT's assumptions (listed in Section \ref{['sec:exp1']}).
• Figure 5: An overview of the Tabby MH modifications for the nested Glaucoma dataset.