Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis

Seunghwan An, Gyeongdong Woo, Jaesung Lim, ChangHyun Kim, Sungchul Hong, Jong-June Jeon

TL;DR

MaCoDE addresses synthetic data generation for mixed-type tabular data by reframing Masked Language Modeling as histogram-based conditional density estimation. It discretizes continuous features via a CDF transform and learns conditional densities across arbitrary conditioning sets using a transformer-based classifier, with masking distributed over all subsets to support arbitrary conditioning. Theoretical results bound the error between true and estimated conditionals in total variation distance, and experiments on 10 real datasets show state-of-the-art joint fidelity and downstream ML utility, plus robust handling of missing data and controllable privacy without retraining. The method supports multiple imputations under MAR and related mechanisms, offering a practical, privacy-aware tool for synthetic data with strong empirical performance and clear limitations tied to Lipschitz assumptions.

Abstract

In this paper, our goal is to generate synthetic data for heterogeneous (mixed-type) tabular datasets with high machine learning utility (MLu). Since the MLu performance depends on accurately approximating the conditional distributions, we focus on devising a synthetic data generation method based on conditional distribution estimation. We introduce MaCoDE by redefining the consecutive multi-class classification task of Masked Language Modeling (MLM) as histogram-based non-parametric conditional density estimation. Our approach enables the estimation of conditional densities across arbitrary combinations of target and conditional variables. We bridge the theoretical gap between distributional learning and MLM by demonstrating that minimizing the orderless multi-class classification loss leads to minimizing the total variation distance between conditional distributions. To validate our proposed model, we evaluate its performance in synthetic data generation across 10 real-world datasets, demonstrating its ability to adjust data privacy levels easily without re-training. Additionally, since masked input tokens in MLM are analogous to missing data, we further assess its effectiveness in handling training datasets with missing values, including multiple imputations of the missing entries.

Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis

TL;DR

MaCoDE addresses synthetic data generation for mixed-type tabular data by reframing Masked Language Modeling as histogram-based conditional density estimation. It discretizes continuous features via a CDF transform and learns conditional densities across arbitrary conditioning sets using a transformer-based classifier, with masking distributed over all subsets to support arbitrary conditioning. Theoretical results bound the error between true and estimated conditionals in total variation distance, and experiments on 10 real datasets show state-of-the-art joint fidelity and downstream ML utility, plus robust handling of missing data and controllable privacy without retraining. The method supports multiple imputations under MAR and related mechanisms, offering a practical, privacy-aware tool for synthetic data with strong empirical performance and clear limitations tied to Lipschitz assumptions.

Abstract

In this paper, our goal is to generate synthetic data for heterogeneous (mixed-type) tabular datasets with high machine learning utility (MLu). Since the MLu performance depends on accurately approximating the conditional distributions, we focus on devising a synthetic data generation method based on conditional distribution estimation. We introduce MaCoDE by redefining the consecutive multi-class classification task of Masked Language Modeling (MLM) as histogram-based non-parametric conditional density estimation. Our approach enables the estimation of conditional densities across arbitrary combinations of target and conditional variables. We bridge the theoretical gap between distributional learning and MLM by demonstrating that minimizing the orderless multi-class classification loss leads to minimizing the total variation distance between conditional distributions. To validate our proposed model, we evaluate its performance in synthetic data generation across 10 real-world datasets, demonstrating its ability to adjust data privacy levels easily without re-training. Additionally, since masked input tokens in MLM are analogous to missing data, we further assess its effectiveness in handling training datasets with missing values, including multiple imputations of the missing entries.
Paper Structure (32 sections, 2 theorems, 24 equations, 8 figures, 16 tables, 1 algorithm)

This paper contains 32 sections, 2 theorems, 24 equations, 8 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposal
  4. Classification Target (Discretization)
  5. Masked Conditional Density Estimation
  6. Theoretical Results
  7. With Missing Data
  8. Experiments
  9. Overview
  10. Evaluation Metrics
  11. Results
  12. Conclusions and Limitations
  13. Appendix / supplemental material
  14. Proof of Proposition 1
  15. Proof of Proposition 2
  16. ...and 17 more sections

Key Result

Proposition 1

Under Assumption assump:CDF and assump:data, for all $j \in I_C$, and ${\bf x} \in \mathbb{R}^p$. Here, $\text{TV}(\cdot, \cdot)$ denotes the total variation distance, and $Bias(\theta)$ is defined as: where ${\bf y}_j|{\bf x}_{-j}$ is a random variable having a categorical distribution such that $\Pr({\bf y}_j = l|{\bf x}_{-j}) = \Pr(g({\bf x}; \hat{F})_j = l|{\bf x}_{-j}) = \pi_{jl}^{*}({\bf x

Figures (8)

  • Figure 1: Overall structure of MaCoDE. In this case, the value of the second column is masked (replaced with '0') and predicted.
  • Figure 2: Trade-off between quality and privacy. Left: feature selection performance. Right: DCR. Error bars represent standard errors. See the Appendix for detailed results.
  • Figure 3: Sensitivity analysis with respect to missingness rate using kings dataset is performed for Q2 under MAR. The analysis focuses on machine learning utility. Results are reported as means and standard errors of the mean from 10 repeated experiments, with error bars representing the standard errors.
  • Figure 4: Trade-off between privacy and quality. Left: feature selection performance (synthetic data quality). Right: DCR (privacy preservability). The means and standard errors of the mean across 10 datasets and 10 repeated experiments are reported. Error bars represent the standard errors of the mean.
  • Figure 5: Q2. Sensitivity analysis of machine learning utility according to missingness rate. Machine learning utility is evaluated using kings dataset under four missing mechanisms. The means and standard errors of the mean across 10 repeated experiments are reported. Error bars represent standard errors.
  • ...and 3 more figures

Theorems & Definitions (9)

  • Definition 1
  • Definition 2: Mask distribution
  • Remark 1
  • Proposition 1
  • Proposition 2
  • Remark 2: How does MaCoDE achieve the remarkable performance in feature selection?
  • proof
  • proof
  • Remark 3