Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Generative Models for Tabular Data: Novel Metrics and Benchmarking

Dayananda Herurkar, Ahmad Ali, Andreas Dengel

TL;DR

This work proposes three novel evaluation metrics: FAED, FPCAD, and RFIS, and demonstrates that FAED effectively captures generative modeling issues overlooked by existing metrics.

Abstract

Generative models have revolutionized multiple domains, yet their application to tabular data remains underexplored. Evaluating generative models for tabular data presents unique challenges due to structural complexity, large-scale variability, and mixed data types, making it difficult to intuitively capture intricate patterns. Existing evaluation metrics offer only partial insights, lacking a comprehensive measure of generative performance. To address this limitation, we propose three novel evaluation metrics: FAED, FPCAD, and RFIS. Our extensive experimental analysis, conducted on three standard network intrusion detection datasets, compares these metrics with established evaluation methods such as Fidelity, Utility, TSTR, and TRTS. Our results demonstrate that FAED effectively captures generative modeling issues overlooked by existing metrics. While FPCAD exhibits promising performance, further refinements are necessary to enhance its reliability. Our proposed framework provides a robust and practical approach for assessing generative models in tabular data applications.

Evaluating Generative Models for Tabular Data: Novel Metrics and Benchmarking

TL;DR

This work proposes three novel evaluation metrics: FAED, FPCAD, and RFIS, and demonstrates that FAED effectively captures generative modeling issues overlooked by existing metrics.

Abstract

Generative models have revolutionized multiple domains, yet their application to tabular data remains underexplored. Evaluating generative models for tabular data presents unique challenges due to structural complexity, large-scale variability, and mixed data types, making it difficult to intuitively capture intricate patterns. Existing evaluation metrics offer only partial insights, lacking a comprehensive measure of generative performance. To address this limitation, we propose three novel evaluation metrics: FAED, FPCAD, and RFIS. Our extensive experimental analysis, conducted on three standard network intrusion detection datasets, compares these metrics with established evaluation methods such as Fidelity, Utility, TSTR, and TRTS. Our results demonstrate that FAED effectively captures generative modeling issues overlooked by existing metrics. While FPCAD exhibits promising performance, further refinements are necessary to enhance its reliability. Our proposed framework provides a robust and practical approach for assessing generative models in tabular data applications.
Paper Structure (18 sections, 4 equations, 2 figures, 3 tables)

This paper contains 18 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Approaches for Tabular Data
  4. Qualitative Assessment of Generative Models (Images and Time Series)
  5. Quantitative Evaluation
  6. Fréchet AutoEncoder Distance (FAED)
  7. Fréchet PCA Distance (FPCAD)
  8. Random Forest Inception Score (RFIS)
  9. Other Evaluation Metrics
  10. Experimental Setup
  11. Datasets
  12. Problem Setup
  13. Computing Evaluation Score
  14. Result
  15. Experiment1: Quality Decrease
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of the proposed evaluation metrics for generative models in the tabular data domain. (a) Fréchet AutoEncoder Distance (FAED) computes the similarity between real and generated data distributions using feature representations extracted from a pre-trained autoencoder. (b) Fréchet PCA Distance (FPCAD) follows a similar approach but utilizes PCA-transformed representations instead of autoencoder embeddings, eliminating the need for pre-training. (c) Random Forest Inception Score (RFIS) assesses the quality of generated samples based on entropy differences between conditional and marginal label distributions, drawing inspiration from the Inception Score used in image evaluation.
  • Figure 2: Results of the quality decrease experiment, illustrating the sensitivity of various evaluation metrics to noise in synthetic data. SDV Fidelity remains largely unaffected, indicating insensitivity to noise. FAED exhibits strong sensitivity, with scores deteriorating significantly as noise levels increase, effectively capturing structural disruptions. FPCAD shows a more gradual decline, but its behavior varies across datasets. RFIS scores increase with noise, reflecting reduced classification confidence and diversity. TRTS detects noise at higher levels but remains less effective than RFIS, while TSTR fails to detect noise altogether. These results highlight the varying effectiveness of metrics in assessing generative model robustness against noise-induced degradation.