Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Tabular Data Detection In the Wild

G. Charbel N. Kindji, Elisa Fromont, Lina Maria Rojas-Barahona, Tanguy Urvoy

TL;DR

Detecting synthetic tabular data across diverse tables is challenging due to cross-table shift. The authors propose four table-agnostic detectors with simple preprocessing (three text-based encodings and one column-based encoding) and evaluate them on 14 real tables generated by four models across six protocols. They report that cross-table learning is feasible within restricted datasets but cross-table transfer remains challenging, with Transformer-based detectors achieving AUCs above $0.70$ in most scenarios and the column-based Transformer hitting as high as $0.92$ for TVAE, while cross-table shift degrades performance for several encodings. The findings indicate that more sophisticated encodings and transfer strategies are needed for robust out-of-domain detection, and they outline a plan to build a cross-table detection benchmark platform.

Abstract

Detecting synthetic tabular data is essential to prevent the distribution of false or manipulated datasets that could compromise data-driven decision-making. This study explores whether synthetic tabular data can be reliably identified across different tables. This challenge is unique to tabular data, where structures (such as number of columns, data types, and formats) can vary widely from one table to another. We propose four table-agnostic detectors combined with simple preprocessing schemes that we evaluate on six evaluation protocols, with different levels of ''wildness''. Our results show that cross-table learning on a restricted set of tables is possible even with naive preprocessing schemes. They confirm however that cross-table transfer (i.e. deployment on a table that has not been seen before) is challenging. This suggests that sophisticated encoding schemes are required to handle this problem.

Synthetic Tabular Data Detection In the Wild

TL;DR

Detecting synthetic tabular data across diverse tables is challenging due to cross-table shift. The authors propose four table-agnostic detectors with simple preprocessing (three text-based encodings and one column-based encoding) and evaluate them on 14 real tables generated by four models across six protocols. They report that cross-table learning is feasible within restricted datasets but cross-table transfer remains challenging, with Transformer-based detectors achieving AUCs above in most scenarios and the column-based Transformer hitting as high as for TVAE, while cross-table shift degrades performance for several encodings. The findings indicate that more sophisticated encodings and transfer strategies are needed for robust out-of-domain detection, and they outline a plan to build a cross-table detection benchmark platform.

Abstract

Detecting synthetic tabular data is essential to prevent the distribution of false or manipulated datasets that could compromise data-driven decision-making. This study explores whether synthetic tabular data can be reliably identified across different tables. This challenge is unique to tabular data, where structures (such as number of columns, data types, and formats) can vary widely from one table to another. We propose four table-agnostic detectors combined with simple preprocessing schemes that we evaluate on six evaluation protocols, with different levels of ''wildness''. Our results show that cross-table learning on a restricted set of tables is possible even with naive preprocessing schemes. They confirm however that cross-table transfer (i.e. deployment on a table that has not been seen before) is challenging. This suggests that sophisticated encoding schemes are required to handle this problem.

Paper Structure

This paper contains 21 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Synthetic Data Detection
  4. Tabular Data Generation
  5. Table-agnostic Embeddings and Encodings
  6. Datasets, Encodings and Detectors
  7. Datasets
  8. Real Data:
  9. Synthetic Data:
  10. Table-agnostic Data Encodings
  11. Text-Based Encodings:
  12. Column-Based Encoding:
  13. Detectors
  14. Detection Setups
  15. Detection Without Distribution Shift
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Table-agnostic data encodings used in this study. At the top: Training data with rows from multiple table structures. On the left: Flat-text encoding, where each row is converted to text. On the right: Column-based encoding with padding (represented by hatched cells). "Num. Features" stands for "Numerical Features" and "Cat. Features" stands for "Categorical Features".
  • Figure 2: Column-based Transformer model architecture for m numerical features and n categorical features.
  • Figure 3: Training and validation AUC performance of the Transformer-based detectors trained under cross-table shift setup.