Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation

TL;DR

This work investigates how inserting Transformer attention into different parts of a Variational Autoencoder affects synthetic tabular data generation. By evaluating six VAE variants on 57 OpenML CC18 datasets, the study reveals a consistent fidelity–diversity trade-off: Transformer placement in latent and decoder representations tends to increase diversity but reduce fidelity, while encoders and initial input representations preserve fidelity better. A key finding is that Transformer interactions in the decoder often act almost like the identity function due to residual connections and layer normalization, whereas latent-area Transformers produce the most pronounced changes in representations. Overall, while Transformers offer more diverse samples, their impact on downstream ML utility is limited, highlighting the need for task-specific Transformer designs for tabular data. The results underscore the importance of representation-level analysis (via CK A) to understand how attention mechanisms modify feature interactions in VAEs.

Abstract

Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships between features, especially when handling mixed data types. In contrast, Transformers, through their attention mechanism, are better suited for capturing complex feature interactions. In this paper, we empirically investigate the impact of integrating Transformers into different components of a VAE. We conduct experiments on 57 datasets from the OpenML CC18 suite and draw two main conclusions. First, results indicate that positioning Transformers to leverage latent and decoder representations leads to a trade-off between fidelity and diversity. Second, we observe a high similarity between consecutive blocks of a Transformer in all components. In particular, in the decoder, the relationship between the input and output of a Transformer is approximately linear.

Figures (6)

• Figure 1: Left: Illustration of an embedding based VAE architecture. Middle: Encoder and Decoder mappings of the considered models. Each block denotes a feature map inside an encoder/decoder, with the respective input/output dimensions. Arrows denote operations performed on each feature representation, and the dashed rectangles describe the Transformer components detached in the considered methods. Right: The Transformer architecture implemented in this work.
• Figure 2: Aggregated gains in performance as Transformers are added to the studied components for $\alpha$-Precision (top-plots) and $\beta$-Recall (bottom-plots), for the Forward (left-plots) and Backward (right-plots) sequences.
• Figure 3: Aggregated similarities for the considered models for each data bucket. Each bar denotes the average similarity measure over all datasets, obtained between the input and output of a Transformer in the corresponding architecture component for the considered models.
• Figure 4: Similarities between block representations of a Transformer for the churn, adult and credit-approval datasets. Each cell of the heatmap denotes the similarity between two feature representations. Higher similarities have a lighter color. Depending on where a Transformer acts, we follow the naming convention T(Enc, Lat, Dec) to denote a given Transformer, while (in, out, block.i) to denote input, output, and internal block layer representations.
• Figure 5: Heatmap similarities between representations on the considered Transformers for the ELD-VAE model, for the adult dataset. Each cell denotes the similarity between two feature representations inside a Transformer. The higher the similarity, the lighter the color of the cell. The Transformer on the left details the representations we extract to measure similarities presented in the heatmaps.