Deep Learning with Tabular Data: A Self-supervised Approach

TL;DR

This work investigates self-supervised training for TabTransformer on tabular data, introducing four input-variant architectures (Vanilla TT, Binned TT, Vanilla-MLP TT, MLP-TT) and evaluating them across three datasets (Adult Census Income, California Housing, Breast Cancer Wisconsin). It employs a masked-input pretraining objective and finetunes on labeled data to assess performance against baselines such as MLP and supervised TabTransformer. Results show that SSL-augmented TabTransformers, particularly the MLP-TT variant, can outperform traditional approaches and reduce reliance on labeled data, albeit with sensitivity to dataset size and input representation. The findings highlight the potential of self-supervised representations to enhance tabular data modeling and offer directions for broader SSL techniques and domain extensions in tabular and multimodal settings.

Abstract

We have described a novel approach for training tabular data using the TabTransformer model with self-supervised learning. Traditional machine learning models for tabular data, such as GBDT are being widely used though our paper examines the effectiveness of the TabTransformer which is a Transformer based model optimised specifically for tabular data. The TabTransformer captures intricate relationships and dependencies among features in tabular data by leveraging the self-attention mechanism of Transformers. We have used a self-supervised learning approach in this study, where the TabTransformer learns from unlabelled data by creating surrogate supervised tasks, eliminating the need for the labelled data. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. To address the challenges faced during the construction of various input settings into the Transformers. Furthermore, a comparative analysis is also been conducted to examine performance of the TabTransformer model against baseline models such as MLP and supervised TabTransformer. The research has presented with a novel approach by creating various variants of TabTransformer model namely, Binned-TT, Vanilla-MLP-TT, MLP- based-TT which has helped to increase the effective capturing of the underlying relationship between various features of the tabular dataset by constructing optimal inputs. And further we have employed a self-supervised learning approach in the form of a masking-based unsupervised setting for tabular data. The findings shed light on the best way to represent categorical and numerical features, emphasizing the TabTransormer performance when compared to established machine learning models and other self-supervised learning methods.

Figures (32)

• Figure 1: Deep Learning Trend from Google Trends
• Figure 2: Supervised Learning
• Figure 3: Self-Supervised Learning
• Figure 4: Self-supervised learning approaches
• Figure 5: Transformer Architecture by authors (transformer)