MambaTab: A Plug-and-Play Model for Learning Tabular Data

TL;DR

MambaTab introduces a plug-and-play tabular learning approach built on the structured state-space model variant Mamba. By employing a compact embedding learner, a single Mamba block, and end-to-end training, it achieves strong performance with dramatically fewer parameters than transformer-based baselines. The method demonstrates robust vanilla supervised, feature incremental, and self-supervised learning across eight public datasets, while maintaining scalability and minimal data wrangling. This yields a practical, memory-efficient solution for diverse tabular datasets with broad applicability in real-world settings.

Abstract

Despite the prevalence of images and texts in machine learning, tabular data remains widely used across various domains. Existing deep learning models, such as convolutional neural networks and transformers, perform well however demand extensive preprocessing and tuning limiting accessibility and scalability. This work introduces an innovative approach based on a structured state-space model (SSM), MambaTab, for tabular data. SSMs have strong capabilities for efficiently extracting effective representations from data with long-range dependencies. MambaTab leverages Mamba, an emerging SSM variant, for end-to-end supervised learning on tables. Compared to state-of-the-art baselines, MambaTab delivers superior performance while requiring significantly fewer parameters, as empirically validated on diverse benchmark datasets. MambaTab's efficiency, scalability, generalizability, and predictive gains signify it as a lightweight, "plug-and-play" solution for diverse tabular data with promise for enabling wider practical applications.

Figures (5)

• Figure 1: Schematic diagram of our proposed method (MambaTab). Left: Data preprocessing and representation learning. The embedding learner module is critical to ensure the embedded feature dimension is the same before and after new features are added under incremental learning. Right: Conversion of input data to prediction values via Mamba and a fully connected layer.
• Figure 2: Illustration of feature incremental learning setting. Feature Set${_i}$, $i = 1,2,3$, have incrementally added features. Feature Set${_X}$ represents the set of features for test data.
• Figure 3: Ablation on block and state expansion factors.
• Figure 4: Ablation on embedded representation space.
• Figure 5: Analysis on the of stacked residual Mamba blocks ($\mathcal{M}$).