Continual Contrastive Learning on Tabular Data with Out of Distribution
Achmad Ginanjar, Xue Li, Priyanka Singh, Wen Hua
TL;DR
The paper tackles the challenge of out-of-distribution generalization for tabular data by proposing Tabular Continual Contrastive Learning (TCCL), a three-component framework with an Encoder, Decoder, and Learner Head. TCCL combines contrastive representation learning with continual-learning mechanisms, including a Fisher-information-based update and a Learner Head to mitigate catastrophic forgetting, enabling robust predictions under distribution shifts. Across eight tabular datasets and a broad baseline set (14 models), TCCL demonstrates strong performance for both classification and regression on OOD data, often outperforming competitors though traditional GBDT methods remain highly competitive on some tasks (e.g., Adult). The results highlight the practical potential of integrating contrastive learning with continual learning to improve tabular OOD robustness, and point to future work on broader dataset validation and further robustness enhancements.
Abstract
Out-of-distribution (OOD) prediction remains a significant challenge in machine learning, particularly for tabular data where traditional methods often fail to generalize beyond their training distribution. This paper introduces Tabular Continual Contrastive Learning (TCCL), a novel framework designed to address OOD challenges in tabular data processing. TCCL integrates contrastive learning principles with continual learning mechanisms, featuring a three-component architecture: an Encoder for data transformation, a Decoder for representation learning, and a Learner Head. We evaluate TCCL against 14 baseline models, including state-of-the-art deep learning approaches and gradient-boosted decision trees (GBDT), across eight diverse tabular datasets. Our experimental results demonstrate that TCCL consistently outperforms existing methods in both classification and regression tasks on OOD data, with particular strength in handling distribution shifts. These findings suggest that TCCL represents a significant advancement in handling OOD scenarios for tabular data.