Tabular Incremental Inference

TL;DR

This work defines Tabular Incremental Inference (TabII), a framework that enables trained tabular models to incorporate new columns during inference by formulating the task through information bottleneck theory. It combines LLM-driven Placeholders, a TabPFN v2-based TabAdapter with LoRA and EWC, and Incremental Sample Condensation (MSA and IISA) to maximize $I(Z; Y)$ while minimizing $I(Z; X')$, with mutual information estimated via MINE. Empirically, TabII achieves state-of-the-art performance across eight public datasets, outperforms baselines in both supervised and unsupervised settings, and demonstrates robust handling of incremental attributes and varying placeholder lengths. The approach offers practical benefits for dynamic tabular environments by reducing retraining needs and efficiently integrating new features at inference time.

Abstract

Tabular data is a fundamental form of data structure. The evolution of table analysis tools reflects humanity's continuous progress in data acquisition, management, and processing. The dynamic changes in table columns arise from technological advancements, changing needs, data integration, etc. However, the standard process of training AI models on tables with fixed columns and then performing inference is not suitable for handling dynamically changed tables. Therefore, new methods are needed for efficiently handling such tables in an unsupervised manner. In this paper, we introduce a new task, Tabular Incremental Inference (TabII), which aims to enable trained models to incorporate new columns during the inference stage, enhancing the practicality of AI models in scenarios where tables are dynamically changed. Furthermore, we demonstrate that this new task can be framed as an optimization problem based on the information bottleneck theory, which emphasizes that the key to an ideal tabular incremental inference approach lies in minimizing mutual information between tabular data and representation while maximizing between representation and task labels. Under this guidance, we design a TabII method with Large Language Model placeholders and Pretrained TabAdapter to provide external knowledge and Incremental Sample Condensation blocks to condense the task-relevant information given by incremental column attributes. Experimental results across eight public datasets show that TabII effectively utilizes incremental attributes, achieving state-of-the-art performance.

Figures (6)

• Figure 1: Illustration of tabular incremental inference, where model $f_\theta$ trained on original attributes $X$ is adapted during inference to incorporate incremental attributes $\tilde{X}$, like significant factors recently proposed in Alzheimer's disease prediction: YWHAG in Nat. Hum. Behav. Guo2024 and MI in Nat. Med. Jack2024. The goal is to enhance model performance by effectively integrating the information from these new attributes into the trained model.
• Figure 2: Illustration of Incremental Inference guided by Information Bottleneck theory. The task is reframed as an optimization problem: maximizing mutual information between the tabular representation $Z$ from our adaptation model and the task label $Y$ while minimizing mutual information between the tabular data $X'$ and the tabular representation $Z$.
• Figure 3: Overview of TabII architecture, which consists of two main components inspired by information bottleneck theory to increase $I(Z; Y)$ and decrease $I(X; Z)$: The Placeholder processes the input tabular data, including training samples with appended zeros $[x_i, 0]$ and test samples $x_j'$, splitting them into three pathways and concatenating the outputs from the tabular encoder ($t_i$, $t_j$), TabAdapter ($p_i$, $p_j$) and Large Language Model encoder ($l_i$, $l_j$) to $r_i$ and $r_j$. Incremental Sample Condensation blocks then fuse these modality representations using Multi-head Self-Attention (MSA) and Interior Incremental Sample Attention (IISA) to generate $z_i$ and $z_j$ useful for downstream tasks. $q_i$, $k_i$, and $v_i$ in IISA represent the query, key, and value vectors for the samples, and $d$ is the dimensionality of the key vectors.
• Figure 4: Estimation of mutual information using FT-Trans*, FT-Trans, TabII's Placeholder, and full TabII on three datasets. The left side shows $I(Z; Y)$ while the right shows $I(X; Z)$; results show that TabII generally achieved the largest $I(Z; Y)$ and smaller $I(X; Z)$, representing great incremental learning ability.
• Figure 5: Box plot showing the average comparative performance across eight datasets for each method group, which equals the method accuracy divided by the optimal accuracy where the model is trained on fully supervised datasets with incremental attributes included. Discard represents methods that neglect incremental attributes. Direct refers to using the original model for incremental attributes. The figure shows that TabII consistently delivers excellent performance, averaging 97% of the optimal situation.