Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learning

TL;DR

This study interrogates whether text-serialized Language Models (LMs) can outperform conventional tabular ML approaches. By systematically evaluating a DistilBERT-based TabLM against SOTA baselines across eight datasets and multiple data-curation strategies (feature selection, scaling, and missing-data imputation), the authors reveal that pre-trained LMs do not universally replace traditional solvers like gradient boosting. Key findings show feature selection often improves LM performance, but serialization sensitivity can dramatically shift predictions, and overall LM superiority is not guaranteed, especially on larger or more complex tabular tasks. The work provides code and data for reproducibility and offers practical guidance on when LM-based approaches may be advantageous in real-world tabular ML settings.

Abstract

Recent research has explored how Language Models (LMs) can be used for feature representation and prediction in tabular machine learning tasks. This involves employing text serialization and supervised fine-tuning (SFT) techniques. Despite the simplicity of these techniques, significant gaps remain in our understanding of the applicability and reliability of LMs in this context. Our study assesses how emerging LM technologies compare with traditional paradigms in tabular machine learning and evaluates the feasibility of adopting similar approaches with these advanced technologies. At the data level, we investigate various methods of data representation and curation of serialized tabular data, exploring their impact on prediction performance. At the classification level, we examine whether text serialization combined with LMs enhances performance on tabular datasets (e.g. class imbalance, distribution shift, biases, and high dimensionality), and assess whether this method represents a state-of-the-art (SOTA) approach for addressing tabular machine learning challenges. Our findings reveal current pre-trained models should not replace conventional approaches.

Figures (9)

• Figure 1: Data Curation In Text Serialization: In this work we explore several common data curation strategies used in tabular machine learning and determine whether these practices should be adopted in Language Model technologies.
• Figure 2: Language Model Benchmark: The evaluation of several LM backbones that are benchmarked to be our TabLM.
• Figure 3: An evaluation using various feature scaling methods that include outlier handling. We see very nuanced results and these methods should be applied with some background knowledge of the dataset.
• Figure 4: Logarithm of Odds Sensitivity Analysis: We analyzed the logarithm of odds (logits) by computing the differences between the imputed model logits and the baseline logits to determine how much each sample in the test set deviated from the original logit value. We observed that both imputation strategies significantly altered the logit values, demonstrating the sensitivity of these strategies towards the raw probabilites.
• Figure 5: Inference Time: We observe significant differences in training times, where boosting methods prove to be far more efficient than current language model technologies. Language models require processing a substantially larger number of parameters and often need GPU support to achieve competitive runtimes.