TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval

TL;DR

Problem: effectively leveraging LLMs for tabular prediction with rich textual content while addressing numeric tokenization instability and long-context limits. Approach: serializes rows as sequences, canonicalizes numbers to signed base-10 scientific notation, continues pretraining a $12$B Gemma 3 model on tabular imputation, and uses FAISS-based retrieval of $k=128$ nearest rows to supply exemplars within a $128{,}000$-token window. Key findings: achieves state-of-the-art or competitive accuracy on semantic benchmarks CARTE and TextTab, with classification improving as more context is used; regression is strong in some settings but does not consistently beat tuned baselines on broader tasks. Significance: demonstrates that LLM-based tabular ICL can outperform tuned baselines in semantic tasks and motivates advances in numeric handling and long-context scaling.

Abstract

We study LLMs for tabular prediction with mixed text, numeric, and categorical fields. We introduce TabGemma, a schema-agnostic in-context learner that treats rows as sequences and tackles two practical hurdles when adapting pretrained LLMs for tabular predictions: unstable numeric tokenization and limited context size. We propose to canonicalize numbers via signed scientific notation and continue pretraining of a 12B Gemma 3 model with a target imputation objective using a large-scale real world dataset. For inference, we use a compact n-gram-based retrieval to select informative exemplars that fit within a 128k-token window. On semantically rich benchmarks, TabGemma establishes a new state of the art on classification across low- and high-data regimes and improves monotonically with more context rows. For regression, it is competitive at small sample sizes but trails conventional approaches as data grows. Our results show that LLMs can be effective tabular in-context learners on highly semantic tasks when paired with dedicated numeric handling and context retrieval, while motivating further advances in numeric modeling and long-context scaling.

Figures (6)

• Figure 1: Illustration of our proposed LLM-based tabular prediction architecture with table serialization and target-imputation objective at training, and local context retrieval at inference.
• Figure 2: Critical difference diagrams on the investigated benchmarks, including all datasets ("all") as well as classification- and regression-only subsets. Note that, due to the limited support of classification tasks on CARTE and TextTab, evaluation was performed over the union of tasks in this case.
• Figure 3: Results on the CARTE benchmark for varying train subsets.
• Figure 4: Results on the CARTE benchmark for varying context rows.
• Figure 5: Dependence of mean rank (lower is better) of AutoGluon (blue) vs. TabGemma (orange) on the number of rows (left) and number of columns (right) across different evaluated benchmarks.