Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis

Zerui Cheng, Jiashuo Liu, Jianzhu Yao, Pramod Viswanath, Ge Zhang, Wenhao Huang

TL;DR

TabularMath introduces a diagnostic benchmark to evaluate computational extrapolation in tabular learning by transforming 114 deterministic math problems into large, verified tabular datasets (233,472 rows). It contrasts regression-focused metrics like $R^2$ with exact-match accuracy (rounded consistency) under out-of-distribution extrapolation, revealing a strong $R^2$ but a large gap to exact computation for most tabular models, especially under OOD. The study shows that TabPFN v2.5 excels at in-distribution regression and maintains positive $R^2$ under distribution shift, while in-context learning (ICL) with GPT-OSS-120B achieves markedly higher exact-match accuracy under extrapolation despite weaker regression fits, highlighting complementary strengths. The work argues for a dual approach to tabular reasoning—leveraging efficient statistical learning for interpolation and example-based program induction for precise extrapolation—and releases the dataset, synthesis pipeline, and evaluation tools to spur further research into robust, computation-oriented tabular models.

Abstract

Standard tabular benchmarks mainly focus on the evaluation of a model's capability to interpolate values inside a data manifold, where models good at performing local statistical smoothing are rewarded. However, there exists a very large category of high-value tabular data, including financial modeling and physical simulations, which are generated based upon deterministic computational processes, as opposed to stochastic and noisy relationships. Therefore, we investigate if tabular models can provide an extension from statistical interpolation to computational extrapolation. We propose TabularMath, a diagnostic benchmark of 114 deterministic problems (233,472 rows) generated from verified programs based on GSM8K and AIME. We evaluate 9 tabular architectures and in-context learning (ICL) with GPT-OSS-120B. On standard regression metrics, TabPFN v2.5 performs remarkably well, achieving R^2=0.998 in-distribution and maintaining positive R^2 even under distribution shift, which is unique among the tabular models we tested. When we measure rounded consistency (exact integer match), a different picture emerges: TabPFN v2.5 drops below 10% on out-of-distribution data, while ICL maintains around 40%. This gap between R^2 and exact-match accuracy suggests that tabular models learn smooth function approximations but struggle to recover precise computational outputs under extrapolation. The two paradigms appear complementary: TabPFN scales efficiently with data; ICL achieves exact computation from few examples. We release all code and data to support further investigation.

TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis

TL;DR

TabularMath introduces a diagnostic benchmark to evaluate computational extrapolation in tabular learning by transforming 114 deterministic math problems into large, verified tabular datasets (233,472 rows). It contrasts regression-focused metrics like with exact-match accuracy (rounded consistency) under out-of-distribution extrapolation, revealing a strong but a large gap to exact computation for most tabular models, especially under OOD. The study shows that TabPFN v2.5 excels at in-distribution regression and maintains positive under distribution shift, while in-context learning (ICL) with GPT-OSS-120B achieves markedly higher exact-match accuracy under extrapolation despite weaker regression fits, highlighting complementary strengths. The work argues for a dual approach to tabular reasoning—leveraging efficient statistical learning for interpolation and example-based program induction for precise extrapolation—and releases the dataset, synthesis pipeline, and evaluation tools to spur further research into robust, computation-oriented tabular models.

Abstract

Standard tabular benchmarks mainly focus on the evaluation of a model's capability to interpolate values inside a data manifold, where models good at performing local statistical smoothing are rewarded. However, there exists a very large category of high-value tabular data, including financial modeling and physical simulations, which are generated based upon deterministic computational processes, as opposed to stochastic and noisy relationships. Therefore, we investigate if tabular models can provide an extension from statistical interpolation to computational extrapolation. We propose TabularMath, a diagnostic benchmark of 114 deterministic problems (233,472 rows) generated from verified programs based on GSM8K and AIME. We evaluate 9 tabular architectures and in-context learning (ICL) with GPT-OSS-120B. On standard regression metrics, TabPFN v2.5 performs remarkably well, achieving R^2=0.998 in-distribution and maintaining positive R^2 even under distribution shift, which is unique among the tabular models we tested. When we measure rounded consistency (exact integer match), a different picture emerges: TabPFN v2.5 drops below 10% on out-of-distribution data, while ICL maintains around 40%. This gap between R^2 and exact-match accuracy suggests that tabular models learn smooth function approximations but struggle to recover precise computational outputs under extrapolation. The two paradigms appear complementary: TabPFN scales efficiently with data; ICL achieves exact computation from few examples. We release all code and data to support further investigation.
Paper Structure (116 sections, 1 theorem, 2 equations, 7 figures, 9 tables)

This paper contains 116 sections, 1 theorem, 2 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Motivation for a New Benchmark
  3. A Motivating Example.
  4. Transforming Math Problems to Tabular Computation
  5. Evaluation Framework.
  6. Main Results.
  7. Why This Is Important.
  8. Contributions.
  9. Scope.
  10. Related Work
  11. Deep Learning for Tabular Data
  12. Tabular Foundation Models
  13. In-Context Learning
  14. Program-Aided Reasoning
  15. Mathematical Reasoning Benchmarks
  16. ...and 101 more sections

Key Result

Corollary 1

Given that a model has acquired the correct program $\mathcal{V}$, then the model should be able to predict the output value for any valid input — including those that are not part of the training distribution. Models that are primarily approximating $P(\mathcal{Y}|\mathcal{X})$ via interpolation ge

Figures (7)

  • Figure 1: The $R^2$--consistency gap. On TabularMath-GSM8k with out-of-distribution test data, ICL achieves $\sim$39% exact-match accuracy while tabular models fall below 10%. TabPFN v2.5 nonetheless achieves strong $R^2$ (0.75 under OOD), indicating that it captures the functional trend even when exact predictions fail. The two approaches exhibit complementary strengths: ICL recovers precise computations from few examples; TabPFN approximates functions efficiently and improves with more data.
  • Figure 2: Verified Tabular Benchmark Synthesis Pipeline. We emphasize that the pipeline defines a general benchmark construction methodology; TabularMath is one concrete instantiation used for evaluation in our paper.
  • Figure 3: Computational Graph for a Simple Math Problem. Math problems encode directed acyclic graphs where inputs have positive or negative effects on the output. Here, price increases the total ($+$), discount decreases it ($-$), and tax increases it ($+$). Crucially, effects interact: tax applies to the discounted subtotal, not the original price. Tabular models see only $(x_1, x_2, x_3) \to y$ pairs---the causal structure is latent. In TabularMath, we use Turing-complete Python programs to characterize the computational graph.
  • Figure 4: The three-phase verified synthesis pipeline. Checking against ground-truth ensures 100% label accuracy.
  • Figure 5: End-to-end example showing how a word problem becomes a tabular regression task. The model must learn the computational relationship, not just interpolate.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Definition 1: Standard Tabular Dataset
  • Corollary 1: Implications of Extrapolation Using the Latent Program Hypothesis