Evaluating Latent Knowledge of Public Tabular Datasets in Large Language Models

TL;DR

This work addresses whether large language models actually reason over tabular data or merely recall widely circulated benchmark material. It introduces a contamination-aware evaluation framework with two tasks (Completion and Existence) and two dataset variants (synthetic marginal and obfuscated) applied to semantic and non-semantic tabular datasets, tested across multiple model families in zero-shot. Findings show that contamination manifests semantically—LMs leverage knowledge about feature meanings and dataset semantics to infer values in familiar benchmarks, but there is no evidence of verbatim memorization of specific records; results also hint at emergent reasoning in some cases. The study highlights the need for contamination-aware benchmarks and evaluation protocols to ensure that reported tabular reasoning capabilities reflect genuine generalization rather than semantic leakage from pretraining data.

Abstract

Large Language Models (LLMs) are increasingly evaluated on their ability to reason over structured data, yet such assessments often overlook a crucial confound: dataset contamination. In this work, we investigate whether LLMs exhibit prior knowledge of widely used tabular benchmarks such as Adult Income, Titanic, and others. Through a series of controlled probing experiments, we reveal that contamination effects emerge exclusively for datasets containing strong semantic cues-for instance, meaningful column names or interpretable value categories. In contrast, when such cues are removed or randomized, performance sharply declines to near-random levels. These findings suggest that LLMs' apparent competence on tabular reasoning tasks may, in part, reflect memorization of publicly available datasets rather than genuine generalization. We discuss implications for evaluation protocols and propose strategies to disentangle semantic leakage from authentic reasoning ability in future LLM assessments.