Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Investigation of Robustness in Large Language Models under Tabular Distortions

Avik Dutta, Harshit Nigam, Hosein Hasanbeig, Arjun Radhakrishna, Sumit Gulwani

TL;DR

This work assesses how large language models handle distorted tabular data in table question answering. It introduces a small, expert-curated dataset with semantic and structural distortions to quantify robustness across model families, input modalities, and tool usage. The findings show that LLMs lack inherent distortion awareness, with structural distortions causing larger errors and vertical misalignments being particularly challenging; prompting with distortion awareness improves performance but is not consistently reliable. The study highlights a need for autonomous table realignment and repair capabilities in future models, beyond reliance on explicit prompts or pre-processing, to enable robust real-world TQA. $R_{ ext{bst}} = rac{ ext{Dist}}{ ext{Can}}$ is used to measure robustness, and results indicate significant gaps even for SoTA models like GPT-5.2 under distortion.

Abstract

We investigate how large language models (LLMs) fail when tabular data in an otherwise canonical representation is subjected to semantic and structural distortions. Our findings reveal that LLMs lack an inherent ability to detect and correct subtle distortions in table representations. Only when provided with an explicit prior, via a system prompt, do models partially adjust their reasoning strategies and correct some distortions, though not consistently or completely. To study this phenomenon, we introduce a small, expert-curated dataset that explicitly evaluates LLMs on table question answering (TQA) tasks requiring an additional error-correction step prior to analysis. Our results reveal systematic differences in how LLMs ingest and interpret tabular information under distortion, with even SoTA models such as GPT-5.2 model exhibiting a drop of minimum 22% accuracy under distortion. These findings raise important questions for future research, particularly regarding when and how models should autonomously decide to realign tabular inputs, analogous to human behavior, without relying on explicit prompts or tabular data pre-processing.

An Empirical Investigation of Robustness in Large Language Models under Tabular Distortions

TL;DR

This work assesses how large language models handle distorted tabular data in table question answering. It introduces a small, expert-curated dataset with semantic and structural distortions to quantify robustness across model families, input modalities, and tool usage. The findings show that LLMs lack inherent distortion awareness, with structural distortions causing larger errors and vertical misalignments being particularly challenging; prompting with distortion awareness improves performance but is not consistently reliable. The study highlights a need for autonomous table realignment and repair capabilities in future models, beyond reliance on explicit prompts or pre-processing, to enable robust real-world TQA. is used to measure robustness, and results indicate significant gaps even for SoTA models like GPT-5.2 under distortion.

Abstract

We investigate how large language models (LLMs) fail when tabular data in an otherwise canonical representation is subjected to semantic and structural distortions. Our findings reveal that LLMs lack an inherent ability to detect and correct subtle distortions in table representations. Only when provided with an explicit prior, via a system prompt, do models partially adjust their reasoning strategies and correct some distortions, though not consistently or completely. To study this phenomenon, we introduce a small, expert-curated dataset that explicitly evaluates LLMs on table question answering (TQA) tasks requiring an additional error-correction step prior to analysis. Our results reveal systematic differences in how LLMs ingest and interpret tabular information under distortion, with even SoTA models such as GPT-5.2 model exhibiting a drop of minimum 22% accuracy under distortion. These findings raise important questions for future research, particularly regarding when and how models should autonomously decide to realign tabular inputs, analogous to human behavior, without relying on explicit prompts or tabular data pre-processing.
Paper Structure (20 sections, 12 figures, 3 tables)

This paper contains 20 sections, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Distortions
  3. Semantic Distortion
  4. Structural Distortion
  5. Dataset Construction
  6. Experimental Setup
  7. Models
  8. Setup.
  9. Metrics.
  10. Evaluation
  11. Distortion Awareness
  12. Semantic vs Structural Distortion
  13. When do models detect distortion?
  14. Related Works
  15. Conclusion
  16. ...and 5 more sections

Figures (12)

  • Figure 1: Illustration of structural and semantic table distortions and their impact on downstream reasoning.
  • Figure 2: Structurally Distorted table for the query -- How many employees did overtime?
  • Figure 3: A python script generated by GPT-5 under distortion-unaware prompting. The model fails to detect the displacement of rows and solve the question by considering only numeric based values in the "OvertimeHours" column".
  • Figure 4: A python script generated by GPT-5 under distortion-aware prompting. It is correctly able to detect and address the shift in rows and aligns them before producing the final outcome.
  • Figure 5: Semantic Distortion Example 1: The units and metrics for measuring Volume and Weight are swapped. The model should be able to understand that ship containers usually do not have volumes even in cubic centimeters in the range of 15.00-25.00
  • ...and 7 more figures