Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Data

TL;DR

This work exposes a systematic format bias in LLMs when reasoning over heterogeneous data (texts, tables, infoboxes, KG-graphs). It introduces a three-stage framework to quantify bias, identify data-level drivers (information richness, structure quality, and format type), and reveal attention-based mechanisms behind bias. A lightweight attention-balancing intervention demonstrates improved integration across sources and better downstream QA, while directionality biases appear rooted in pretraining. The study outlines practical mitigations—preprocessing, inference-time attention control, and format-aware training—to enable more robust, fair heterogeneous data reasoning in LLMs.

Abstract

Large Language Models (LLMs) are increasingly employed in applications that require processing information from heterogeneous formats, including texts, tables, infoboxes, and knowledge graphs. However, systematic biases toward particular formats may undermine LLMs' ability to integrate heterogeneous data impartially, potentially resulting in reasoning errors and increased risks in downstream tasks. Yet it remains unclear whether such biases are systematic, which data-level factors drive them, and what internal mechanisms underlie their emergence. In this paper, we present the first comprehensive study of format bias in LLMs through a three-stage empirical analysis. The first stage explores the presence and direction of bias across a diverse range of LLMs. The second stage examines how key data-level factors influence these biases. The third stage analyzes how format bias emerges within LLMs' attention patterns and evaluates a lightweight intervention to test its effectiveness. Our results show that format bias is consistent across model families, driven by information richness, structure quality, and representation type, and is closely associated with attention imbalance within the LLMs. Based on these investigations, we identify three future research directions to reduce format bias: enhancing data pre-processing through format repair and normalization, introducing inference-time interventions such as attention re-weighting, and developing format-balanced training corpora. These directions will support the design of more robust and fair heterogeneous data processing systems.

Figures (9)

• Figure 1: Format bias affects the LLM's decision.
• Figure 2: Examples of the four data formats used in our experiments: texts, tables, infoboxes, and KGs.
• Figure 3: Average DCR across models under heterogeneous and homogeneous formats. Error bars show standard error.
• Figure 4: Heatmap of FPR between format pairs across LLMs. Asterisks (*) indicate statistical significance under a two-sided binomial test with null hypothesis $\text{FPR}=0.5$.
• Figure 5: LLM biases across conditions (averaged over ten LLMs). Top: Information Richness; Bottom: Structure Quality. Bars show the proportion of responses favoring the former input, the latter, or both.