LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs

TL;DR

This work provides the first systematic study of output format bias in LLMs, introducing formal metrics and an estimator (EstTrueE) to fairly gauge true model performance under diverse format constraints. It defines a comprehensive evaluation framework across MCQ, wrapping, list, and mapping formats (15 variants) and demonstrates substantial format bias across eight tasks and four models. The authors link bias to format-token priors in training data and show that prompting techniques (demonstrations, repeated format instructions) and especially finetuning with synthesized format data can dramatically reduce bias, achieving near-uniform format adherence and substantial improvements in cross-format performance variance (e.g., reducing ChatGPT’s wrapping variance from $235.33^2$ to $0.71^2$). The study offers practical mitigation strategies and highlights the need to consider format bias in industrial deployments, aiming for fairer, more reliable LLM behavior across varying output formats.

Abstract

We present the first systematic evaluation examining format bias in performance of large language models (LLMs). Our approach distinguishes between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. We then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. Subsequently, we present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping -- covering 15 widely-used formats. Our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. We further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. Based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. Our methods successfully reduce the variance in ChatGPT's performance among wrapping formats from 235.33 to 0.71 (%$^2$).

Figures (6)

• Figure 1: A MMLU example hendrycks2020measuring with ChatGPT across different formats. In Case (1), the model can answer the question but fails to bold only the answer, hindering automatic evaluation. In Case (2), the model follows the format but produces an incorrect result. In Case (3), the model yields the correct answer and format. These show bias in ChatGPT's performance across formats.
• Figure 2: Average estimated true accuracy (\ref{['ssec:format-eval-metrics']}) results of MCQ benchmarks across models (left) and datasets (right) showing performance bias of LLMs across formats.
• Figure 3: Average estimated true Accuracy (MCQ) and F1 (GSM8K, HotpotQA, FairytaleQA) scores (\ref{['ssec:format-eval-metrics']}) across models (left) and across benchmarks (right), showing performance bias of LLMs across $7$ widely used wrapping methods.
• Figure 4: Average $EstTrueF1$ (SemEval2017) and $EstTrueMAP$ (SciDocsRR) (\ref{['ssec:format-eval-metrics']}) across models (left) and benchmarks (right) showing performance difference of LLMs across $4$ widely used list formats.
• Figure 5: Average estimated true F1 scores (\ref{['ssec:format-eval-metrics']}) across models (left) and benchmarks (right) showing performance bias of LLMs across $2$ widely used mapping formats.

Theorems & Definitions (4)

• Definition 3.1: Systematic Evaluation Score ($SysE$)
• Definition 3.2: True Evaluation Score ($TrueE$)
• Theorem 3.1: Reliability of $EstTrueE$
• proof : Proof of \ref{['theorem:theorem1']}