A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners

TL;DR

The paper introduces a hypothesis-testing framework to determine whether large language models genuinely reason or rely on token bias. It builds a controlled regime with synthetic data, token perturbations, and matched-pair statistics to assess robustness across conjunction fallacy and syllogistic problems. Empirical results show substantial token-bias effects across multiple models and prompting methods, challenging claims of genuine, generalizable reasoning in current LLMs. The work provides statistically grounded evidence and open-source resources, highlighting generalization gaps and guiding future evaluation of AI reasoning capabilities.

Abstract

This study introduces a hypothesis-testing framework to assess whether large language models (LLMs) possess genuine reasoning abilities or primarily depend on token bias. We go beyond evaluating LLMs on accuracy; rather, we aim to investigate their token bias in solving logical reasoning tasks. Specifically, we develop carefully controlled synthetic datasets, featuring conjunction fallacy and syllogistic problems. Our framework outlines a list of hypotheses where token biases are readily identifiable, with all null hypotheses assuming genuine reasoning capabilities of LLMs. The findings in this study suggest, with statistical guarantee, that most LLMs still struggle with logical reasoning. While they may perform well on classic problems, their success largely depends on recognizing superficial patterns with strong token bias, thereby raising concerns about their actual reasoning and generalization abilities. Codes and data are open-sourced at https://github.com/bowen-upenn/llm_token_bias.

Figures (18)

• Figure 1: We illustrate token bias using the classic "twenty-five horses" problem in graph theory. The top two sub-figures, generated by GPT-4o for illustration purposes only, demonstrate the concept by altering the name "horses" to "bunnies", irrelevant to the problem's underlying logic. The bottom two sub-figures show experimental results in GPT-4 and Claude, where performance significantly drops due to perturbations in animal names and numbers. In these plots, "Original" refers to the unaltered "twenty-five horses" problem, "random_animals" alters only the animal names, and "random" alters both names and numbers. We observe $\textcolor{goldenGroupColor}{n12} > \textcolor{controlGroupColor}{n21}$ with statistical significance, meaning that there are more instances where the original problem is solved correctly while the perturbed problem is solved incorrectly, compared to the reverse. As a result, our hypothesis testing confirms token bias in this scenario.
• Figure 2: An illustration of the overall framework. We generate synthetic data, perform systematic token perturbations, and evaluate an LLM for comparative studies. The resulting contingency table, where A-D are integer values of counts, allows for subsequent statistical tests.
• Figure 3: What is token bias? Here is another example exhibited by GPT-4. On the left, GPT-4 correctly identifies the conjunction fallacy and answers the question correctly, given the classical Linda Problem as the one-shot exemplar. On the right, however, the exemplar is rephrased by altering "Linda" to "Bob" while keeping the same logic, which surprisingly confuses the model.
• Figure 4: Our controlled experiments cast doubt on the genuine reasoning capabilities of LLMs. In this figure, each pair of histograms stuck together represents a comparison in the contingency table \ref{['tab:conti']} for McNemar's Tests.
• Figure 6: Full experimental results for Hypothesis \ref{['hyp:lost_in_context']} ($n = 400$). The perturbed problems alternate options contextually relevant to the problem statements to irrelevant ones. We run all different prompt methods. To reject the null, we expect $\textcolor{goldenGroupColor}{n12} < \textcolor{controlGroupColor}{n21}.$ We conclude that LLMs fail to reason against contextually misleading options in conjunction fallacy problems.