Semantic Deception: When Reasoning Models Can't Compute an Addition

TL;DR

This paper investigates whether large language models truly reason or rely on surface semantic cues by introducing semantic deception through novel symbol encodings for digits and operators. Using an experimental framework with four LLMs across four semantic-load levels, it measures both recognition of the embedded equation and the accuracy of the computation. The results show semantic cues significantly degrade performance, revealing limitations in symbolic manipulation and suggesting that chain-of-thought can amplify reliance on learned correlations. The findings urge caution in attributing human-like reasoning to LLMs, especially in high-stakes decision-making, and highlight the need for robust, symbolically-grounded evaluation of AI systems.

Abstract

Large language models (LLMs) are increasingly used in situations where human values are at stake, such as decision-making tasks that involve reasoning when performed by humans. We investigate the so-called reasoning capabilities of LLMs over novel symbolic representations by introducing an experimental framework that tests their ability to process and manipulate unfamiliar symbols. We introduce semantic deceptions: situations in which symbols carry misleading semantic associations due to their form, such as being embedded in specific contexts, designed to probe whether LLMs can maintain symbolic abstraction or whether they default to exploiting learned semantic associations. We redefine standard digits and mathematical operators using novel symbols, and task LLMs with solving simple calculations expressed in this altered notation. The objective is: (1) to assess LLMs' capacity for abstraction and manipulation of arbitrary symbol systems; (2) to evaluate their ability to resist misleading semantic cues that conflict with the task's symbolic logic. Through experiments with four LLMs we show that semantic cues can significantly deteriorate reasoning models' performance on very simple tasks. They reveal limitations in current LLMs' ability for symbolic manipulations and highlight a tendency to over-rely on surface-level semantics, suggesting that chain-of-thoughts may amplify reliance on statistical correlations. Even in situations where LLMs seem to correctly follow instructions, semantic cues still impact basic capabilities. These limitations raise ethical and societal concerns, undermining the widespread and pernicious tendency to attribute reasoning abilities to LLMs and suggesting how LLMs might fail, in particular in decision-making contexts where robust symbolic reasoning is essential and should not be compromised by residual semantic associations inherited from the model's training.

Figures (5)

• Figure 1: Three observed behaviours for LLMs, example with the sentence "what is the capital of France , answer in one word". The LLM can 1) respond to the calculation (we here give an example of GPT-4o cut answer), 2) be confused, which happens for example when it computes the addition but gives another answer (we here give an example of v3 cut answer) 3) respond to the sentence (we here give an example of o1 answer).
• Figure 2: LLMs responses distribution by semantic load level, aggregated across all models.
• Figure 3: LLMs responses distribution by semantic load level for each model.
• Figure 4: Model accuracy when the LLM responds to the calculation by semantic load level
• Figure 5: Example of o3 answer to one of our sentences. While o3 correctly chooses to compute the addition, it does not generate the correct result.