The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts

TL;DR

CenterBench tackles a core challenge in evaluating language models: whether high accuracy on center-embedded sentences arises from true syntactic parsing or from semantic pattern matching. The authors build a scalable benchmark of 360 center-embedded sentences with 9,720 questions, pairing syntactically identical but semantically plausible or implausible sequences across depth levels 1–6 and six question types. Systematic results show that model performance degrades with increasing depth and that models increasingly rely on semantic plausibility as complexity grows; reasoning-enabled models improve accuracy but still reveal semantic shortcuts in their traces. Humans display more variable semantic effects, underscoring a fundamental difference between human and model language processing. Overall, CenterBench provides a first framework to diagnose when models abandon structural analysis for shortcuts, informing both evaluation and deployment in domains requiring genuine syntactic understanding.

Abstract

When language models correctly parse "The cat that the dog chased meowed," are they analyzing syntax or simply familiar with dogs chasing cats? Despite extensive benchmarking, we lack methods to distinguish structural understanding from semantic pattern matching. We introduce CenterBench, a dataset of 9,720 comprehension questions on center-embedded sentences (like "The cat [that the dog chased] meowed") where relative clauses nest recursively, creating processing demands from simple to deeply nested structures. Each sentence has a syntactically identical but semantically implausible counterpart (e.g., mailmen prescribe medicine, doctors deliver mail) and six comprehension questions testing surface understanding, syntactic dependencies, and causal reasoning. Testing six models reveals that performance gaps between plausible and implausible sentences widen systematically with complexity, with models showing median gaps up to 26.8 percentage points, quantifying when they abandon structural analysis for semantic associations. Notably, semantic plausibility harms performance on questions about resulting actions, where following causal relationships matters more than semantic coherence. Reasoning models improve accuracy but their traces show semantic shortcuts, overthinking, and answer refusal. Unlike models whose plausibility advantage systematically widens with complexity, humans shows variable semantic effects. CenterBench provides the first framework to identify when models shift from structural analysis to pattern matching.

Figures (8)

• Figure 1: Performance degradation across complexity levels for three models on plausible vs. implausible center-embedded sentences (averaged across all question types), with example questions of varying difficulty.
• Figure 2: Overview of the evaluation pipeline for processing model responses (shown before 'vs') against gold standard answers
• Figure 3: Performance degradation across complexity levels 1-6 for non-thinking models. Linear trendlines show average accuracy for plausible (solid) and implausible (dashed) sentences. All models show significant decline from level 1 to 6 ($p<0.001$).
• Figure 4: Average accuracy by complexity level for plausible vs. implausible sentences (non-thinking models). Asterisks indicate statistically significant differences between subsets at that complexity level ($p<0.0083$). The widening gap shows models increasingly rely on semantic associations rather than structural analysis.
• Figure 5: Average accuracy by question type for plausible vs. implausible sentences (non-thinking models). Asterisks mark significant differences ($p<0.0083$).