Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning

TL;DR

This work tackles Semantic Inertia, the tendency of pretrained priors to dominate in-context reasoning when rules change. It uses Baba Is You to demonstrate that natural-language representations entangle semantics and dynamics, causing larger models to amplify priors, while executable-code grounding decouples logic from appearance. The authors introduce Code-Grounded Vistas (LCV), an amortized theory-induction framework trained with counterfactual contrastive alignment to synthesize world models in a single forward pass, enabling robust, real-time planning under mutable ontologies. With extensive experiments against multiple baselines, LCV outperforms inference-heavy approaches in both accuracy and efficiency, and shows strong generalization to unseen maps and rule combinations. The results imply that representational choices—code-grounded reasoning versus natural language—critically determine whether scaling improves or impairs contextual reasoning in dynamic domains.

Abstract

LLMs struggle with Semantic Inertia: the inability to inhibit pre-trained priors (e.g., "Lava is Dangerous") when dynamic, in-context rules contradict them. We probe this phenomenon using Baba Is You, where physical laws are mutable text rules, enabling precise evaluation of models' ability to override learned priors when rules change. We quantatively observe that larger models can exhibit inverse scaling: they perform worse than smaller models when natural language reasoning requires suppressing pre-trained associations (e.g., accepting "Lava is Safe"). Our analysis attributes this to natural language encoding, which entangles descriptive semantics and logical rules, leading to persistent hallucinations of familiar physics despite explicit contradictory rules. Here we show that representing dynamics as executable code, rather than descriptive text, reverses this trend and enables effective prior inhibition. We introduce Code-Grounded Vistas (LCV), which fine-tunes models on counterfactual pairs and identifies states with contradictory rules, thereby forcing attention to logical constraints rather than visual semantics. This training-time approach outperforms expensive inference-time search methods in both efficiency and accuracy. Our results demonstrate that representation fundamentally determines whether scaling improves or impairs contextual reasoning. This challenges the assumption that larger models are universally better, with implications for domains that require dynamic overriding of learned priors.

Figures (7)

• Figure 1: Rule mutability and reasoning paradigms in Baba Is You. The Baba Is You environment externalizes game logic as manipulable text blocks, allowing the rules governing object affordances and agent identity to be dynamically rewritten (e.g., "Baba is You" $\rightarrow$ "Wall is You"). We study two reasoning paradigms in this setting: natural language reasoning, which operates over descriptive semantics and is prone to semantic inertia, and code-grounded reasoning, which treats rules as executable constraints and enables explicit state tracking under mutable laws.
• Figure 2: Inverse scaling in natural language vs. restored scaling in code. In Natural Language prompting (red lines), larger models often exhibit worse performance due to semantic interference, with models showing clear inverse scaling. Code Grounding (green lines) decouples logical operations from semantic priors, restoring positive scaling laws where computational scale translates to improved contextual reasoning.
• Figure 3: Overview of lcv. (Up) Amortized Contrastive Theory Induction: We perform sft on paired samples with the same environment state $s$ but contradictory rule-sets $r$ (e.g., WALL IS STOPvs.WALL IS YOU. This directly targets semantic inertia by making surface symbols non-diagnostic: the model must generate different executable kernels solely from the active rules, disentangling object names from their pre-trained affordance priors. (Down) Online Inference and Planning: At test time, the model synthesizes the executable transition theory in a single pass, which is compiled and used by a classical planner.
• Figure A1: Overview of the Baba Is You environment.
• Figure A2: Visualizing the BabaBench Difficulty Hierarchy. We present three tiers of increasing neuro-symbolic friction. Unlike traditional RL environments with static affordances, our benchmark requires agents to handle: (a) aligned semantics, (b) counterintuitive inhibitory control, and (c) dynamic rule restructuring.