Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamics Reveals Structure: Challenging the Linear Propagation Assumption

Hoyeon Chang, Bálint Mucsányi, Seong Joon Oh

TL;DR

This work questions whether first-order gradient updates preserve logical coherence under the Linear Propagation Assumption (LPA). It formalizes relational knowledge with relation algebra and analyzes the geometry of linearized updates via Systematic Linear Propagation (SLP). The authors prove that negation equivariance enforces a tensor-factorized feature structure separating entity-pair context from relation content, and that converse invariance enforces symmetric/antisymmetric positional alignment; however, a fundamental obstruction arises for composition, showing that linear conjunction must be bilinear, which is incompatible with negation, forcing a collapse of the feature map. These results provide a geometric explanation for observed failures in knowledge editing, the reversal curse, and multi-hop reasoning, and motivate a shift toward logical geometric deep learning where update-time symmetries guide the design of representations and updates.

Abstract

Neural networks adapt through first-order parameter updates, yet it remains unclear whether such updates preserve logical coherence. We investigate the geometric limits of the Linear Propagation Assumption (LPA), the premise that local updates coherently propagate to logical consequences. To formalize this, we adopt relation algebra and study three core operations on relations: negation flips truth values, converse swaps argument order, and composition chains relations. For negation and converse, we prove that guaranteeing direction-agnostic first-order propagation necessitates a tensor factorization separating entity-pair context from relation content. However, for composition, we identify a fundamental obstruction. We show that composition reduces to conjunction, and prove that any conjunction well-defined on linear features must be bilinear. Since bilinearity is incompatible with negation, this forces the feature map to collapse. These results suggest that failures in knowledge editing, the reversal curse, and multi-hop reasoning may stem from common structural limitations inherent to the LPA.

Dynamics Reveals Structure: Challenging the Linear Propagation Assumption

TL;DR

This work questions whether first-order gradient updates preserve logical coherence under the Linear Propagation Assumption (LPA). It formalizes relational knowledge with relation algebra and analyzes the geometry of linearized updates via Systematic Linear Propagation (SLP). The authors prove that negation equivariance enforces a tensor-factorized feature structure separating entity-pair context from relation content, and that converse invariance enforces symmetric/antisymmetric positional alignment; however, a fundamental obstruction arises for composition, showing that linear conjunction must be bilinear, which is incompatible with negation, forcing a collapse of the feature map. These results provide a geometric explanation for observed failures in knowledge editing, the reversal curse, and multi-hop reasoning, and motivate a shift toward logical geometric deep learning where update-time symmetries guide the design of representations and updates.

Abstract

Neural networks adapt through first-order parameter updates, yet it remains unclear whether such updates preserve logical coherence. We investigate the geometric limits of the Linear Propagation Assumption (LPA), the premise that local updates coherently propagate to logical consequences. To formalize this, we adopt relation algebra and study three core operations on relations: negation flips truth values, converse swaps argument order, and composition chains relations. For negation and converse, we prove that guaranteeing direction-agnostic first-order propagation necessitates a tensor factorization separating entity-pair context from relation content. However, for composition, we identify a fundamental obstruction. We show that composition reduces to conjunction, and prove that any conjunction well-defined on linear features must be bilinear. Since bilinearity is incompatible with negation, this forces the feature map to collapse. These results suggest that failures in knowledge editing, the reversal curse, and multi-hop reasoning may stem from common structural limitations inherent to the LPA.
Paper Structure (46 sections, 17 theorems, 125 equations, 4 figures)

This paper contains 46 sections, 17 theorems, 125 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Motivation: Gradient Misalignment in LLMs
  4. Preliminary: Relation Algebra
  5. Queries, Scores, and Linearized Features
  6. Queries.
  7. Scores and Linearization.
  8. Systematic Linear Propagation
  9. SLP Induces Tensor-Factorized Features
  10. Negation Equivariance Forces Tensor Factorization
  11. Converse Equivariance Forces Positional Alignment
  12. The Collapse of Linear Conjunction
  13. Related Work
  14. Linearized Dynamics and Model Adaptation.
  15. Systematicity and Variable Binding.
  16. ...and 31 more sections

Key Result

Theorem 1

Let $\phi:Q\to W$ be the feature map defined by $q\mapsto \phi_q$. If $\phi$ satisfies SLP, then there exist real vector spaces $\{C_i\}_i$, $\{R_i\}_i$ and an isomorphism $W \cong \bigoplus_i (C_i \otimes R_i)$ such that where $u_{i,k}: E \times E \to C_i$ and $v_{i,k}: \mathcal{R} \to R_i$. Moreover, negation acts locally as a sign flip on each relation component, i.e., $v_{i,k}(\neg r) = -v_{i

Figures (4)

  • Figure 1: Geometric interpretation of logical equivariance.(Left) A query $q$ is associated with a score $s_\theta(q)$ (e.g., log probability), and its gradient $\phi_q = \nabla s_\theta(q)$ as a feature. Under the LPA, a local parameter update $\Delta\theta$ induces a score change approximated by the inner product: $\Delta s(q) = s_{\theta+\Delta\theta}(q)-s_\theta(q)\approx \langle \phi_q, \Delta\theta \rangle$. Logical consistency under direction-agnostic first-order propagation requires that any local parameter change enhancing $q$ suppresses $\neg q$ (i.e., $\Delta s(\neg q) = -\Delta s(q)$), which necessitates that the gradient vectors be anti-aligned ($\phi_{\neg q} \approx -\phi_q$). (Right) This geometric requirement induces a commutative diagram where symbolic negation $\neg$ in the query space corresponds to a linear inversion $-I$ in the gradient feature space.
  • Figure 2: Gradient alignment hinders negation consistency. Cosine similarities between gradients of facts and their negations. Contrary to the theoretical requirement for anti-alignment ($=-1$), empirical gradients are strongly positively aligned ($\approx0.85$). The gradients are computed with respect to the parameters of the last Transformer block and LM head. See \ref{['app:experimental-setup']} for detailed setup.
  • Figure 3: The incompatibility of logical conjunction and LPA.Top Path: Logical idempotence maps $(\neg p, \neg p) \to \neg p$, expecting feature $-\phi_p$. Bottom Path: Linearization gives $(-\phi_p, -\phi_p)$, and the bilinearity of $\tilde{F}$ yields $+\phi_p$. The only possible way to commute the two paths is setting $\phi_p=0$, leading to a collapse.
  • Figure 4: Gradient alignment across different scales and architectures. The positive alignment phenomenon persists in (a) Qwen3-30B and (b) OLMo-3-7B. Both distributions are heavily skewed towards positive cosine similarity, demonstrating that the geometric mismatch for linear negation propagation is consistent across model scale and architecture.

Theorems & Definitions (35)

  • Definition 1: Linearized feature
  • Definition 2: Logical equivariance
  • Definition 3: Systematic Linear Propagation (SLP)
  • Theorem 1: Context-Relation Factorization (Proof in \ref{['app:proof-thm2']})
  • Theorem 2: Symmetric-Antisymmetric Alignment (Proof in \ref{['app:proof-thm3']})
  • Definition 4: Conjunction-faithful features under LPA
  • Lemma 1: Substitution
  • Lemma 2: Kernel Stability Yields Bilinearity (Proof in \ref{['app:bilinearity']})
  • Theorem 3: Structural Collapse of Linear Conjunction
  • proof
  • ...and 25 more