Understanding or Memorizing? A Case Study of German Definite Articles in Language Models

TL;DR

The paper investigates whether German definite singular articles in language models are encoded via abstract rule-like generalization or memorized surface associations. It introduces Gradiend, a gradient-based interpretability technique that learns a scalar update direction for targeted gender–case transitions and tests generalization across related cells. Across multiple models, Gradiend reveals that interventions shift article probabilities beyond the trained cell and show substantial overlap in the most affected parameters within article groups, challenging a purely rule-based account and supporting memorization-like mechanisms. These findings highlight the nuanced nature of grammatical knowledge in neural models and provide a principled approach for probing linguistic rules in LMs, with code and data released for further research.

Abstract

Language models perform well on grammatical agreement, but it is unclear whether this reflects rule-based generalization or memorization. We study this question for German definite singular articles, whose forms depend on gender and case. Using GRADIEND, a gradient-based interpretability method, we learn parameter update directions for gender-case specific article transitions. We find that updates learned for a specific gender-case article transition frequently affect unrelated gender-case settings, with substantial overlap among the most affected neurons across settings. These results argue against a strictly rule-based encoding of German definite articles, indicating that models at least partly rely on memorized associations rather than abstract grammatical rules.

Figures (38)

• Figure 1: Illustration of factual ($y^F$) and alternative ($y^A$) targets for the gender-case transition $(\textsc{Neut},\textsc{Nom})\rightleftarrows(\textsc{Neut},\textsc{Dat})$. Non-target cells form identity pairs (only one shown). Dataset labels (e.g., $D_\textsc{Nom}^\textsc{Neut}$) denote the corresponding gender-case datasets (Section \ref{['sec:data']}).
• Figure 2: Data generation: spaCy determines gender and case of articles to determine the target dataset.
• Figure 3: Encoded value distribution of $G_{\textsc{Nom}}^{\textsc{Fem},\textsc{Masc}}$ (other in Figures \ref{['fig:encoded-all-models-ND_F']}-\ref{['fig:encoded-all-models-GA_N']}).
• Figure 4: Patterns of generalizations, exemplified using $G_{\textsc{Nom}}^{\textsc{Fem},\textsc{Masc}}$ ($der\,{\to}\,die$).
• Figure 5: $G_{\textsc{Nom},\textsc{Dat}}^{\textsc{Fem}}$ applied to GermanBERT for the $der\,{\to}\,die$ transition: mean article probabilities and LMS across learning rates $\alpha$. The candidate range ($\alpha>0$ and before the $99\%$ LMS drop) is shaded gray. Highlighted LMS points mark the base model (left) and $\alpha^\star$ (maximizing $\mathbb{P}(der)$ on $D_\textsc{Dat}^\textsc{Fem}$ in gray area).