Analyzing the Inner Workings of Transformers in Compositional Generalization
Ryoma Kumon, Hitomi Yanaka
TL;DR
This work investigates how Transformers achieve (or fail to achieve) compositional generalization by locating subnetworks that drive generalization and performing causal analyses of syntactic features. Using a from-scratch encoder–decoder Transformer trained on machine translation and semantic parsing, the authors identify a subnetwork that generalizes well yet relies on non-compositional strategies learned early in training, while the full model depends on syntactic cues. The study combines subnetwork probing with LEACE-based concept erasure to causally assess the role of syntactic constituency and dependency in two PP-based generalization patterns (pp-iobj and pp-subj). Findings indicate that while syntax contributes to generalization, a non-compositional solution within a subnetwork can dominate, underscoring the need for stronger inductive biases to achieve robust compositionality in transformers. These insights advance understanding of linguistic competence in neural models and guide future work on architectural and data-driven biases to promote true compositional generalization.
Abstract
The compositional generalization abilities of neural models have been sought after for human-like linguistic competence. The popular method to evaluate such abilities is to assess the models' input-output behavior. However, that does not reveal the internal mechanisms, and the underlying competence of such models in compositional generalization remains unclear. To address this problem, we explore the inner workings of a Transformer model by finding an existing subnetwork that contributes to the generalization performance and by performing causal analyses on how the model utilizes syntactic features. We find that the model depends on syntactic features to output the correct answer, but that the subnetwork with much better generalization performance than the whole model relies on a non-compositional algorithm in addition to the syntactic features. We also show that the subnetwork improves its generalization performance relatively slowly during the training compared to the in-distribution one, and the non-compositional solution is acquired in the early stages of the training.