A Theoretical Analysis of Compositional Generalization in Neural Networks: A Necessary and Sufficient Condition

TL;DR

This work tackles the challenge of compositional generalization in neural networks by deriving a necessary and sufficient condition that ties together architectural structure, representation, and training data properties. The main result asserts that CG is achievable if and only if the model exhibits structural alignment, unambiguous representation, and minimized representation, formalized with graph isomorphisms and mapping properties. The paper provides rigorous proofs, a minimal illustrative example, and discussion on how these conditions can guide pre-training assessment and architectural design. Taken together, the results offer a principled inductive-bias framework for designing models with robust compositional capabilities and for evaluating them prior to extensive training.

Abstract

Compositional generalization is a crucial property in artificial intelligence, enabling models to handle novel combinations of known components. While most deep learning models lack this capability, certain models succeed in specific tasks, suggesting the existence of governing conditions. This paper derives a necessary and sufficient condition for compositional generalization in neural networks. Conceptually, it requires that (i) the computational graph matches the true compositional structure, and (ii) components encode just enough information in training. The condition is supported by mathematical proofs. This criterion combines aspects of architecture design, regularization, and training data properties. A carefully designed minimal example illustrates an intuitive understanding of the condition. We also discuss the potential of the condition for assessing compositional generalization before training. This work is a fundamental theoretical study of compositional generalization in neural networks.

Figures (4)

• Figure 1: An example of notations. Normal fonts (e.g., $h, h_1$) are nodes, and bold fonts (e.g., $\mathbf{h}, \mathbf{h}_1$) are corresponding input node sequences. An arrow points from the input node sequence to the output node of a component. On the left, $(h, \mathbf{h})$ and $\forall i \in \{1, \dots, n\}: (h_i, \mathbf{h}_i)$ are components. Similar on the right.
• Figure 2: Visual proof of Lemma \ref{['lemma:minimum']}. Left: When $|A|$ is minimized to match $|B|$, the mapping becomes bijective. Right: When $|A'|$ is not minimized, multiple elements in $A'$ map to the same element in $B$, violating one-to-one mapping. This illustrates that one-to-one mapping is equivalent to minimal domain size under well-defined onto mappings.
• Figure 3: Illustrative proof of the induction step (Lemma \ref{['lemma:test_one-to-one']}). At each step with test node $B$, there is a training node $A$ with $\mathbf{z}^A = \mathbf{z}^B$ (Definition \ref{['definition:reference_graph_set']}). To leverage the induction condition, we break down the comparison of $A$ and $B$ into their inputs. For each input index $i$, there is a training node $C_i$. This figure shows the relationships between the nodes. Training nodes are white, and test nodes are gray. Two nodes have the same value if connected by any style of line. Solid lines are given by preconditions. $\alpha$ is from the equal reference inputs. $\beta_z$ and $\beta_h$ are from the induction condition. Dashed lines are assigned one by one during the induction step. I is from $\alpha$ and $\beta_z$. II is from I by the one-to-one mapping in training. III is from II and $\beta_h$. It applies to all input nodes $i$. By the deterministic property of components (Definition \ref{['definition:component']}), we have $z^A = z^B$ from $\alpha$ and $h^A = h^B$ from III.
• Figure 4: The algorithm for the SCAN jump task. Syntax $T_i$ and semantics $V_i$ word embeddings are regularized to minimize their varieties.

Theorems & Definitions (53)

• Theorem 1: Necessary and Sufficient Condition
• Definition 1: Component
• Definition 2: Computational Graph
• Definition 3: Graph Set
• Definition 4: Reference Graph Set
• Definition 5: Compositional Generalization
• Definition 6: Structural Alignment
• Definition 7: Unambiguous Representation
• Definition 8: Minimized Representation
• Lemma 1: Mappings on Nodes