Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What makes Models Compositional? A Theoretical View: With Supplement

Parikshit Ram, Tim Klinger, Alexander G. Gray

TL;DR

The paper develops a neuro-symbolic framework to understand compositional generalization in sequence models by formalizing compositional functions via a token encoder $e$, a computation DAG $D(X)$, a span processor $g$, and a read-out $h$, with the forward pass $f(X) = h\left( g^{\otimes D(X)}\left( e(x_1,1), \ldots, e(x_L,L)\right) \right)$. It introduces the Compositional Complexity and locus of influence (LoI) to quantify how the DAG structure and the neural components interact to shape input sensitivity, and it applies these notions to characterize recurrent, convolutional, and transformer architectures, including efficient transformers with input-dependent computation graphs. The paper proves theoretical guarantees linking compositional structure to expressivity and systematic generalization, and distinguishes between input-dependent and input-agnostic cDAGs, showing limitations of the latter for approximating the former. These results offer a principled lens for analyzing architectural choices and inform design for improved compositional generalization in practical tasks. Overall, the framework provides rigorous, quantitative insights into when and why modern sequence models struggle with compositional tasks and how to engineer architectures that better exploit hierarchical structure.

Abstract

Compositionality is thought to be a key component of language, and various compositional benchmarks have been developed to empirically probe the compositional generalization of existing sequence processing models. These benchmarks often highlight failures of existing models, but it is not clear why these models fail in this way. In this paper, we seek to theoretically understand the role the compositional structure of the models plays in these failures and how this structure relates to their expressivity and sample complexity. We propose a general neuro-symbolic definition of compositional functions and their compositional complexity. We then show how various existing general and special purpose sequence processing models (such as recurrent, convolution and attention-based ones) fit this definition and use it to analyze their compositional complexity. Finally, we provide theoretical guarantees for the expressivity and systematic generalization of compositional models that explicitly depend on our proposed definition and highlighting factors which drive poor empirical performance.

What makes Models Compositional? A Theoretical View: With Supplement

TL;DR

The paper develops a neuro-symbolic framework to understand compositional generalization in sequence models by formalizing compositional functions via a token encoder , a computation DAG , a span processor , and a read-out , with the forward pass . It introduces the Compositional Complexity and locus of influence (LoI) to quantify how the DAG structure and the neural components interact to shape input sensitivity, and it applies these notions to characterize recurrent, convolutional, and transformer architectures, including efficient transformers with input-dependent computation graphs. The paper proves theoretical guarantees linking compositional structure to expressivity and systematic generalization, and distinguishes between input-dependent and input-agnostic cDAGs, showing limitations of the latter for approximating the former. These results offer a principled lens for analyzing architectural choices and inform design for improved compositional generalization in practical tasks. Overall, the framework provides rigorous, quantitative insights into when and why modern sequence models struggle with compositional tasks and how to engineer architectures that better exploit hierarchical structure.

Abstract

Compositionality is thought to be a key component of language, and various compositional benchmarks have been developed to empirically probe the compositional generalization of existing sequence processing models. These benchmarks often highlight failures of existing models, but it is not clear why these models fail in this way. In this paper, we seek to theoretically understand the role the compositional structure of the models plays in these failures and how this structure relates to their expressivity and sample complexity. We propose a general neuro-symbolic definition of compositional functions and their compositional complexity. We then show how various existing general and special purpose sequence processing models (such as recurrent, convolution and attention-based ones) fit this definition and use it to analyze their compositional complexity. Finally, we provide theoretical guarantees for the expressivity and systematic generalization of compositional models that explicitly depend on our proposed definition and highlighting factors which drive poor empirical performance.
Paper Structure (29 sections, 21 theorems, 46 equations, 7 figures, 2 tables)

This paper contains 29 sections, 21 theorems, 46 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Our contributions.
  3. Related Work
  4. Defining Compositionality
  5. Compositional Complexity.
  6. Cleanly-separable systematic sub-structures.
  7. Existing Models as Compositional Functions
  8. Recurrent Composition
  9. Convolutional Composition with Pooling
  10. Transformers
  11. Efficient transformers.
  12. Discussion
  13. Conclusion
  14. Technical Details for § \ref{['sec:cdefs']}
  15. Node ordering at each level of the cDAG
  16. ...and 14 more sections

Key Result

Proposition 1

With unidirectional recurrent composition, the maximum absolute LoI is ${\boldsymbol{\delta}} \triangleq c^{L-1}$, with a maximum relative LoI of ${\boldsymbol{\beta}} \triangleq (c^L - c^{L-1})/(2c^L -c^{L-1} - 1)$.

Figures (7)

  • Figure 1: cDAGs for $f(X)$ (left) and $f(X')$ (right) in \ref{['ex:cfunc-1']}. Nodes are labeled $l$:$i$ (level $l$, index $i$). Sources are Fuchsia, sinks are Sepia, and internal nodes are Blue.
  • Figure 2: cDAGs for $\mathsf f(X)$ (left) and $\mathsf f(X')$ (right) in \ref{['ex:cfunc-2']}. Nodes are labeled $l$:$i$ (level $l$, index $i$). Sources are Fuchsia, sinks are Sepia, and internal nodes are Blue.
  • Figure 3: Existing models - I. Nodes are labeled $l$:$i$ (level $l$, index $i$). Sources are Fuchsia, sinks are Sepia, and internal nodes are Blue.
  • Figure 4: Existing models - II. Nodes are labeled $l$:$i$ (level $l$, index $i$). Sources are Fuchsia, sinks are Sepia, and internal nodes are Blue.
  • Figure 5: Existing models - Extra details
  • ...and 2 more figures

Theorems & Definitions (43)

  • Definition 1
  • Example 1
  • Example 2
  • Definition 2: LoI of a source node
  • Definition 3
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • ...and 33 more