Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Product Interaction: An Algebraic Formalism for Deep Learning Architectures

Haonan Dong, Chun-Wun Cheng, Angelica I. Aviles-Rivero

TL;DR

This work introduces product interactions, a unifying algebraic framework in which neural network layers are generated by composing a single multiplication operator over structured algebras and then applying nonlinear activations. It shows that CNNs and symmetry-aware networks correspond to order $1$ product interactions, while attention corresponds to order $3$ and multi-level constructions yield SE(3) attention and tensor-product attention at higher orders. By formalizing signal embedding, a symmetry principle, and a systematic path to increasing interaction order, the framework explains a wide range of architectures as instances of the same algebraic primitive. The authors also discuss design principles, including the impact of symmetry constraints and algebraic structure on performance, and outline future directions for exploring higher-order interactions and new algebraic constructions.

Abstract

In this paper, we introduce product interactions, an algebraic formalism in which neural network layers are constructed from compositions of a multiplication operator defined over suitable algebras. Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order. Our central observation is that algebraic expressions in modern neural networks admit a unified construction in terms of linear, quadratic, and higher-order product interactions. Convolutional and equivariant networks arise as symmetry-constrained linear product interactions, while attention and Mamba correspond to higher-order product interactions.

Product Interaction: An Algebraic Formalism for Deep Learning Architectures

TL;DR

This work introduces product interactions, a unifying algebraic framework in which neural network layers are generated by composing a single multiplication operator over structured algebras and then applying nonlinear activations. It shows that CNNs and symmetry-aware networks correspond to order product interactions, while attention corresponds to order and multi-level constructions yield SE(3) attention and tensor-product attention at higher orders. By formalizing signal embedding, a symmetry principle, and a systematic path to increasing interaction order, the framework explains a wide range of architectures as instances of the same algebraic primitive. The authors also discuss design principles, including the impact of symmetry constraints and algebraic structure on performance, and outline future directions for exploring higher-order interactions and new algebraic constructions.

Abstract

In this paper, we introduce product interactions, an algebraic formalism in which neural network layers are constructed from compositions of a multiplication operator defined over suitable algebras. Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order. Our central observation is that algebraic expressions in modern neural networks admit a unified construction in terms of linear, quadratic, and higher-order product interactions. Convolutional and equivariant networks arise as symmetry-constrained linear product interactions, while attention and Mamba correspond to higher-order product interactions.
Paper Structure (49 sections, 140 equations, 1 figure, 8 tables)

This paper contains 49 sections, 140 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Existing Algebraic Viewpoints.
  4. Product Interaction
  5. Algebraic Formulation
  6. Product interaction
  7. Symmetry Principle
  8. Product Interaction in Deep Learning
  9. Product interaction with self-interaction order $1$
  10. Product interaction with self-interaction order $2$
  11. Product interaction with self-interaction order $3$
  12. Multi-level product interaction with higher order
  13. Structure of Product Interactions
  14. Conclusion and Future Work
  15. Mamba as product interaction operator
  16. ...and 34 more sections

Figures (1)

  • Figure 1: Summary of networks involved in main text and their corresponding product interaction class. The red block is determined by symmetry principle. The green line shows increasing interaction order in Mamba. The yellow line shows transition from CNN to attention which increases interaction order form 1 to 3. The blue line is multi-level construction of SE(3) attention which increase interaction order of TFNs from 1 to 3.

Theorems & Definitions (9)

  • Definition 3.1: Algebra
  • Example 3.2: Auxiliary structural algebra 1
  • Example 3.3: Auxiliary structural algebra 2
  • Definition 3.4: Signal embedding
  • Example 3.5: Structural operator
  • Definition 3.6: Multiplication operator
  • Definition 3.7: Product interaction
  • Definition 3.8: Self-interaction order
  • Definition 3.9: Symmetry principle