Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers

Pim de Haan, Taco Cohen, Johann Brehmer

TL;DR

This work generalizes the GATr architecture into a blueprint that allows one to construct a scalable transformer architecture given any geometric (or Clifford) algebra, and studies versions of this architecture for Euclidean, projective, and conformal algebras.

Abstract

The Geometric Algebra Transformer (GATr) is a versatile architecture for geometric deep learning based on projective geometric algebra. We generalize this architecture into a blueprint that allows one to construct a scalable transformer architecture given any geometric (or Clifford) algebra. We study versions of this architecture for Euclidean, projective, and conformal algebras, all of which are suited to represent 3D data, and evaluate them in theory and practice. The simplest Euclidean architecture is computationally cheap, but has a smaller symmetry group and is not as sample-efficient, while the projective model is not sufficiently expressive. Both the conformal algebra and an improved version of the projective algebra define powerful, performant architectures.

Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers

TL;DR

This work generalizes the GATr architecture into a blueprint that allows one to construct a scalable transformer architecture given any geometric (or Clifford) algebra, and studies versions of this architecture for Euclidean, projective, and conformal algebras.

Abstract

The Geometric Algebra Transformer (GATr) is a versatile architecture for geometric deep learning based on projective geometric algebra. We generalize this architecture into a blueprint that allows one to construct a scalable transformer architecture given any geometric (or Clifford) algebra. We study versions of this architecture for Euclidean, projective, and conformal algebras, all of which are suited to represent 3D data, and evaluate them in theory and practice. The simplest Euclidean architecture is computationally cheap, but has a smaller symmetry group and is not as sample-efficient, while the projective model is not sufficiently expressive. Both the conformal algebra and an improved version of the projective algebra define powerful, performant architectures.
Paper Structure (35 sections, 3 theorems, 28 equations, 2 figures, 2 tables)

This paper contains 35 sections, 3 theorems, 28 equations, 2 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. GEOMETRIC ALGEBRAS
  3. Geometric algebra
  4. Geometric algebras for 3D space
  5. THE GENERALIZED GEOMETRIC ALGEBRA TRANSFORMER
  6. The original Geometric Algebra Transformer
  7. Generalizing GATr to arbitrary algebras
  8. Constructing equivariant maps
  9. Normalization layers
  10. THEORETICAL COMPARISON
  11. Multilinear expressivity
  12. Absolute positions
  13. Distance-based attention
  14. EXPERIMENTS
  15. $n$-body modelling
  16. ...and 20 more sections

Key Result

Proposition 1

Let $l \ge 1$.

Figures (2)

  • Figure 1: The representations of points in the EGA, PGA and CGA, shown with one spatial dimension for visualization clarity. The dashed lines shows the possible coordinates of points.
  • Figure 2: $n$-body modelling. We show the mean squared error as a function of the number of training samples. We compare E-GATr, P-GATr, iP-GATR, and C-GATr to the equivariant SE(3)-Transformer Fuchs2020-bw and SEGNN Brandstetter2022-hw as well as to a vanilla transformer.

Theorems & Definitions (7)

  • Proposition 1
  • Conjecture 2
  • Proposition 3
  • proof
  • Proposition 4
  • proof
  • Conjecture 5