Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tunable Soft Equivariance with Guarantees

Md Ashiqur Rahman, Lim Jun Hao, Jeremiah Jiang, Teck-Yian Lim, Raymond A. Yeh

Abstract

Equivariance is a fundamental property in computer vision models, yet strict equivariance is rarely satisfied in real-world data, which can limit a model's performance. Controlling the degree of equivariance is therefore desirable. We propose a general framework for constructing soft equivariant models by projecting the model weights into a designed subspace. The method applies to any pre-trained architecture and provides theoretical bounds on the induced equivariance error. Empirically, we demonstrate the effectiveness of our method on multiple pre-trained backbones, including ViT and ResNet, across image classification, semantic segmentation, and human-trajectory prediction tasks. Notably, our approach improves the performance while simultaneously reducing equivariance error on the competitive ImageNet benchmark.

Tunable Soft Equivariance with Guarantees

Abstract

Equivariance is a fundamental property in computer vision models, yet strict equivariance is rarely satisfied in real-world data, which can limit a model's performance. Controlling the degree of equivariance is therefore desirable. We propose a general framework for constructing soft equivariant models by projecting the model weights into a designed subspace. The method applies to any pre-trained architecture and provides theoretical bounds on the induced equivariance error. Empirically, we demonstrate the effectiveness of our method on multiple pre-trained backbones, including ViT and ResNet, across image classification, semantic segmentation, and human-trajectory prediction tasks. Notably, our approach improves the performance while simultaneously reducing equivariance error on the competitive ImageNet benchmark.

Paper Structure

This paper contains 44 sections, 4 theorems, 109 equations, 8 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Towards Soft Equivariant Networks
  5. Soft equivariance for continuous groups
  6. Soft equivariance for discrete groups
  7. Soft equivariant layers in practice
  8. Experiments
  9. Validating tunable softness level
  10. Image classification
  11. Semantic segmentation
  12. Human trajectory prediction
  13. Ablation studies
  14. Conclusion
  15. Additional Visualization following Fig. \ref{['fig:teaser']}
  16. ...and 29 more sections

Key Result

Lemma 1

The weight matrix $\Theta$ is equivariant if it satisfies the condition where $\Theta'_{lk}$ are blocks of $\Theta'$ corresponding to the blocks of ${\bm{\Sigma}}_{\mathcal{Y}}$ and ${\bm{\Sigma}}_{\mathcal{X}}$ of dimensions $\dim({\bm{T}}_l)\times\dim({\bm{S}}_k)$.

Figures (8)

  • Figure 1: Visualization of the ViT dosovitskiy2020image weights with our soft equivariance layer (w.r.t.$90^\circ$ rotation) under different softness levels, along with the corresponding extracted features and the equivariance errors. Our tunable design allows the layers' weights to transition smoothly from perfectly equivariant to fully non-equivariant behavior in a controlled manner.
  • Figure 2: Tunable softness results (cAcc & iErr). Compared to RPP, ours achieves comparable performance with better iErr across two groups.
  • Figure 3: Visualization of the ViT dosovitskiy2020image weights with our soft equivariance layer (w.r.t.$90^\circ$ rotation) under different softness levels, along with the corresponding extracted features and the equivariance errors. Our tunable design allows the layers' weights to transition smoothly from perfectly equivariant to fully non-equivariant behavior in a controlled manner.
  • Figure 4: Visualization of the ViT dosovitskiy2020image weights with our soft equivariance layer (w.r.t.$10^\circ$ rotation) under different softness levels, along with the corresponding extracted features and the equivariance errors. Our tunable design allows the layers' weights to transition smoothly from perfectly equivariant to fully non-equivariant behavior in a controlled manner.
  • Figure 5: Visualization of condition of strict equivariance condition on the weight $\Theta'$ which is described in Lemma \ref{['clm:schur_equiv']}.
  • ...and 3 more figures

Theorems & Definitions (22)

  • Definition 1: $\eta$-Soft Equivariant
  • Claim 1
  • proof
  • Claim 2
  • proof
  • Lemma 1
  • proof
  • Claim 3
  • proof
  • Lemma 2
  • ...and 12 more