A Geometric Approach to Steerable Convolutions

TL;DR

This work addresses extending CNNs to full rotation and translation equivariance by proposing a geometric derivation of steerable convolutions in arbitrary $d$-dimensional space, grounded in the action of the special Euclidean group $SE(d)$ and Fourier-space representations. It develops a unified framework for equivariant linear maps, derives first- and higher-layer steerable convolution equations, and introduces a practical interpolation-based implementation that reduces parameters and enhances robustness to noise. The approach naturally explains the emergence of spherical harmonics and Clebsch–Gordan decompositions as intrinsic to the symmetry structure, and provides theoretical bounds on equivariance loss due to interpolation and discretization. Empirically, interpolation-based steerable filters outperform Cartesian-based methods across multiple 2D and 3D datasets and exhibit stronger resilience to perturbations, suggesting substantial practical impact for rotation-aware vision tasks and medical imaging applications.

Abstract

In contrast to the somewhat abstract, group theoretical approach adopted by many papers, our work provides a new and more intuitive derivation of steerable convolutional neural networks in $d$ dimensions. This derivation is based on geometric arguments and fundamental principles of pattern matching. We offer an intuitive explanation for the appearance of the Clebsch--Gordan decomposition and spherical harmonic basis functions. Furthermore, we suggest a novel way to construct steerable convolution layers using interpolation kernels that improve upon existing implementation, and offer greater robustness to noisy data.

Figures (8)

• Figure 1: Illustration of the core principle of steerable convolutions in two dimensions. For each pixel location $\mathbf{x}$, the filter is matched against the input at all possible orientations $\theta$. (a) The patch grid $f^{\textrm{patch}}_{\mathbf{x},\theta}$ is generally not aligned with the underlying image grid. (b) To address this, both the patches centered at $\mathbf{x}$ and the filters are expressed in polar coordinates $(r,\theta)$.
• Figure 2: Illustration of the interpolation kernel $\mathcal{I}$ at a fixed $\textbf{y}_0$ in one dimension. The nearest-neighbor interpolation is an example of a discontinuous kernel, hence corresponding to the case $\alpha=0$ in Definition \ref{['defn: interpolation']}(c). In contrast, linear interpolation varies linearly between adjacent grid points, hence corresponding to the case $\alpha=1$. In $d$ dimensions, the kernel $\mathcal{I}$ is constructed by multiplying the kernel values along each dimensions.
• Figure 3: Comparison of the two implementation strategies for steerable convolutions discussed in Remark \ref{['remark: time']}. Each network consists of two convolutional layers, with runtime measured across varying parameter settings shown on the $x$-axis. When one parameter (e.g., “Channels”) was varied, the others (“Radial Resolution” and “Frequency Cutoff”) were fixed at their largest values shown in the plot. Runtimes correspond to inputs of size $1\times28\times28$ for 2D and $1\times32\times32\times32$ for 3D, averaged over $100$ runs (excluding $5$ burn-ins) and normalized per sample using batchsizes of $10$ and $5$ for 2D and 3D, respectively.
• Figure 4: Illustration of a uniform grid on $\mathbb{S}^2$. The polar and azimuthal angles are uniformly discretized into $32$ grid points. Only discrete rotations about the $z$ axis preserve the grid structure. Rotations about the $x$ or $y$ axes generally do not map grid points back onto the grid, breaking rotational alignment.
• Figure 5: Simulation of equivariance error as a function of rotation in 2D and 3D. Each model consists of two convolutional layers separated by a normalization layer (Appendix \ref{['sec: Normalization']}) and followed by a flattening layer (Appendix \ref{['sec: Flattening']}), yielding rotation invariance. The error is defined as $\frac{\|\mathcal{M}(R\cdot f) - \mathcal{M}(f)\|_\infty}{\|f\|_1}$, where $\mathcal{M}$ denotes the model, $f$ the input, and $R$ the rotation. Model weights and inputs were sampled from a Gaussian distribution. Shaded regions indicate $90\%$ Wald confidence intervals over $100$ independent runs. A Fourier cutoff of $4$ was used for the 2D model and $0$ for the 3D model.

Theorems & Definitions (64)

• Definition 1
• Definition 2
• Proposition 1
• proof
• Theorem 1
• Remark 1
• Remark 2
• Proposition 2
• Remark 3
• Remark 4