Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers

TL;DR

This work tackles the problem of robustly classifying spatially transformed inputs by introducing Inverse Transformation Search (ITS), an inference-time mechanism that canonicalises perturbed inputs through a sparsified inverse transformation tree over the 2D affine group. ITS is model-agnostic and relies on energy- and group-based confidence measures, including a Monte Carlo dropout-based Bayesian view and a stable group-induced curvature score, to identify a canonical form without retraining. The authors demonstrate zero-shot improvements on rotated and affine perturbations across MNIST, GTSRB, and ImageNet-scale benchmarks (via ViT-B16 on SI-Rotation), showing ITS can surpass baselines that require explicit inductive biases or data augmentation. The approach enables robust spatial invariance for classifiers in settings where runtime can be traded for inference-time resilience, with clear limitations and directions for future work in extending transformations and reducing computational cost.

Abstract

Deep neural networks are applied in more and more areas of everyday life. However, they still lack essential abilities, such as robustly dealing with spatially transformed input signals. Approaches to mitigate this severe robustness issue are limited to two pathways: Either models are implicitly regularised by increased sample variability (data augmentation) or explicitly constrained by hard-coded inductive biases. The limiting factor of the former is the size of the data space, which renders sufficient sample coverage intractable. The latter is limited by the engineering effort required to develop such inductive biases for every possible scenario. Instead, we take inspiration from human behaviour, where percepts are modified by mental or physical actions during inference. We propose a novel technique to emulate such an inference process for neural nets. This is achieved by traversing a sparsified inverse transformation tree during inference using parallel energy-based evaluations. Our proposed inference algorithm, called Inverse Transformation Search (ITS), is model-agnostic and equips the model with zero-shot pseudo-invariance to spatially transformed inputs. We evaluated our method on several benchmark datasets, including a synthesised ImageNet test set. ITS outperforms the utilised baselines on all zero-shot test scenarios.

Figures (12)

• Figure 1: Inspired by the human recognition process for perturbed inputs, we propose a novel inference framework coined *its. Similar to humans transforming the given input mentally or physically by tilting their heads or stepping forward, for instance. Instead of the usual likelihood estimation of a given sample, the pre-trained model can evaluate different input transformations before a final decision is made. The output of this process is the most familiar transformation of the query.
• Figure 2: (left) Centre-Rotating an equilateral triangle and (right) an exemplary confidence surface.
• Figure 3: Probability of recognising a transformed sample to be the canonical form. Each evaluation point represents the mean over the FashionMNIST test set.
• Figure 4: Our proposed *its on a perturbed image. In each level of the search tree, the orbit of one subgroup is evaluated. All samples are evaluated using \ref{['eq:group_confidence']} and a pre-trained classifier.
• Figure 5: An example iteration of our algorithm with two hypotheses $H_0$ and $H_1$, and three generated subgroups, each with a cardinality of $|G|=9$. The first layer rotates the input, the second layer scales it, and the last layer shears it.

Theorems & Definitions (13)

• Definition 2.1: Group
• Definition 2.2: Generator
• Definition 2.3: Orbit
• Definition 2.4: Stabiliser
• Definition 2.5: Variances
• Theorem 3.1: Group Confidence
• Theorem 4.1: Group Elimination
• Lemma 4.1: Convolution over Finite Orbits
• proof
• proof