SONIC: Spectral Oriented Neural Invariant Convolutions

TL;DR

SONIC introduces a principled, resolution-invariant spectral operator for deep networks by representing filters in the continuous frequency domain with a small set of oriented modes. Each mode uses a transfer function $T_m(\boldsymbol{\omega})$ parameterised by orientation $\boldsymbol{v_m}$, scale $s_m$, complex damping $a_m$, and transverse decay $\tau_m$, combined across channels via low-rank matrices $B$ and $C$ to form $\widehat{H}_{k,c}(\boldsymbol{\omega}) = \sum_m C_{km} T_m(\boldsymbol{\omega}) B_{mc}$; the model is trained end-to-end with FFT-based filtering and a residual spatial update. This framework delivers global receptive fields with substantial parameter efficiency, and demonstrates robustness to geometric transformations, noise, and resolution shifts across synthetic benchmarks, 3D medical image segmentation, and large-scale natural images, while frequently matching or exceeding convolutional and attention-based baselines at an order of magnitude fewer parameters. By decoupling the spectral operator from discrete grid sampling, SONIC provides a scalable alternative to conventional spatial or spectral operators, with potential for hybrid spectral-spatial architectures to further enhance local detail and global context.

Abstract

Convolutional Neural Networks (CNNs) rely on fixed-size kernels scanning local patches, which limits their ability to capture global context or long-range dependencies without very deep architectures. Vision Transformers (ViTs), in turn, provide global connectivity but lack spatial inductive bias, depend on explicit positional encodings, and remain tied to the initial patch size. Bridging these limitations requires a representation that is both structured and global. We introduce SONIC (Spectral Oriented Neural Invariant Convolutions), a continuous spectral parameterisation that models convolutional operators using a small set of shared, orientation-selective components. These components define smooth responses across the full frequency domain, yielding global receptive fields and filters that adapt naturally across resolutions. Across synthetic benchmarks, large-scale image classification, and 3D medical datasets, SONIC shows improved robustness to geometric transformations, noise, and resolution shifts, and matches or exceeds convolutional, attention-based, and prior spectral architectures with an order of magnitude fewer parameters. These results demonstrate that continuous, orientation-aware spectral parameterisations provide a principled and scalable alternative to conventional spatial and spectral operators.

Figures (11)

• Figure 1: (a) Taxonomy of large receptive-field operators.
• Figure 2: A SONIC block applies a learned frequency response $\widehat{H}(\omega)$ to the input: the feature map is transformed to the Fourier domain, modulated by $\widehat{H}$, and returned to the spatial domain before normalization and a residual ReLU fusion.
• Figure 4: Comparison of ResNet-50 variants and related architectures on ImageNet under $224\times224$ evaluation.
• Figure 5: Relative performance degradation under resolution changes on ImageNet.
• Figure 7: Qualitative visual examples on the SynthShape benchmark