Spectral Complex Autoencoder Pruning: A Fidelity-Guided Criterion for Extreme Structured Channel Compression

TL;DR

SCAP tackles the problem of extreme structured channel pruning by measuring per-channel redundancy with a spectral, frequency-domain, complex interaction descriptor learned by a tiny autoencoder. A fidelity-based score, optionally fused with a magnitude term, drives threshold-based pruning to yield structurally valid, highly compressed networks with modest accuracy loss after fine-tuning. The approach is memory-efficient, layer-local, and supported by a theoretical link between fidelity and normalized reconstruction error, plus an alignment-based perturbation bound for the retained channels. Empirical results on CIFAR-10/100 across VGG16, ResNet, and DenseNet backbones demonstrate substantial FLOP and parameter reductions (often >80–90%), with competitive or favorable accuracy recovery after short fine-tuning. The method offers a principled, scalable pathway to deployment-ready model compression in resource-constrained environments.

Abstract

We propose Spectral Complex Autoencoder Pruning (SCAP), a reconstruction-based criterion that measures functional redundancy at the level of individual output channels. For each convolutional layer, we construct a complex interaction field by pairing the full multi-channel input activation as the real part with a single output-channel activation (spatially aligned and broadcast across input channels) as the imaginary part. We transform this complex field to the frequency domain and train a low-capacity autoencoder to reconstruct normalized spectra. Channels whose spectra are reconstructed with high fidelity are interpreted as lying close to a low-dimensional manifold captured by the autoencoder and are therefore more compressible; conversely, channels with low fidelity are retained as they encode information that cannot be compactly represented by the learned manifold. This yields an importance score (optionally fused with the filter L1 norm) that supports simple threshold-based pruning and produces a structurally consistent pruned network. On VGG16 trained on CIFAR-10, at a fixed threshold of 0.6, we obtain 90.11% FLOP reduction and 96.30% parameter reduction with an absolute Top-1 accuracy drop of 1.67% from a 93.44% baseline after fine-tuning, demonstrating that spectral reconstruction fidelity of complex interaction fields is an effective proxy for channel-level redundancy under aggressive compression.

Figures (9)

• Figure 1: Overview of SCAP (Spectral Complex Autoencoder Pruning). The method operates layer-by-layer and contains two stages. Training: for each convolutional layer $\ell$ and each output channel $k$, we build a channel-wise interaction descriptor by using the layer input activation as the real component and the resized-and-broadcast output feature map $Y^{(\ell)}_k$ as the imaginary component (broadcast to match the channel dimension of the input). After standardization and FFT, we obtain frequency-domain real/imag components, and train lightweight autoencoders separately for the real and imaginary parts using the average of the two MSE reconstruction losses. Pruning: given the trained per-layer autoencoders, we compute each channel's self-reconstruction fidelity by applying the same preprocessing, reconstructing in the frequency domain, transforming back with inverse FFT, and measuring cosine fidelity between the original and reconstructed (real, imag) interaction components. We define importance as $1-\mathrm{Fid}$ and optionally fuse it with a layer-normalized set-$\ell_1$ magnitude term (additive fusion). Channels are kept or removed by a fixed threshold $\tau$ (we report only $\tau\in\{0.5,0.6\}$), followed by standard fine-tuning of the structured pruned network.
• Figure 2: Ablation at threshold $\tau=0.5$ on CIFAR-100. l1-none denotes fidelity-only importance (removing the $\ell_1$ magnitude term), while SCAP denotes the default additive fusion (fidelity + set-$\ell_1$). We report Top-1 accuracy during fine-tuning for different backbones.
• Figure 3: Compression comparison (FR/PR) for the ablation at $\tau=0.5$ on CIFAR-100. l1-none removes the $\ell_1$ magnitude term; SCAP uses the default additive fusion (fidelity + set-$\ell_1$).
• Figure 4: Ablation at threshold $\tau=0.6$ on CIFAR-100. l1-none denotes fidelity-only importance (removing the $\ell_1$ magnitude term), while SCAP denotes the default additive fusion (fidelity + set-$\ell_1$). We report Top-1 accuracy during fine-tuning for different backbones.
• Figure 5: Compression comparison (FR/PR) for the ablation at $\tau=0.6$ on CIFAR-100. l1-none removes the $\ell_1$ magnitude term; SCAP uses the default additive fusion (fidelity + set-$\ell_1$).

Theorems & Definitions (8)

• Proposition 1: Fidelity--error identity
• proof
• Lemma 6.1: Aligned channel extraction is stable
• proof
• Lemma 6.2: Non-normalized fidelity--error identity
• proof
• Theorem 1: Fidelity bounds aligned channel perturbation
• proof