Anisotropic Pooling for LUT-realizable CNN Image Restoration

TL;DR

This work tackles the isotropy-induced limitations of LUT-based CNN image restoration by introducing anisotropic pooling methods. It formalizes a multi-orientation fusion framework and proposes two mechanisms: Generalized Median Pooling (GMP) for robust, differentiable outlier suppression and Orientation-Aware Pooling (OAP), which uses a tiny coefficient LUT to predict content-dependent orientation weights, preserving LUT efficiency. The methods are integrated into three LUT-based baselines and evaluated across super-resolution, denoising, deblocking, and deblurring, showing consistent PSNR/SSIM gains with negligible overhead; OAP generally provides the strongest improvements and near-lossless deployment profiles. These results demonstrate that adaptive orientation fusion can substantially narrow the gap between lightweight LUT approaches and larger CNNs, enabling high-quality restoration on edge devices and real-time systems. $\hat{\mathbf{y}}(\mathbf{p}) = \sum_{i=1}^{k} \alpha_i(\mathbf{p}) \mathbf{x}_i$, with $\boldsymbol{\alpha}(\mathbf{p}) \in \Delta^{k-1}$, and GMP as $\alpha_i^{\text{GMP}} = \frac{\exp(-d_i/\tau)}{\sum_j \exp(-d_j/\tau)}$, where $d_i = \|\mathbf{x}_i - \bar{\mathbf{x}}\|$ and $\mathbf{y}^{\text{GMP}} = \sum_i \alpha_i^{\text{GMP}} \mathbf{x}_i$.

Abstract

Table look-up realization of image restoration CNNs has the potential of achieving competitive image quality while being much faster and resource frugal than the straightforward CNN implementation. The main technical challenge facing the LUT-based CNN algorithm designers is to manage the table size without overly restricting the receptive field. The prevailing strategy is to reuse the table for small pixel patches of different orientations (apparently assuming a degree of isotropy) and then fuse the look-up results. The fusion is currently done by average pooling, which we find being ill suited to anisotropic signal structures. To alleviate the problem, we investigate and discuss anisotropic pooling methods to replace naive averaging for improving the performance of the current LUT-realizable CNN restoration methods. First, we introduce the method of generalized median pooling which leads to measurable gains over average pooling. We then extend this idea by learning data-dependent pooling coefficients for each orientation, so that they can adaptively weigh the contributions of differently oriented pixel patches. Experimental results on various restoration benchmarks show that our anisotropic pooling strategy yields both perceptually and numerically superior results compared to existing LUT-realizable CNN methods.

Figures (6)

• Figure 1: Trade-off between PSNR and storage size for the $\times4$ image super-resolution task. The proposed orientation-aware pooling (OAP) mechanism consistently improves the reconstruction quality of LUT-based super-resolution models, while introducing only a negligible increase in storage.
• Figure 2: Illustration of the average pooling mechanism in a LUT-realizable CNN for image restoration. A $2\times2$ input patch is rotated by $0^\circ$, $90^\circ$, $180^\circ$, and $270^\circ$, processed by the network, and then inversely rotated before being merged through average pooling. This common averaging operation effectively expands the receptive field to $3\times3$, but implicitly assumes that all orientations contribute equally.
• Figure 3: Illustration of anisotropic signal structures within a $3\times3$ image patch (extracted from the red square). Different orientations produce distinct estimates, showing that simple averaging (108) can yield suboptimal results when outliers (56) distort the prediction. In contrast, the proposed generalized median pooling (GMP = 132.5) provides a more robust and accurate estimate, closely matching the target value (= 130).
• Figure 4: Illustration of the proposed generalized median pooling (GMP) strategy. Patches whose values are closer to the average receive higher weights, while outliers are downweighted, resulting in a robust, median-like aggregation that suppresses anomalies and preserves structural consistency.
• Figure 5: Overview of the proposed OAP (referenced in Sec. \ref{['subsec:oap']}). A tiny coefficient LUT $C(\mathbf{p})$ predicts orientation weights $\alpha_i$ from the input patch, which then fuse the inverse-rotated predictions from $k$ orientations. The mechanism favors orientations aligned with local structures while preserving LUT efficiency.