ShiftLUT: Spatial Shift Enhanced Look-Up Tables for Efficient Image Restoration

TL;DR

This work proposes ShiftLUT, a novel framework that attains the largest receptive field among all LUT-based methods while maintaining high efficiency, and proposes an asymmetric dual-branch architecture that allocates more computation to the information-dense branch, substantially reducing inference latency without compromising restoration quality.

Abstract

Look-Up Table based methods have emerged as a promising direction for efficient image restoration tasks. Recent LUT-based methods focus on improving their performance by expanding the receptive field. However, they inevitably introduce extra computational and storage overhead, which hinders their deployment in edge devices. To address this issue, we propose ShiftLUT, a novel framework that attains the largest receptive field among all LUT-based methods while maintaining high efficiency. Our key insight lies in three complementary components. First, Learnable Spatial Shift module (LSS) is introduced to expand the receptive field by applying learnable, channel-wise spatial offsets on feature maps. Second, we propose an asymmetric dual-branch architecture that allocates more computation to the information-dense branch, substantially reducing inference latency without compromising restoration quality. Finally, we incorporate a feature-level LUT compression strategy called Error-bounded Adaptive Sampling (EAS) to minimize the storage overhead. Compared to the previous state-of-the-art method TinyLUT, ShiftLUT achieves a 3.8$\times$ larger receptive field and improves an average PSNR by over 0.21 dB across multiple standard benchmarks, while maintaining a small storage size and inference time. The code is available at: https://github.com/Sailor-t/ShiftLUT .

Figures (10)

• Figure 1: Model comparison in terms of storage size, PSNR and runtime on Set5 for x4 super-resolution. Our method produces a family of models that has the smallest storage size and occupies the top-left corner, indicating superior performances (PSNR on y-axis) with fast inference speed (Runtime on x-axis).
• Figure 2: (a) Overall architecture of ShiftLUT. (b) The structure of LSS, which consists of an offset prediction network and a spatial shift operator. In Stage 1, the network predicts floating-point offsets $(\Delta x, \Delta y)$, which are applied via bilinear interpolation. In Stage 2, the offsets are replaced with integer-valued approximations, computed by rounding the average offset from Stage 1. (c) Illustration of the EAS inference pipeline with an example using two sampling steps. EAS precomputes and caches interpolated LUT outputs into a reusable buffer, replacing per-pixel interpolation with a single query operation for faster inference.
• Figure 3: Local Attribution Map (LAM) visualization for a $16 \times 16$ output patch. A larger DI indicates that a wider range of pixels contributes to the output result. Our method with LSS shows larger DI and better performance than the variant without LSS.
• Figure 4: The symmetric network architecture (top) and its corresponding LSB feature sparsity (bottom). The layer-wise analysis shows that feature sparsity in the LSB branch increases significantly with network depth.
• Figure 5: Qualitative comparison for $4\times$ super-resolution on different images.