LeRF: Learning Resampling Function for Adaptive and Efficient Image Interpolation

TL;DR

LeRF introduces Learning Resampling Function, a framework that learns per-pixel adaptive resampling functions to interpolate images efficiently while retaining the flexibility of DNN priors. By predicting per-pixel hyper-parameters for resampling functions (e.g., amplified linear or anisotropic Gaussian) via a neural network, LeRF blends traditional interpolation’s local continuity with data-driven priors. The authors present two model families: LUT-accelerated efficiency-oriented variants (LeRF-L/G/Net) with directional ensembles and edge-sensitive indexing, and a performance-oriented LeRF-Net++ that cascades with a pre-trained upsampler to approach DNN-level results. Extensive experiments demonstrate that LeRF-L/G can outperform fixed interpolation in arbitrary-scale upsampling, while LeRF-Net++ achieves competitive performance with much higher efficiency, confirming the method’s practical value for diverse devices and tasks.

Abstract

Image resampling is a basic technique that is widely employed in daily applications, such as camera photo editing. Recent deep neural networks (DNNs) have made impressive progress in performance by introducing learned data priors. Still, these methods are not the perfect substitute for interpolation, due to the drawbacks in efficiency and versatility. In this work, we propose a novel method of Learning Resampling Function (termed LeRF), which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption of interpolation. Specifically, LeRF assigns spatially varying resampling functions to input image pixels and learns to predict the hyper-parameters that determine the shapes of these resampling functions with a neural network. Based on the formulation of LeRF, we develop a family of models, including both efficiency-orientated and performance-orientated ones. To achieve interpolation-level efficiency, we adopt look-up tables (LUTs) to accelerate the inference of the learned neural network. Furthermore, we design a directional ensemble strategy and edge-sensitive indexing patterns to better capture local structures. On the other hand, to obtain DNN-level performance, we propose an extension of LeRF to enable it in cooperation with pre-trained upsampling models for cascaded resampling. Extensive experiments show that the efficiency-orientated version of LeRF runs as fast as interpolation, generalizes well to arbitrary transformations, and outperforms interpolation significantly, e.g., up to 3dB PSNR gain over Bicubic for x2 upsampling on Manga109. Besides, the performance-orientated version of LeRF reaches comparable performance with existing DNNs at much higher efficiency, e.g., less than 25% running time on a desktop GPU.

Figures (18)

• Figure 1: LeRF assigns resampling functions to input pixels and learns to predict the hyper-parameters that determine the orientations of these continuous functions for resampling under arbitrary transformations.
• Figure 2: Performance-efficiency trade-off of image resampling methods. mPSNR values are obtained on DIV2K for in-scale homographic warping. The running time is evaluated on an NVIDIA RTX 3090 GPU for producing $1280 \times 720$ images through $\times 4$ upsampling.
• Figure 3: Comparison of exiting resampling methods and LeRF. (a) Interpolation assumes a spatial-invariant fixed function $\Phi(\cdot)$ to predict resampling weights $W$. Representative method: Bilinear, Bicubic. (b) adaptive resampling methods interpolate or filter the interpolated image based on hand-designed rules $\Gamma$. Representative method: NEDI DBLP:journals/tip/LiO01a and RAISR DBLP:journals/tci/RomanoIM17. (c) DNN-based methods rely on feature extraction network ($f_e(\cdot)$), reconstruction network ($f_r(\cdot)$), and kernel prediction network ($f_{k}(\cdot)$) or implicit neural function (unified $f_{k}(\cdot)$ and $f_{r}(\cdot)$). Representative method: Meta-SR DBLP:conf/cvpr/HuM0WT019, LIIF DBLP:conf/cvpr/ChenL021, SRwarp DBLP:conf/cvpr/SonL21, and LTEW DBLP:conf/eccv/LeeCJ22. (b) LeRF learns spatially varying resampling function $\Phi_{\Theta}$ with a DNN $f(\cdot)$ that predicts hyper-parameters $\Theta$ for each pixel.
• Figure 4: Visualization of fixed resampling functions (left) and adaptive resampling functions (right).
• Figure 5: LUT acceleration of DNN and our adaptations. (a) A learned neural network can be accelerated by traversing all possible inputs $i_*$, pre-computing all corresponding outputs $v_*$, and saving them to LUTs DBLP:conf/cvpr/JoK21mulutmulut23splut. (b) In the proposed directional ensemble strategy, only predictions along the same direction are averaged, instead of all directions in rotation ensemble DBLP:conf/cvpr/JoK21. (c) We introduce edge-sensitive indexing patterns to better capture edge orientations. The pixels covered by directional ensemble are depicted with dashed boxes.