Test-time Cost-and-Quality Controllable Arbitrary-Scale Super-Resolution with Variable Fourier Components

TL;DR

This work proposes a novel SR method using a Recurrent Neural Network (RNN) with the Fourier representation, which achieves a lower PSNR drop than other state-of-the-art arbitrary-scale SR methods.

Abstract

Super-resolution (SR) with arbitrary scale factor and cost-and-quality controllability at test time is essential for various applications. While several arbitrary-scale SR methods have been proposed, these methods require us to modify the model structure and retrain it to control the computational cost and SR quality. To address this limitation, we propose a novel SR method using a Recurrent Neural Network (RNN) with the Fourier representation. In our method, the RNN sequentially estimates Fourier components, each consisting of frequency and amplitude, and aggregates these components to reconstruct an SR image. Since the RNN can adjust the number of recurrences at test time, we can control the computational cost and SR quality in a single model: fewer recurrences (i.e., fewer Fourier components) lead to lower cost but lower quality, while more recurrences (i.e., more Fourier components) lead to better quality but more cost. Experimental results prove that more Fourier components improve the PSNR score. Furthermore, even with fewer Fourier components, our method achieves a lower PSNR drop than other state-of-the-art arbitrary-scale SR methods.

Figures (8)

• Figure 1: Conceptual illustration of the proposed CQ controllable arbitrary-scale SR. The RNN-based network estimates Fourier components one by one. By adjusting the number of RNN recurrences, the model can control the trade-off between computational cost and SR quality.
• Figure 2: Overview of LIIF-based arbitrary-scale SR methods. The implicit function, $f$, is designed for each method.
• Figure 3: Coordinate system in LIIF-based methods. The blue-colored symbols indicate the latent code, while black-colored symbols indicate the position. $\mathbf{z}_{i,j}$ denotes the latent code at $(i, j)$, where $i$ and $j$ are the indices of the latent representation. $\mathbf{z}_{i,j}$ is tiled in the HR coordinate system at regular intervals. The latent code nearest to the query position, $x_q$, is selected as $\mathbf{z}$, and its position in the HR coordinate system is denoted as $\mathbf{v}^*$.
• Figure 4: Implementations of the implicit function, $f$, in (a) LIIF liif and (b) LTE lte.
• Figure 5: Our proposed implicit function. $\boldsymbol{\alpha}_t$ and $\boldsymbol{\beta}_t$ denotes the hidden state and output of RNN at the $t$-th recurrence. The initial hidden state $\boldsymbol{\alpha}_0$ is obtained from the latent code $\mathbf{z}$ through the function $\phi$. $\boldsymbol{\beta}_t$ is fed into the amplitude estimator $h_a$ and frequency estimator $h_f$, and obtain $\mathbf{A}_t$ and $\mathbf{F}_t$. After that, $\mathbf{A}_t$ and $\mathbf{F}_t$ are transformed by $\gamma$ and then passed to the next RNN recurrence. Additionally, phase information is estimated from the cell size, $\mathbf{c}$, $\mathbf{A}_t$, and $\mathbf{F}_t$, as described in Sec. \ref{['sec:methods']}-\ref{['sec:others']}. Since the number of RNN recurrences $T$ can be adjusted freely at the test time, our proposed method can produce a variable number of Fourier components. These estimated Fourier components are summed to produce the final RGB value.