QueryCDR: Query-Based Controllable Distortion Rectification Network for Fisheye Images

TL;DR

QueryCDR tackles the challenge of rectifying fisheye distortions across varying distortion degrees without retraining. It introduces a Distortion-aware Learnable Query Mechanism (DLQM) to generate per-layer, position-aware control signals from a set of learnable queries, and pairs it with two controllable blocks—Controllable Convolution Modulating Block ($\text{CCMB}$) and Controllable Attention Modulating Block ($\text{CAMB}$)—to guide rectification in a U-shaped, multi-scale framework. The method is trained in two stages (coarse pre-training on a single distortion degree and fine-tuning across multiple degrees) to achieve broad generalization, validated by quantitative PSNR/SSIM gains and qualitative texture preservation on synthetic and real fisheye data. The results demonstrate superior performance across nine distortion degrees without retraining, enabling practical, controllable fisheye rectification with reduced data collection costs. Overall, DLQM and the dual-modulation architecture offer a robust, scalable solution for real-world fisheye correction with varying distortion patterns.

Abstract

Fisheye image rectification aims to correct distortions in images taken with fisheye cameras. Although current models show promising results on images with a similar degree of distortion as the training data, they will produce sub-optimal results when the degree of distortion changes and without retraining. The lack of generalization ability for dealing with varying degrees of distortion limits their practical application. In this paper, we take one step further to enable effective distortion rectification for images with varying degrees of distortion without retraining. We propose a novel Query-Based Controllable Distortion Rectification network for fisheye images (QueryCDR). In particular, we first present the Distortion-aware Learnable Query Mechanism (DLQM), which defines the latent spatial relationships for different distortion degrees as a series of learnable queries. Each query can be learned to obtain position-dependent rectification control conditions, providing control over the rectification process. Then, we propose two kinds of controllable modulating blocks to enable the control conditions to guide the modulation of the distortion features better. These core components cooperate with each other to effectively boost the generalization ability of the model at varying degrees of distortion. Extensive experiments on fisheye image datasets with different distortion degrees demonstrate our approach achieves high-quality and controllable distortion rectification.

Figures (7)

• Figure 1: Different approaches to fisheye image distortion rectification. (a) Regression-Based: Using a neural network to predict distortion-related parameters, then apply rectification algorithms $R$ for rectification. (b) Generation-Based: Input the distorted fisheye image and directly generate the rectified image end-to-end. (c) Control-Based: Users provide control conditions to guide the rectification process, resulting in promising rectified images of various distortion degrees.
• Figure 2: Overview of our proposed Query-Based Controllable Distortion Rectification network (QueryCDR). The Distortion-aware Learnable Query Mechanism (DLQM) extracts control conditions from user-given queries and feeds them layer by layer into the rectification network. The rectification network is composed of Controllable Convolution Modulating Blocks (CCMB) and Controllable Attention Modulating Blocks (CAMB), which modulate the input features $F_{in}$ with control conditions $F_c$, enabling controllable rectification process.
• Figure 3: Given an input image, QueryCDR can accurately produce results with different rectification degrees by feeding different queries. Moreover, by interpolating between different queries, we can achieve smooth continuous rectification for any distortion degree. For examples, $Q_{1.25}=0.75Q_1+0.25Q_2$, and $Q_{8.5}=0.5Q_8+0.5Q_9$.
• Figure 4: Qualitative results on synthetic fisheye images.
• Figure 5: Qualitative results on real-world fisheye images.