Auto-regressive transformation for image alignment

TL;DR

Auto-Regressive Transformation (ART) tackles robust image alignment under challenging conditions such as feature sparsity, extreme scale differences, and large deformations by performing coarse-to-fine, auto-regressive refinement of per-pixel transformations. It fuses multi-scale pyramid features with a cross-attention layer to guide refinement toward relevant regions and introduces stochastic point supervision to learn from diverse samples. ART demonstrates state-of-the-art performance across retinal and planar benchmarks, outperforming feature-based, intensity-based, and iterative refinement methods, and shows robustness to initialization and domain shifts. The approach offers flexible inference by adjusting the number of refinement steps and holds practical promise for applications in medical imaging, remote sensing, and scene analysis where reliable alignment is essential.

Abstract

Existing methods for image alignment struggle in cases involving feature-sparse regions, extreme scale and field-of-view differences, and large deformations, often resulting in suboptimal accuracy. Robustness to these challenges improves through iterative refinement of the transformation field while focusing on critical regions in multi-scale image representations. We thus propose Auto-Regressive Transformation (ART), a novel method that iteratively estimates the coarse-to-fine transformations within an auto-regressive framework. Leveraging hierarchical multi-scale features, our network refines the transformations using randomly sampled points at each scale. By incorporating guidance from the cross-attention layer, the model focuses on critical regions, ensuring accurate alignment even in challenging, feature-limited conditions. Extensive experiments across diverse datasets demonstrate that ART significantly outperforms state-of-the-art methods, establishing it as a powerful new method for precise image alignment with broad applicability.

Figures (8)

• Figure 1: Alignment Results in Challenging Scenarios. For image pairs with sparse features, scale differences, deformations, degradations, and domain shifts, our method performs coarse-to-fine auto-regressive transformation refinement, achieving superior alignment even in challenging scenarios where state-of-the-art method struggles. The zoomed-in boxes show the local alignment results, and the highlighted vessel image below illustrates the intersection (yellow) between the two images (red and green).
• Figure 2: Method Overview. Auto-Regressive Transformation (ART) iteratively refines the transformation $\mathcal{D}$ for image pairs $\mathcal{I}$ in a coarse-to-fine manner. This sampling strategy allows ART to operate across diverse domains, including retinal, scenery, and map datasets.
• Figure 3: Overall Framework.ART first extracts multi-scale features $\mathcal{F}_s$ and $\mathcal{F}_d$ from the input image pair $\mathcal{I}_s$ and $\mathcal{I}_d$. At each sampling step $k$, the corresponding features, $\mathcal{F}_s^k$ and $\mathcal{F}_d^k$, are passed through the Cross-Attention Layer (CAL) to identify the correlated features that guide the network's focus on regions requiring refinement. The attentive feature map $\tilde{\mathcal{F}}_{s \rightarrow d}^k$ is then used to refine the transform parameters $\mathcal{D}_{\mathcal{M}}^k$ and $\mathcal{D}_{\mathcal{A}}^k$ to $\mathcal{D}_{\mathcal{M}}^{k+1}$ and $\mathcal{D}_{\mathcal{A}}^{k+1}$ through multiple convolutional networks. This auto-regressive process continues until the initialized transform parameters $\mathcal{D}_{\mathcal{M}}^0$ and $\mathcal{D}_{\mathcal{A}}^0$ reach the full resolution of the input image pair $\mathcal{I}_s$ and $\mathcal{I}_d$.
• Figure 4: Point Warping. At sampling step $k$, the extracted source points set $\mathcal{P}_s^k$ is warped to $\tilde{\mathcal{P}}_{s \rightarrow d}^k$ by sequentially multiplying with the corresponding values of the transform parameter $\mathcal{D}_{\mathcal{M}}^k$ and adding $\mathcal{D}_{\mathcal{A}}^k$ for each point.
• Figure 5: Qualitative Evaluation on Retinal Datasets. Across various domains, ART robustly identifies sufficient matches compared to SuperRetina liu2022semi, GeoFormer liu2023geometrized and RetinaRegNet sivaraman2024retinaregnetzeroshotapproachretinal. Source key points for corrrespondence search are selected from randomly distinct positions. The zoomed-in boxes highlight overlaid local regions.