FastJAM: a Fast Joint Alignment Model for Images

TL;DR

FastJAM addresses the need for fast, scalable joint alignment (JA) of image sets by replacing dense feature maps with a sparse keypoint (KP) graph and a lightweight Graph Neural Network. The method constructs a graph from pairwise KP matches (intra- and inter-image edges), applies a 5-layer GraphSAGE to propagate alignment cues, pools per-image embeddings, and predicts homographies via a Lie-algebraic, inverse-compositional objective. The core contributions include the first inverse-compositional JA loss over sparse KP representations, object-centric KP extraction with DP-Means clustering to reduce redundancy, and a pipeline that achieves under $50$ seconds on benchmarks while matching or exceeding prior JA methods in alignment quality. Empirically, FastJAM outperforms contemporary JA approaches on SPair-71k and CUB-200 in accuracy and dramatically reduces computation time, highlighting the practicality of sparse graphs for large-scale image alignment in real-world workflows.

Abstract

Joint Alignment (JA) of images aims to align a collection of images into a unified coordinate frame, such that semantically-similar features appear at corresponding spatial locations. Most existing approaches often require long training times, large-capacity models, and extensive hyperparameter tuning. We introduce FastJAM, a rapid, graph-based method that drastically reduces the computational complexity of joint alignment tasks. FastJAM leverages pairwise matches computed by an off-the-shelf image matcher, together with a rapid nonparametric clustering, to construct a graph representing intra- and inter-image keypoint relations. A graph neural network propagates and aggregates these correspondences, efficiently predicting per-image homography parameters via image-level pooling. Utilizing an inverse-compositional loss, that eliminates the need for a regularization term over the predicted transformations (and thus also obviates the hyperparameter tuning associated with such terms), FastJAM performs image JA quickly and effectively. Experimental results on several benchmarks demonstrate that FastJAM achieves results better than existing modern JA methods in terms of alignment quality, while reducing computation time from hours or minutes to mere seconds. Our code is available at our project webpage, https://bgu-cs-vil.github.io/FastJAM/

Figures (5)

• Figure 1: Joint alignment with FastJAM. Given a set of images of the same object, or of different objects from the same category (e.g., motorbikes), our method aligns all images in seconds, compared to other methods (minutes Barel:ECCV:2024:spacejam or hours Ofri:CVPR:2023:neuracongealinglGupta:ICCV:2023:ASIC).
• Figure 2: Overview of the FastJAM architecture. Given a set of images, we extract sparse keypoints (KPs) and pairwise correspondences using an off-the-shelf matcher (left; only red-dot matches are shown for clarity). A graph is built by linking KPs within each image (intra-image edges) and across matched pairs (inter-image edges). A GNN with $L$ layers propagates alignment information through this graph (center). Image-level features are then obtained via mean pooling and used to predict per-image homography parameters $(\btheta_i)$ for joint alignment.
• Figure 3: Comparison of joint alignment frameworks. Left: Atlas-based methods align each image independently to a canonical space $\mathcal{C}$ by minimizing variance. Middle: Existing inverse-compositional (IC) methods estimate $\mathcal{C}$ implicitly via relative transformations ($T^{\btheta_i} \circ T^{-\btheta_j}$), but process images independently. Right: FastJAM follows the IC paradigm, but differs from previous approaches in that 1) the loss is computed between KPs and 2) all images are processed simultaneously (the model process the entire KPs graph during its forward pass), allowing shared reasoning across all images.
• Figure 4: Canonical Space Visualization. We visualize the canonical space ($\mathcal{C}$) via a predefined RGB colormap. The first row shows an example of color projection from the canonical space onto a reference triangle. From the second row, we color each image $I_i$ by applying its inverse transformation on $\mathcal{C}$ (i.e., $\mathcal{C}\circ T^{-\btheta_i}$). FastJAM maps semantically similar regions to the same areas of $\mathcal{C}$, as shown by the consistent color mapping.
• Figure 5: JA Visual Comparison. We compare FastJAM with SpaceJAM Barel:ECCV:2024:spacejam on both rigid (TV) and non-rigid (Sheep) classes. In both cases, FastJAM alignment is visually better, where the improvement is particularly noticeable for close-up images, such as the middle or rightmost sheep.