MatchAttention: Matching the Relative Positions for High-Resolution Cross-View Matching

TL;DR

An attention mechanism, MatchAttention, that dynamically matches relative positions for cross-view matching and to handle cross-view occlusions, gated cross-MatchAttention and a consistency-constrained loss are proposed.

Abstract

Cross-view matching is fundamentally achieved through cross-attention mechanisms. However, matching of high-resolution images remains challenging due to the quadratic complexity and lack of explicit matching constraints in the existing cross-attention. This paper proposes an attention mechanism, MatchAttention, that dynamically matches relative positions. The relative position determines the attention sampling center of the key-value pairs given a query. Continuous and differentiable sliding-window attention sampling is achieved by the proposed BilinearSoftmax. The relative positions are iteratively updated through residual connections across layers by embedding them into the feature channels. Since the relative position is exactly the learning target for cross-view matching, an efficient hierarchical cross-view decoder, MatchDecoder, is designed with MatchAttention as its core component. To handle cross-view occlusions, gated cross-MatchAttention and a consistency-constrained loss are proposed. These two components collectively mitigate the impact of occlusions in both forward and backward passes, allowing the model to focus more on learning matching relationships. When applied to stereo matching, MatchStereo-B ranked 1st in average error on the public Middlebury benchmark and requires only 29ms for KITTI-resolution inference. MatchStereo-T can process 4K UHD images in 0.1 seconds using only 3GB of GPU memory. The proposed models also achieve state-of-the-art performance on KITTI 2012, KITTI 2015, ETH3D, and Spring flow datasets. The combination of high accuracy and low computational complexity makes real-time, high-resolution, and high-accuracy cross-view matching possible. Project page: https://github.com/TingmanYan/MatchAttention.

Figures (6)

• Figure 1: MatchAttention block. Left: At the $l$-th layer of the transformer, the input feature tokens $F^l$ and the relative position $R_{pos}^l$ are concatenated in channel dimension and fed as input of MatchAttention after a LayerNorm. $F^l$ and $R_{pos}^l$ are updated by the residual connection which outputs $F^{l+1}$ and $R_{pos}^{l+1}$, where the token features are further processed by a feed-forward network. Center and right: Between layers $l$ and $l+1$, the relative position is updated dynamically. Here we illustrate with an effective $3\times 3$ contiguous sampling window computed with its nearest $4\times 4$ token window at discrete positions.
• Figure 2: MatchAttention mechanism. The input is defined as $\hat{F}^l = F^l || R_{pos}^l$ for self-MatchAttention or $\hat{F}_0^l$ and $\hat{F}_1^l$ concatenated along the batch dimension for cross-MatchAttention. For each query $\mathbf{q}_i=Q[i,:]$, the sampling window is determined by $\mathbf{r}_i=R_{pos}[i,:]$. Subsequently, three main operations are performed within the sampling window: (1) query-key similarity computation, (2) BilinearSoftmax, and (3) value aggregation. Finally, the relative position is updated in tandem with token features via the residual connection.
• Figure 3: BilinearSoftmax consists of bilinear sampling of $\mathbf{p}_k$, scattering the sampled points from the expansion window $\overline{\mathcal{W}}$ into four sub-windows ($\mathcal{W}_{nw}$, $\mathcal{W}_{ne}$, $\mathcal{W}_{sw}$, $\mathcal{W}_{se}$), computing softmax within sub-windows, multiplying bilinear weights, and then gathering to obtain the final attention weights. The subscript $i$ is omitted for simplicity.
• Figure 4: Cross-view matching and feature aggregation. For a given query in the reference view, its matching window in the target view is computed by the relative position. Features within the matching window are aggregated from the target view to the reference view via MatchAttention and added to the input features of the reference view through the residual connection.
• Figure 5: MatchDecoder architecture. Given cross-view inputs $I_0$ and $I_1$, the encoder extracts features at scales $s=1/4, 1/8, 1/16, 1/32$. Initial correlation estimates the initial $R_{pos}$ at $1/32$ scale. $R_{pos}$ is iteratively refined by the MatchAttention block that consists of self-MatchAttention, cross-MatchAttention, and ConvGLU shi2023transnext. The final output $R_{pos}$ is upsampled from $1/4$ scale to full resolution using convex upsample raft.