Learning Sequence Descriptor based on Spatio-Temporal Attention for Visual Place Recognition

TL;DR

This paper tackles Visual Place Recognition (VPR) under perceptual aliasing by introducing a spatio-temporal sequence descriptor that fuses intra-frame spatial patterns with inter-frame temporal dynamics. A dual-branch Transformer architecture learns spatial and temporal representations from image patches, using a sliding window and relative positional encoding to capture robust spatio-temporal patterns, with NetVLAD aggregating the final descriptor. The method, trained with a max-margin triplet loss, outperforms state-of-the-art sequence-descriptor baselines across MSLS, NordLand, and Oxford RobotCar, while analyses reveal benefits of temporal interactions and relative positioning. Practically, the approach offers improved robustness to appearance and illumination changes and dynamic objects, albeit with higher online computation and memory demands, suggesting a trade-off for real-time VPR systems. The work provides a solid foundation for integrating temporal dynamics into sequence-based VPR and points to avenues for efficiency and further performance gains.

Abstract

Visual Place Recognition (VPR) aims to retrieve frames from a geotagged database that are located at the same place as the query frame. To improve the robustness of VPR in perceptually aliasing scenarios, sequence-based VPR methods are proposed. These methods are either based on matching between frame sequences or extracting sequence descriptors for direct retrieval. However, the former is usually based on the assumption of constant velocity, which is difficult to hold in practice, and is computationally expensive and subject to sequence length. Although the latter overcomes these problems, existing sequence descriptors are constructed by aggregating features of multiple frames only, without interaction on temporal information, and thus cannot obtain descriptors with spatio-temporal discrimination.In this paper, we propose a sequence descriptor that effectively incorporates spatio-temporal information. Specifically, spatial attention within the same frame is utilized to learn spatial feature patterns, while attention in corresponding local regions of different frames is utilized to learn the persistence or change of features over time. We use a sliding window to control the temporal range of attention and use relative positional encoding to construct sequential relationships between different features. This allows our descriptors to capture the intrinsic dynamics in a sequence of frames.Comprehensive experiments on challenging benchmark datasets show that the proposed approach outperforms recent state-of-the-art methods.The code is available at https://github.com/tiev-tongji/Spatio-Temporal-SeqVPR.

Figures (6)

• Figure 1: The architecture of our proposed method. Given a continuous sequence of raw frames $I_{1}$, $I_{2}$, $\ldots$, $I_{L}$, we employ a Convolutional Neural Network (CNN) to map each frame to feature maps and then split these maps into patches. A Linear Projection is subsequently employed to map the patch features to embeddings $\{x^{1}_{1,\ldots,N},x^{2}_{1,\ldots,N},\ldots,x^{L}_{1,\ldots,N}\}$. These embeddings from individual frames are then passed through Spatial Transformer Encoders, applying self-attention for spatial information interaction. This process yields a set of transformed embeddings $\{x^{s1}_{1,\ldots,N},\ldots,x^{sL}_{1,\ldots,N}\}$. Furthermore, the embeddings across different frames within the sliding window are input into a Temporal Transformer Encoder to fuse temporal information, which generates $\{x^{t1}_{1,\ldots,N},\ldots,x^{tL}_{1,\ldots,N}\}$. Finally, the embeddings from two branches are combined, and the NetVLAD layer is employed to aggregate these embeddings to generate a sequence descriptor.
• Figure 2: The different positional embeddings and sliding windows. Here we show spatial absolute positional embedding (a) and relative positional embedding (c), as well as temporal absolute positional embedding (b) and relative positional embedding (d), where we use $A$ to represent the absolute positional embedding and $R$ for the relative positional embedding. Dashed arrows indicate that information is passed between the two patches. Solid arrows indicate to fuse positional embeddings to the information passing. It's important to note that absolute positional embeddings are independent of the inter-patch relationships. In contrast, relative positional embeddings vary based on the position relationship between patches.
• Figure 3: Visualizations on attention. Here are the attentions of our method and SeqVLAD for different regions of the query sequences which is in Figure \ref{['qualitative']} (a), Figure \ref{['qualitative']} (c). Red portions indicate more focus, and blue portions indicate less focus. Compared to SeqVLAD, our method focuses less on dynamic objects and more on road elements.
• Figure 4: Ablation Studies for spatio-temporal effectiveness and positional embedding. We show the comparison of Recall@N performances with only spatial or temporal module, and two kinds of positional embedding or w/o position information. In addition, the positional embedding is trained from scratch without pre-trained parameters.
• Figure 5: Ablation Studies for sliding window settings. We show the comparison of Recall@N performances with different sliding window settings, where m/s demonstrates the size and stride of the sliding window respectively.