JIST: Joint Image and Sequence Training for Sequential Visual Place Recognition

TL;DR

This work tackles sequential Visual Place Recognition under limited sequence data by introducing Joint Image and Sequence Training (JIST), a multi-task framework that jointly learns from image-to-image VPR data and sequence data. It introduces SeqGeM, a learnable temporal aggregation layer that produces compact, length-invariant descriptors, enabling fast matching with 512-D descriptors. The approach achieves state-of-the-art performance with smaller descriptors, faster inference, and strong robustness to sequence length and frame ordering, validated on MSLS, Melbourne, SF-XL, and RobotCar datasets. Practically, JIST enables scalable, real-time place recognition suitable for large-scale mapping, with deployment considerations discussed for edge devices like Jetson Nano and potential speedups via approximate nearest neighbor search. The work also outlines limitations and avenues for future improvements, including knowledge distillation and extending the multi-branch framework to additional data sources.

Abstract

Visual Place Recognition aims at recognizing previously visited places by relying on visual clues, and it is used in robotics applications for SLAM and localization. Since typically a mobile robot has access to a continuous stream of frames, this task is naturally cast as a sequence-to-sequence localization problem. Nevertheless, obtaining sequences of labelled data is much more expensive than collecting isolated images, which can be done in an automated way with little supervision. As a mitigation to this problem, we propose a novel Joint Image and Sequence Training protocol (JIST) that leverages large uncurated sets of images through a multi-task learning framework. With JIST we also introduce SeqGeM, an aggregation layer that revisits the popular GeM pooling to produce a single robust and compact embedding from a sequence of single-frame embeddings. We show that our model is able to outperform previous state of the art while being faster, using 8 times smaller descriptors, having a lighter architecture and allowing to process sequences of various lengths. Code is available at https://github.com/ga1i13o/JIST

Figures (5)

• Figure 1: Our multi-task training framework allows to surpass previous SOTA in performance. Thanks to our novel layer SeqGeM, we are able to cut down the matching time by an order of magnitude.
• Figure 2: Overview of the JIST framework. At training time (left) we use two branches, one for sequences and one for single-images. Each branch has a separate loss, while sharing part of their weights. The multi-task training allows to obtain discriminative frame-wise embeddings by exploiting the powerful representations learned by the backbone and fully connected from single images. At test time (right) we only use the sequences branch, and we follow the standard image retrieval pipeline: embeddings are extracted for both database and queries sequences, and then a prediction for database sequence that is most similar to the query is computed through a kNN. Note that in a real-world scenario, the potentially expensive embeddings extraction for database sequences can be performed offline, making the framework fast (more information on efficiency in \ref{['sec:realworld_deployment']}).
• Figure 3: Sketch of our proposed SeqGeM layer. Given D-dimensional feature vectors from L frames, SeqGeM produces a single descriptor/embedding of dimensionality D, which contains information from the whole sequence.
• Figure 4: Precision-Recall curves computed on MSLS test set for the most relevant methods. All models are trained with a ResNet-18 backbone except TimeSformer, which uses a custom backbone.
• Figure 5: The plot shows how different methods react to changes in the dimension of test-time sequence length (i.e. number of frames). All methods are trained with fixed sequence length of 5.