MMS-VPR: Multimodal Street-Level Visual Place Recognition Dataset and Benchmark

TL;DR

This work tackles the gap in Visual Place Recognition by offering MMS-VPR, a pedestrian-scale, multimodal street-level dataset collected in a non-Western urban district. It introduces MMS-VPRlib, a framework and library that fuses image, video, and text representations via frozen backbones and a graph-based cross-modal model to exploit urban topology. Extensive experiments show that multimodal and structure-aware approaches significantly outperform unimodal baselines, demonstrating robust VPR in dense, dynamic street environments. By providing smartphone-friendly collection protocols and open-source tooling, MMS-VPR lowers the barrier to building diverse, scalable geospatial AI benchmarks for urban analytics and navigation research.

Abstract

Existing visual place recognition (VPR) datasets predominantly rely on vehicle-mounted imagery, lack multimodal diversity and underrepresent dense, mixed-use street-level spaces, especially in non-Western urban contexts. To address these gaps, we introduce MMS-VPR, a large-scale multimodal dataset for street-level place recognition in complex, pedestrian-only environments. The dataset comprises 78,575 annotated images and 2,512 video clips captured across 207 locations in a ~70,800 $\mathrm{m}^2$ open-air commercial district in Chengdu, China. Each image is labeled with precise GPS coordinates, timestamp, and textual metadata, and covers varied lighting conditions, viewpoints, and timeframes. MMS-VPR follows a systematic and replicable data collection protocol with minimal device requirements, lowering the barrier for scalable dataset creation. Importantly, the dataset forms an inherent spatial graph with 125 edges, 81 nodes, and 1 subgraph, enabling structure-aware place recognition. We further define two application-specific subsets -- Dataset_Edges and Dataset_Points -- to support fine-grained and graph-based evaluation tasks. Extensive benchmarks using conventional VPR models, graph neural networks, and multimodal baselines show substantial improvements when leveraging multimodal and structural cues. MMS-VPR facilitates future research at the intersection of computer vision, geospatial understanding, and multimodal reasoning. The dataset is publicly available at https://huggingface.co/datasets/Yiwei-Ou/MMS-VPR.

Figures (20)

• Figure 1: Overview of the MMS-VPR dataset construction.
• Figure 2: Images of the same pedestrian street collected during daytime (left) and nighttime (right) indicate that the lighting environment may dramatically transform the visual characteristics of places.
• Figure 3: Visualization of the place types and overall spatial layout in the proposed dataset.
• Figure 4: Overall structure of MMS-VPRlib. MMS-VPRlib uses a multimodal integration framework that models textual, images, and videos in an end-to-end training manner.
• Figure 5: Model performance under different experimental settings