LaVPR: Benchmarking Language and Vision for Place Recognition

TL;DR

LaVPR introduces a large-scale, open benchmark that augments established visual place recognition datasets with dense natural-language descriptions to enable robust multi-modal and cross-modal localization. The study shows that language descriptions act as a stable semantic prior, improving performance of compact visual backbones and enabling efficient deployment, while also establishing a LoRA-based MS-loss cross-modal framework that significantly enhances text-to-vision localization. Through extensive ablations, the work demonstrates the necessity of joint vision-language alignment over sequential re-ranking and highlights the trade-offs between fusion strategies, backbone choices, and language encoder capacities. Overall, LaVPR provides a practical, scalable pathway toward resilient, language-informed localization suitable for real-world applications and resource-constrained devices.

Abstract

Visual Place Recognition (VPR) often fails under extreme environmental changes and perceptual aliasing. Furthermore, standard systems cannot perform "blind" localization from verbal descriptions alone, a capability needed for applications such as emergency response. To address these challenges, we introduce LaVPR, a large-scale benchmark that extends existing VPR datasets with over 650,000 rich natural-language descriptions. Using LaVPR, we investigate two paradigms: Multi-Modal Fusion for enhanced robustness and Cross-Modal Retrieval for language-based localization. Our results show that language descriptions yield consistent gains in visually degraded conditions, with the most significant impact on smaller backbones. Notably, adding language allows compact models to rival the performance of much larger vision-only architectures. For cross-modal retrieval, we establish a baseline using Low-Rank Adaptation (LoRA) and Multi-Similarity loss, which substantially outperforms standard contrastive methods across vision-language models. Ultimately, LaVPR enables a new class of localization systems that are both resilient to real-world stochasticity and practical for resource-constrained deployment. Our dataset and code are available at https://github.com/oferidan1/LaVPR.

Figures (14)

• Figure 1: Example image from the Amstertime-La benchmark. Aligned description in LaVPR: 'Leftmost red brick building with an upper window and a black wrought iron balcony railing, cream-colored framed storefront with a display window, recessed wooden entrance door, narrow brick wall section with house number '3', grey vertical downspout, middle cream-colored framed shop with a glass door, large storefront window displaying "de Witte TandenWinkel", black lower facade panel, red brick upper floor with a rectangular window, blue square sign, grey electrical box, rightmost red brick building with an upper window, yellow scalloped awning labeled "KAAS CHEESE".'
• Figure 2: Qualitative results (Amstertime dataset). From left to right: Query, MixVPR Top-1, La-MixVPR Top-1 (using CAT Fusion). Correct matches are bordered in green; incorrect in red. Additional Results are provided in the Appendix, Section \ref{['subsec:appendix_qualitative_results']}.
• Figure 3: Example pairs from the Amstertime-La dataset. The dataset is curated by identifying shared signage and scene text between query (left) and database (right) images to ensure fine-grained semantic alignment.
• Figure 4: Example images from MSLS-Blur and MSLS-Weather subset, left is original query image, middle is blurred augmentation, right is augmented with rain, snow and fog.
• Figure 5: Qualitative visualization of Step 2: Open-set object detection using SAM3. Given an input image and a set of object queries extracted from the generated description, SAM3 produces segmentation masks for visually identifiable objects.