Hybrid guided variational autoencoder for visual place recognition

TL;DR

Visual place recognition in indoor, GPS-denied environments is challenged by memory constraints and robustness. The authors propose a hardware-friendly hybrid guided VAE with a spiking neural network encoder and an ANN decoder that processes event-based vision, achieving disentangled, location-focused latent representations; a 16-d excitation scheme and guided training enable cross-scene generalization. On the Aachen-indoor-VPR dataset, the model attains classification accuracies around $89\%$ and localization errors below $0.5$ m in $90\%$ of cases for the 16-excitation variant, while maintaining generalization to unseen places. This work demonstrates a low-power, low-latency VPR approach suitable for neuromorphic hardware, advancing reliable indoor navigation and providing datasets and methods for hardware-accelerated VPR research.

Abstract

Autonomous agents such as cars, robots and drones need to precisely localize themselves in diverse environments, including in GPS-denied indoor environments. One approach for precise localization is visual place recognition (VPR), which estimates the place of an image based on previously seen places. State-of-the-art VPR models require high amounts of memory, making them unwieldy for mobile deployment, while more compact models lack robustness and generalization capabilities. This work overcomes these limitations for robotics using a combination of event-based vision sensors and an event-based novel guided variational autoencoder (VAE). The encoder part of our model is based on a spiking neural network model which is compatible with power-efficient low latency neuromorphic hardware. The VAE successfully disentangles the visual features of 16 distinct places in our new indoor VPR dataset with a classification performance comparable to other state-of-the-art approaches while, showing robust performance also under various illumination conditions. When tested with novel visual inputs from unknown scenes, our model can distinguish between these places, which demonstrates a high generalization capability by learning the essential features of location. Our compact and robust guided VAE with generalization capabilities poses a promising model for visual place recognition that can significantly enhance mobile robot navigation in known and unknown indoor environments.

Figures (9)

• Figure 1: Turtlebot4 equipped with optical markers and a pair of event cameras.
• Figure 2: Layout of walls in the office-like environment (black 'cross' bars) and the manoeuvre routine of robot during dataset recording (curve in color-map, counter-clockwise). The color of each cell denotes the coloring of each class in upcoming T-SNE visualization.
• Figure 3: A typical RGB image from normal-illumination dataset (left), the rendered event frame (left 2nd), the event frame after preprocessing (left 3rd), and the RGB image of the same place in darkness (right).
• Figure 4: This VAE is unguided without the two additional classifiers. It is guided when jointly trained with an excitation classifier which takes the first 16 latent variables as input, and an inhibition classifier which takes the remaining as input.
• Figure 5: T-SNE of 64-digit embeddings of ugVAE (left), 16-digit excitation portion (middle) and 48-digit inhibition portion (right) of gVAE16-v.