NetVLAD: CNN architecture for weakly supervised place recognition

TL;DR

NetVLAD introduces a trainable generalized VLAD pooling layer that integrates into CNNs to produce compact, discriminative descriptors for visual place recognition. The method is trained end-to-end using weak supervision derived from Google Street View Time Machine data via a novel triplet ranking loss, enabling robustness to viewpoint and lighting changes. Empirical results on Pittsburgh250k and Tokyo 24/7 show substantial gains over off-the-shelf CNN features and traditional descriptors, achieving state-of-the-art performance for compact representations and strong retrieval results on standard benchmarks. The two core innovations—NetVLAD pooling and weakly supervised end-to-end learning—are broadly applicable to related retrieval tasks beyond place recognition.

Abstract

We tackle the problem of large scale visual place recognition, where the task is to quickly and accurately recognize the location of a given query photograph. We present the following three principal contributions. First, we develop a convolutional neural network (CNN) architecture that is trainable in an end-to-end manner directly for the place recognition task. The main component of this architecture, NetVLAD, is a new generalized VLAD layer, inspired by the "Vector of Locally Aggregated Descriptors" image representation commonly used in image retrieval. The layer is readily pluggable into any CNN architecture and amenable to training via backpropagation. Second, we develop a training procedure, based on a new weakly supervised ranking loss, to learn parameters of the architecture in an end-to-end manner from images depicting the same places over time downloaded from Google Street View Time Machine. Finally, we show that the proposed architecture significantly outperforms non-learnt image representations and off-the-shelf CNN descriptors on two challenging place recognition benchmarks, and improves over current state-of-the-art compact image representations on standard image retrieval benchmarks.

Figures (12)

• Figure 1: Our trained NetVLAD descriptor correctly recognizes the location (b) of the query photograph (a) despite the large amount of clutter (people, cars), changes in viewpoint and completely different illumination (night vs daytime). Please see appendix \ref{['sup:res']} for more examples.
• Figure 2: CNN architecture with the NetVLAD layer. The layer can be implemented using standard CNN layers (convolutions, softmax, L2-normalization) and one easy-to-implement aggregation layer to perform aggregation in equation \ref{['eq:vladlayer']} ("VLAD core"), joined up in a directed acyclic graph. Parameters are shown in brackets.
• Figure 3: Benefits of supervised VLAD. Red and green circles are local descriptors from two different images, assigned to the same cluster (Voronoi cell). Under the VLAD encoding, their contribution to the similarity score between the two images is the scalar product (as final VLAD vectors are L2-normalized) between the corresponding residuals, where a residual vector is computed as the difference between the descriptor and the cluster's anchor point. The anchor point $\hbox{\boldmath $\bf c$}_k$ can be interpreted as the origin of a new coordinate system local to the the specific cluster $k$. In standard VLAD, the anchor is chosen as the cluster centre ($\times$) in order to evenly distribute the residuals across the database. However, in a supervised setting where the two descriptors are known to belong to images which should not match, it is possible to learn a better anchor ($\star$) which causes the scalar product between the new residuals to be small.
• Figure 4: Google Street View Time Machine examples. Each column shows perspective images generated from panoramas from nearby locations, taken at different times. A well designed method can use this source of imagery to learn to be invariant to changes in viewpoint and lighting (a-c), and to moderate occlusions (b). It can also learn to suppress confusing visual information such as clouds (a), vehicles and people (b-c), and to chose to either ignore vegetation or to learn a season-invariant vegetation representation (a-c). More examples are given in appendix \ref{['sup:TM']}.
• Figure 5: Comparison of our methods versus off-the-shelf networks and state-of-the-art. The base CNN architecture is denoted in brackets: (A)lexNet and (V)GG-16. Trained representations (red and magenta for AlexNet and VGG-16) outperform by a large margin off-the-shelf ones (blue, cyan, green for AlexNet, Places205, VGG-16), $f_{VLAD}$ (-o-) works better than $f_{max}$ (-x-), and our $f_{VLAD}$+whitening (-$\ast$-) representation based on VGG-16 sets the state-of-the-art on all datasets. Torii15 only evaluated on Tokyo 24/7 as the method relies on depth data not available in other datasets. Additional results are shown in appendix \ref{['sup:res']}.