Can Transformers Capture Spatial Relations between Objects?

TL;DR

This work identifies a simple RelatiViT architecture and demonstrates that it outperforms all current approaches on spatial relation prediction in in-the-wild settings, and is the first method to convincingly outperform naive baselines on spatial relation prediction in in-the-wild settings.

Abstract

Spatial relationships between objects represent key scene information for humans to understand and interact with the world. To study the capability of current computer vision systems to recognize physically grounded spatial relations, we start by proposing precise relation definitions that permit consistently annotating a benchmark dataset. Despite the apparent simplicity of this task relative to others in the recognition literature, we observe that existing approaches perform poorly on this benchmark. We propose new approaches exploiting the long-range attention capabilities of transformers for this task, and evaluating key design principles. We identify a simple "RelatiViT" architecture and demonstrate that it outperforms all current approaches. To our knowledge, this is the first method to convincingly outperform naive baselines on spatial relation prediction in in-the-wild settings. The code and datasets are available in \url{https://sites.google.com/view/spatial-relation}.

Figures (8)

• Figure 1: Illustrations of spatial relation prediction (SRP) task on Rel3D and SpatialSense+ datasets.
• Figure 2: Three issues of original SpatialSense: a) mixing the object-centric and viewer-centric annotations; b) ambiguity caused by polysemous words; c) language bias.
• Figure 3: Four different architecture designs for SRP.
• Figure 4: Comparison of attention maps. In each subfigure, the 1st and 2nd rows visualize the attention matrices averaged from the subject and object query embeddings respectively. The 1st column is the original image with the bounding box; the 2nd one is the attention map reflecting how much the query attends to the context image (context aggregation); the 3rd and 4th ones show the attention between two queries (pair-wise interaction).
• Figure 5: The performance on different sizes of the Rel3D training set.