A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective

TL;DR

This survey comprehensively analyzes how graph neural networks and graph transformers are leveraged across computer vision tasks, organized by data modality and task. It details foundational GNN variants (recurrent, spectral/spatial convolution, and new techniques) and graph transformers, then surveys applications in 2D images, videos, 3D data, vision-language tasks, and medical imaging. Key contributions include a task-oriented taxonomy, in-depth coverage of representative methods, and insights into limitations and future directions such as automatic graph construction, cross-modal reasoning, and graph-based knowledge integration. The work highlights the flexibility of relational inductive biases to capture complex dependencies in visual data and informs practitioners on where GNNs and graph transformers offer tangible advantages over traditional CNN/Transformer approaches.

Abstract

Graph Neural Networks (GNNs) have gained momentum in graph representation learning and boosted the state of the art in a variety of areas, such as data mining (\emph{e.g.,} social network analysis and recommender systems), computer vision (\emph{e.g.,} object detection and point cloud learning), and natural language processing (\emph{e.g.,} relation extraction and sequence learning), to name a few. With the emergence of Transformers in natural language processing and computer vision, graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation while avoiding strict structural inductive biases. In this paper, we present a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective. Specifically, we divide their applications in computer vision into five categories according to the modality of input data, \emph{i.e.,} 2D natural images, videos, 3D data, vision + language, and medical images. In each category, we further divide the applications according to a set of vision tasks. Such a task-oriented taxonomy allows us to examine how each task is tackled by different GNN-based approaches and how well these approaches perform. Based on the necessary preliminaries, we provide the definitions and challenges of the tasks, in-depth coverage of the representative approaches, as well as discussions regarding insights, limitations, and future directions.

Figures (12)

• Figure 1: Overview of the landscape of GNNs and graph Transformers in computer vision.
• Figure 2: Two types of graph convolutional operations.
• Figure 3: ML-GCN (Figure used courtesy of chen2019multi).
• Figure 4: SGRN (Figure used courtesy of xu2019spatial).
• Figure 5: Graph R-CNN (Figure used courtesy of yang2018graph).