Knowledge-aware Evolutionary Graph Neural Architecture Search

TL;DR

KEGNAS employs the knowledge base to train a knowledge model and a deep multi-output Gaussian process (DMOGP) in one go, which generates and evaluates transfer architectures in only a few GPU seconds, and shows that KEGNAS swiftly generates top-performance architectures.

Abstract

Graph neural architecture search (GNAS) can customize high-performance graph neural network architectures for specific graph tasks or datasets. However, existing GNAS methods begin searching for architectures from a zero-knowledge state, ignoring the prior knowledge that may improve the search efficiency. The available knowledge base (e.g. NAS-Bench-Graph) contains many rich architectures and their multiple performance metrics, such as the accuracy (#Acc) and number of parameters (#Params). This study proposes exploiting such prior knowledge to accelerate the multi-objective evolutionary search on a new graph dataset, named knowledge-aware evolutionary GNAS (KEGNAS). KEGNAS employs the knowledge base to train a knowledge model and a deep multi-output Gaussian process (DMOGP) in one go, which generates and evaluates transfer architectures in only a few GPU seconds. The knowledge model first establishes a dataset-to-architecture mapping, which can quickly generate candidate transfer architectures for a new dataset. Subsequently, the DMOGP with architecture and dataset encodings is designed to predict multiple performance metrics for candidate transfer architectures on the new dataset. According to the predicted metrics, non-dominated candidate transfer architectures are selected to warm-start the multi-objective evolutionary algorithm for optimizing the #Acc and #Params on a new dataset. Empirical studies on NAS-Bench-Graph and five real-world datasets show that KEGNAS swiftly generates top-performance architectures, achieving 4.27% higher accuracy than advanced evolutionary baselines and 11.54% higher accuracy than advanced differentiable baselines. In addition, ablation studies demonstrate that the use of prior knowledge significantly improves the search performance.

Figures (9)

• Figure 1: An illustrative example of the GCN for NAS-Bench-Graph. The architecture encoding consists of the macro space and the candidate operations. In the macro space encoding, the value of the $i$th position represents the index of the previous vertex of vertex $i$. For example, the value of the 2nd position indicates that the index of the previous vertex of vertex 2 is 1. In the candidate operation encoding, the value of the $i$th position represents the candidate operation for vertex $i$. For example, the value of the 2nd position indicates that the candidate operation for vertex 2 is GCN.
• Figure 2: Overview of KEGNAS, which consists of two phases: training and search. A knowledge model and a DMOGP learn how to generate and evaluate candidate transfer architectures for a new task based on a prior knowledge base from NAS-Bench-Graph, respectively. The selected transfer architectures are employed to warm-start the MOEA.
• Figure 3: Overview of the knowledge model.
• Figure 4: Flowchart of the DMOGP.
• Figure 5: Similarity across different tasks using three metrics.