Is uniform expressivity too restrictive? Towards efficient expressivity of graph neural networks

TL;DR

This paper investigates the expressivity landscape of Graph Neural Networks (GNNs) with respect to GC2 queries, contrasting uniform expressivity (size independent of graph size) with non-uniform regimes. It proves that uniform expressivity is impossible for wide Pfaffian activations such as Sigmoid and Tanh, by constructing GC2 queries that cannot be uniformly expressed; it then shows that a form of almost-uniform expressivity is achievable using step-like activations, yielding network sizes that grow only polylogarithmically with graph degree ($O(d\,\log\Delta)$ or $O(d\,\log\log\Delta)$). The authors provide constructive proofs leveraging step-like convergence to a linear threshold and validate the theory with numerical experiments that illustrate the practical viability of non-uniform expressivity. Overall, the work clarifies the trade-off between uniform guarantees and practical expressivity, offering provable bounds and empirical support for efficient non-uniform GNN expressivity on large graphs.

Abstract

Uniform expressivity guarantees that a Graph Neural Network (GNN) can express a query without the parameters depending on the size of the input graphs. This property is desirable in applications in order to have number of trainable parameters that is independent of the size of the input graphs. Uniform expressivity of the two variable guarded fragment (GC2) of first order logic is a well-celebrated result for Rectified Linear Unit (ReLU) GNNs [Barcelo & al., 2020]. In this article, we prove that uniform expressivity of GC2 queries is not possible for GNNs with a wide class of Pfaffian activation functions (including the sigmoid and tanh), answering a question formulated by [Grohe, 2021]. We also show that despite these limitations, many of those GNNs can still efficiently express GC2 queries in a way that the number of parameters remains logarithmic on the maximal degree of the input graphs. Furthermore, we demonstrate that a log-log dependency on the degree is achievable for a certain choice of activation function. This shows that uniform expressivity can be successfully relaxed by covering large graphs appearing in practical applications. Our experiments illustrates that our theoretical estimates hold in practice.

Figures (4)

• Figure 1: Main logical classes to describe the uniform expressivity of GNNs. If no restriction is made on the activation and aggregation functions, standard message passing GNNs can express slightly more than GC2 queries, but less than GFO+C queries. The sum-aggregation aggregation combine GNNs ($\Sigma$-AC-GNNs) exactly express GC2 queries.
• Figure 2: Tree $T[x, k, m]$ with root $s$
• Figure 3: Separation power of $\mathop{\mathrm{CReLU}}\nolimits$ GNN vs. Pfaffian GNN above as function of the graph size
• Figure 4: Learning queries with Pfaffian GNN vs ReLU GNN. The mean square error per graph on the test set is displayed as a function of the order of the graph.

Theorems & Definitions (53)

• Definition 2.1: Neural network
• Definition 2.2: Graph Neural Network (GNN)
• Definition 2.3: GC2 barcelo2020logicalgrohe2021logic
• Remark 2.1
• Remark 2.2
• Definition 2.4
• Example 2.1: barcelo2020logical
• Definition 2.5
• Proposition 2.1
• Definition 3.1