Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs

TL;DR

This paper tackles fairness in graph neural networks applied to knowledge graphs, addressing the gap that existing benchmarks largely omit large, real-world knowledge graphs. It introduces the FairGraphBase benchmarking framework and constructs three knowledge-graph-derived datasets from YAGO, DBpedia, and Wikidata, each with target labels and sensitive attributes. The study evaluates 96 configurations across multiple GNN backbones and three early-stopping settings, using metrics such as ACC, AUC, F1, and fairness measures $\Delta SP$ and $\Delta Eop$, and finds that knowledge graphs exhibit clearer trade-offs between accuracy and fairness while training dynamics (e.g., early stopping) significantly influence outcomes. The work provides open datasets and reproducible resources to guide future development of fairness-aware GNNs in knowledge-graph applications and informs practitioners about the trade-offs and scalability considerations involved.

Abstract

Graph neural networks (GNNs) are powerful tools for learning from graph-structured data but often produce biased predictions with respect to sensitive attributes. Fairness-aware GNNs have been actively studied for mitigating biased predictions. However, no prior studies have evaluated fairness-aware GNNs on knowledge graphs, which are one of the most important graphs in many applications, such as recommender systems. Therefore, we introduce a benchmarking study on knowledge graphs. We generate new graphs from three knowledge graphs, YAGO, DBpedia, and Wikidata, that are significantly larger than the existing graph datasets used in fairness studies. We benchmark inprocessing and preprocessing methods in different GNN backbones and early stopping conditions. We find several key insights: (i) knowledge graphs show different trends from existing datasets; clearer trade-offs between prediction accuracy and fairness metrics than other graphs in fairness-aware GNNs, (ii) the performance is largely affected by not only fairness-aware GNN methods but also GNN backbones and early stopping conditions, and (iii) preprocessing methods often improve fairness metrics, while inprocessing methods improve prediction accuracy.

Figures (22)

• Figure 1: $\Delta$SP ($\downarrow$) and AUC ($\uparrow$) in synthetic graphs
• Figure 2: $\Delta$SP ($\downarrow$) and AUC ($\uparrow$) in social graphs
• Figure 3: $\Delta$SP ($\downarrow$) and AUC ($\uparrow$) in knowledge graphs
• Figure 4: $\Delta$Eop ($\downarrow$) and ACC ($\uparrow$) in synthetic graphs
• Figure 5: $\Delta$Eop ($\downarrow$) and ACC ($\uparrow$) in social graphs