RelBench: A Benchmark for Deep Learning on Relational Databases

TL;DR

RelBench addresses the underutilization of relational structure in predictive modeling by introducing a public benchmark and an end-to-end relational deep learning framework ($RDL$). It represents databases as heterogeneous temporal graphs where nodes are entities and edges encode primary-foreign key relationships, enabling joint learning across tables with temporal awareness. The paper provides seven real-world datasets, 30 predictive tasks across three task types, and an open-source implementation with a public leaderboard, demonstrating that $RDL$ can match or surpass a skilled data-scientist baseline while dramatically reducing human effort (averaging a 96% reduction in hours and a 94% reduction in lines of code). This work showcases the practical viability of end-to-end $RDL$ for complex multi-tabular data and sets the stage for extensive future research in graph-based relational learning, multi-task pre-training, and scalable training strategies, potentially transforming predictive modeling over relational databases. $RDL$ leverages a temporal graph with PK-FK edges, a deep tabular encoder, and graph neural networks to exploit relational signals that traditional feature engineering often misses, thereby enabling more powerful and efficient real-world decision-making.

Abstract

We present RelBench, a public benchmark for solving predictive tasks over relational databases with graph neural networks. RelBench provides databases and tasks spanning diverse domains and scales, and is intended to be a foundational infrastructure for future research. We use RelBench to conduct the first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024), which combines graph neural network predictive models with (deep) tabular models that extract initial entity-level representations from raw tables. End-to-end learned RDL models fully exploit the predictive signal encoded in primary-foreign key links, marking a significant shift away from the dominant paradigm of manual feature engineering combined with tabular models. To thoroughly evaluate RDL against this prior gold-standard, we conduct an in-depth user study where an experienced data scientist manually engineers features for each task. In this study, RDL learns better models whilst reducing human work needed by more than an order of magnitude. This demonstrates the power of deep learning for solving predictive tasks over relational databases, opening up many new research opportunities enabled by RelBench.

Figures (18)

• Figure 1: RelBench enables training and evaluation of deep learning models on relational databases. RelBench supports framework agnostic data loading, task specification, standardized data splitting, standardized evaluation metrics, and a leaderboard for tracking progress. RelBench also includes a pilot implementation of the relational deep learning blueprint of fey2023relational.
• Figure 2: Example RelBench schema for rel-trial. RelBench databases have complex relational structure and rich column features.
• Figure 3: RDL vs. Data Scientist. Relational Deep Learning matches or outperforms the data scientist in 11 of 15 tasks. Left shows entity classification AUROC, right shows entity regression, reporting MAE normalized so that the RDL MAE is always 1.
• Figure 4: RDL vs. Data Scientist. Relational Deep Learning reduces the hours of human work required to solve a new task by 96% on average (from 12.3 to 0.5 hours). Left shows node-level classification, right shows node-level regression.
• Figure 5: RDL vs. Data Scientist. Relational Deep Learning reduces the new lines of code needed to solve a new task by 94%. Left shows entity classification, right shows entity regression.