Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective

TL;DR

The paper addresses how foundation models acquire relational world knowledge by proposing a hypergraph recovery framework in which the world is a weighted hypergraph and pre-training data are samples from hyperedges under a perception mapping. It establishes population-level identifiability, a minimax data-efficiency theory, and near-optimal sample complexity for Masked Modeling, while extending the framework to multimodal entity alignment. Theoretical results are complemented by synthetic and real-world experiments that show learned relational structures align with ground-truth relations and that larger, more capable models yield stronger relational recovery. This work provides a rigorous mathematical foundation linking PTMs with hypergraph theory to analyze and improve relational learning and multimodal alignment.

Abstract

Foundation Models (FMs) have demonstrated remarkable insights into the relational dynamics of the world, leading to the crucial question: how do these models acquire an understanding of world hybrid relations? Traditional statistical learning, particularly for prediction problems, may overlook the rich and inherently structured information from the data, especially regarding the relationships between objects. We introduce a mathematical model that formalizes relational learning as hypergraph recovery to study pre-training of FMs. In our framework, the world is represented as a hypergraph, with data abstracted as random samples from hyperedges. We theoretically examine the feasibility of a Pre-Trained Model (PTM) to recover this hypergraph and analyze the data efficiency in a minimax near-optimal style. By integrating rich graph theories into the realm of PTMs, our mathematical framework offers powerful tools for an in-depth understanding of pre-training from a unique perspective and can be used under various scenarios. As an example, we extend the framework to entity alignment in multimodal learning.

Figures (34)

• Figure 1: Our hypergraph recovery framework for relational learning in PTMs. The relational model of the world is viewed as a hypergraph. Data are generated by sampling hyperedges from the world relational model and mapping them to perception domains. PTMs learn the entity relations from the data. Recovered relational hypergraphs can be evaluated from the PTMs.
• Figure 2: Extension of our hypergraph framework to entity alignment in multimodal learning (taking vision and language for illustration). The relational hypergraphs in different modalities can be reconstructed from data. The entities from different modalities can be aligned by matching the relational hypergraphs. "Rec." represents "Reconstruct".
• Figure 3: Evaluation results of synthetic relational learning. (a) STAR graphs with different numbers of edges ($m=n-1$). (b) STAR graphs with different range ratios. (c) Graphs with different MM path lengths. For each, the experiments are repeated for $5$ times and the evaluation results are averaged over the $5$ trials.
• Figure 4: Evaluation results of different LLMs for the real-world relational subgraph generated from the source word "table". We use different letters to represent different entities (see \ref{['subappendix:rwre']} for their correspondences). The graphs (from left to right) are the ground truth (extracted from ConceptNet), evaluation results of LLAMA-2-70B, GPT-3.5, and GPT-4, respectively.
• Figure 5: Frucht graph.

Theorems & Definitions (13)

• Definition 4.3: Relational Learning
• Definition 4.4: $\epsilon$-Approximate Relational Learning
• Definition 4.5: ($\epsilon$-Approximate) Relational Learning of Models
• Theorem 5.1: Identifiability
• Theorem 5.2: Information Theoretical Lower Bound
• Theorem 5.6: Upper Bound by MM
• Proposition 6.1
• Lemma 1.1
• proof : Proof of Lemma \ref{['lm:rr']}
• Lemma 1.2