Relation-First Modeling Paradigm for Causal Representation Learning toward the Development of AGI

TL;DR

This work argues that traditional object-first, i.i.d.-based learning struggles to capture causal dynamics, especially under interventional questions and timing. It introduces the relation-first paradigm and formalizes dynamic causal relations using $X\xrightarrow{\theta}Y$ with timing-augmented variables $\mathcal{X}=\langle X,t\rangle$ and $\mathcal{Y}=\langle Y,\tau\rangle$, including definitions and theorems (e.g., EI, dynamic timing, and sequential causality). As a practical instantiation, the paper proposes Relation-Indexed Representation Learning (RIRL), featuring a micro-causal architecture with invertible autoencoders, stacking of relation-indexed representations, and a latent-space exploration algorithm to uncover DAG-like causal routines. Empirical demonstrations on synthetic hydrology data show that RIRL can reconstruct high-dimensional dynamics, disentangle hierarchical causal components, and discover underlying DAG structures, albeit with data requirements and multi-timeline challenges acknowledged. Collectively, the work outlines a forward-looking framework for causality in AI that emphasizes relational information, dynamic timing, and reusable latent indices, with potential implications for developing AGI systems capable of forward-looking, dynamic reasoning.

Abstract

The traditional i.i.d.-based learning paradigm faces inherent challenges in addressing causal relationships, which has become increasingly evident with the rise of applications in causal representation learning. Our understanding of causality naturally requires a perspective as the creator rather than observer, as the ``what...if'' questions only hold within the possible world we conceive. The traditional perspective limits capturing dynamic causal outcomes and leads to compensatory efforts such as the reliance on hidden confounders. This paper lays the groundwork for the new perspective, which enables the \emph{relation-first} modeling paradigm for causality. Also, it introduces the Relation-Indexed Representation Learning (RIRL) as a practical implementation, supported by experiments that validate its efficacy.

Figures (18)

• Figure 1: Causal Emergence $EI(\phi)>0$ stems from overlooking the potential existence of $do(Y)$.
• Figure 2: The relation-first symbolic definition of causal relationship versus mere correlation.
• Figure 3: Illustrative examples for dynamical dependence and independence. The observational dependence from $\mathcal{Y}_1$ to $\mathcal{Y}_2$ is displayed as $\overrightarrow{y_1 y_2}$, where red and blue indicate two different data instances.
• Figure 4: The $do(Y)$-Paradox in traditional Causality Modeling vs. modern Representation Learning.
• Figure 5: Accessing AGI as a black-box, with human-mediated parts colored in blue. A practically usable system demands long-term representation accumulations and refinements, which mirrors our learning process.