Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analytic DAG Constraints for Differentiable DAG Learning

Zhen Zhang, Ignavier Ng, Dong Gong, Yuhang Liu, Mingming Gong, Biwei Huang, Kun Zhang, Anton van den Hengel, Javen Qinfeng Shi

TL;DR

This work rethinks differentiable DAG learning by casting DAG constraints as trace conditions of analytic matrix functions. By defining a flexible function class $\mathcal{F}$ and proving closure under differentiation, addition, and multiplication, it unifies prior constraints and enables the construction of higher-order, gradient-rich constraints with finite convergence radii. A path-following optimization framework with Hadamard-mapped adjacency matrices and $\ell_1$ regularization is proposed to mitigate gradient vanishing and numerical instability, with efficient gradient computation through caching and logarithmic-time updates. Across linear and nonlinear experiments, higher-order analytic DAG constraints consistently reduce gradient vanishing and improve structure recovery compared to state-of-the-art baselines, demonstrating robust DAG discovery under known and unknown data scales. The approach also provides theoretical insights into non-convexity via Hessian spectral radius, informing the trade-offs between gradient strength and optimization difficulty.

Abstract

Recovering the underlying Directed Acyclic Graph (DAG) structures from observational data presents a formidable challenge, partly due to the combinatorial nature of the DAG-constrained optimization problem. Recently, researchers have identified gradient vanishing as one of the primary obstacles in differentiable DAG learning and have proposed several DAG constraints to mitigate this issue. By developing the necessary theory to establish a connection between analytic functions and DAG constraints, we demonstrate that analytic functions from the set $\{f(x) = c_0 + \sum_{i=1}^{\infty}c_ix^i | \forall i > 0, c_i > 0; r = \lim_{i\rightarrow \infty}c_{i}/c_{i+1} > 0\}$ can be employed to formulate effective DAG constraints. Furthermore, we establish that this set of functions is closed under several functional operators, including differentiation, summation, and multiplication. Consequently, these operators can be leveraged to create novel DAG constraints based on existing ones. Using these properties, we design a series of DAG constraints and develop an efficient algorithm to evaluate them. Experiments in various settings demonstrate that our DAG constraints outperform previous state-of-the-art comparators. Our implementation is available at https://github.com/zzhang1987/AnalyticDAGLearning.

Analytic DAG Constraints for Differentiable DAG Learning

TL;DR

This work rethinks differentiable DAG learning by casting DAG constraints as trace conditions of analytic matrix functions. By defining a flexible function class and proving closure under differentiation, addition, and multiplication, it unifies prior constraints and enables the construction of higher-order, gradient-rich constraints with finite convergence radii. A path-following optimization framework with Hadamard-mapped adjacency matrices and regularization is proposed to mitigate gradient vanishing and numerical instability, with efficient gradient computation through caching and logarithmic-time updates. Across linear and nonlinear experiments, higher-order analytic DAG constraints consistently reduce gradient vanishing and improve structure recovery compared to state-of-the-art baselines, demonstrating robust DAG discovery under known and unknown data scales. The approach also provides theoretical insights into non-convexity via Hessian spectral radius, informing the trade-offs between gradient strength and optimization difficulty.

Abstract

Recovering the underlying Directed Acyclic Graph (DAG) structures from observational data presents a formidable challenge, partly due to the combinatorial nature of the DAG-constrained optimization problem. Recently, researchers have identified gradient vanishing as one of the primary obstacles in differentiable DAG learning and have proposed several DAG constraints to mitigate this issue. By developing the necessary theory to establish a connection between analytic functions and DAG constraints, we demonstrate that analytic functions from the set can be employed to formulate effective DAG constraints. Furthermore, we establish that this set of functions is closed under several functional operators, including differentiation, summation, and multiplication. Consequently, these operators can be leveraged to create novel DAG constraints based on existing ones. Using these properties, we design a series of DAG constraints and develop an efficient algorithm to evaluate them. Experiments in various settings demonstrate that our DAG constraints outperform previous state-of-the-art comparators. Our implementation is available at https://github.com/zzhang1987/AnalyticDAGLearning.

Paper Structure

This paper contains 31 sections, 10 theorems, 60 equations, 1 figure, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. DAG and Linear SEM
  4. Continuous DAG learning
  5. Analytic DAG constraints
  6. Analytic functions as DAG constraints
  7. Constructing DAG constraints by functional operator
  8. Gradient Vanishing and Numeric Stability
  9. Efficiently Computation
  10. Overall optimization framework
  11. Non-convexity analysis of analytic DAG constraints
  12. Experiments
  13. Linear SEM with known ground truth scale
  14. Linear SEM with unknown ground truth scale
  15. Experimental results on nonlinear cases
  16. ...and 16 more sections

Key Result

Proposition 1

Let $\tilde{\mathop{\mathrm{\mathbf{B}}}\nolimits}\in \mathbb{R}_{\geq 0}^{d\times d}$ with $\rho(\tilde{\mathop{\mathrm{\mathbf{B}}}\nolimits}) < r$ be the weighted adjacency matrix of a directed graph $\mathop{\mathrm{\mathcal{G}}}\nolimits$, and let $f$ be an analytic function in the form of eq:m

Figures (1)

  • Figure 1: Gradient Norm v.s. Optimization Iterations. Top: Frobenius norm of gradients vs. gradient descent steps for the first two iterations in Algorithm 2. Bottom: Frobenius norm of gradients vs. gradient descent steps for the last three iterations in Algorithm 2. In most cases, our higher-order DAG constraints exhibit larger gradient norms compared to DAGMA, enabling our algorithm to often converge to better solutions than DAGMA.

Theorems & Definitions (19)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Proposition 7
  • Corollary 8
  • Lemma 9
  • proof
  • ...and 9 more