Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Backpropagation from KL Projections: Differential and Exact I-Projection Correspondences

Manish Krishan Lal

Abstract

We establish two correspondences between reverse-mode automatic differentiation (backpropagation at a given forward-pass point) and compositions of projection maps in Kullback--Leibler (KL) geometry. In both settings, message passing enforces agreement and factorization constraints through KL projections. In the first setting, backpropagation arises as the differential of a KL projection map on a lifted deterministic computation graph. In the second setting, on complete and decomposable sum--product networks, the same reverse-mode quantities coincide with exact probabilistic marginals and are realized by a KL I-projection. The distinction is that, in the first setting, projection induces structure, whereas, in the second, structure makes the projection exact. This pedagogical note highlights the relation among backpropagation, belief propagation, and KL projection algorithms and provides a perspective that unifies learning, sampling, and inference under a common geometric operator.

Backpropagation from KL Projections: Differential and Exact I-Projection Correspondences

Abstract

We establish two correspondences between reverse-mode automatic differentiation (backpropagation at a given forward-pass point) and compositions of projection maps in Kullback--Leibler (KL) geometry. In both settings, message passing enforces agreement and factorization constraints through KL projections. In the first setting, backpropagation arises as the differential of a KL projection map on a lifted deterministic computation graph. In the second setting, on complete and decomposable sum--product networks, the same reverse-mode quantities coincide with exact probabilistic marginals and are realized by a KL I-projection. The distinction is that, in the first setting, projection induces structure, whereas, in the second, structure makes the projection exact. This pedagogical note highlights the relation among backpropagation, belief propagation, and KL projection algorithms and provides a perspective that unifies learning, sampling, and inference under a common geometric operator.
Paper Structure (31 sections, 10 theorems, 63 equations)

This paper contains 31 sections, 10 theorems, 63 equations.

Table of Contents

  1. Introduction
  2. Geometric setting
  3. Two regimes
  4. Relation to prior work
  5. Preliminaries and notation
  6. Probability simplices and relative interiors.
  7. Bregman divergences and KL geometry.
  8. Left (I-) and right (reverse-KL) projections.
  9. Product families.
  10. Factor graphs.
  11. Messages and normalized BP.
  12. Beliefs and gauge invariance.
  13. Consensus sets.
  14. Regime (A): Delta-lifted backpropagation as the differential of a KL projection algorithm
  15. Projection algorithm: BP as a hybrid of KL projections
  16. ...and 16 more sections

Key Result

Proposition 3.1

For a finite factor graph $G$ with strictly positive factors and initialization $q_{-1}$ as above, the iterates of the above algorithm coincide with normalized sum--product (BP) on $G$. In particular:

Theorems & Definitions (18)

  • Proposition 3.1: Normalized BP as a hybrid of KL projections walsh2010bp
  • Lemma 3.2: Delta-factor chain rule
  • proof
  • Theorem 3.3: Backpropagation as log-derivatives of downward BP messages
  • proof
  • Corollary 3.4: Layered backpropagation as the differential of local KL projection steps
  • Theorem 4.1: Log-derivatives equal marginals
  • Lemma 4.2: Tree case
  • proof
  • Lemma 4.3: Unfolding preserves upward values
  • ...and 8 more