Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Networks

Shreyas Chaudhari, Srinivasa Pranav, José M. F. Moura

TL;DR

This work introduces Gradient Networks (GradNets) for directly parameterizing and learning gradients of functions, along with Monotone Gradient Networks (mGradNets) that correspond to gradients of convex functions. It provides universal approximation proofs for gradients of broad function classes, including sums of ridge functions and transformed ridge compositions, and presents architecture variants (GradNet-C/M and mGradNet-C/M) with provable symmetry or PSD properties of Jacobians. The authors empirically validate these networks on gradient-field estimation and Hamiltonian dynamics tasks, achieving substantial gains over existing methods in high dimensions and nonconvex settings. The results establish GradNets as a flexible, theoretically grounded framework for gradient-based learning with practical impact in inverse problems, optimal transport, and physics-informed modeling.

Abstract

Directly parameterizing and learning gradients of functions has widespread significance, with specific applications in inverse problems, generative modeling, and optimal transport. This paper introduces gradient networks (GradNets): novel neural network architectures that parameterize gradients of various function classes. GradNets exhibit specialized architectural constraints that ensure correspondence to gradient functions. We provide a comprehensive GradNet design framework that includes methods for transforming GradNets into monotone gradient networks (mGradNets), which are guaranteed to represent gradients of convex functions. Our results establish that our proposed GradNet (and mGradNet) universally approximate the gradients of (convex) functions. Furthermore, these networks can be customized to correspond to specific spaces of potential functions, including transformed sums of (convex) ridge functions. Our analysis leads to two distinct GradNet architectures, GradNet-C and GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and mGradNet-M. Our empirical results demonstrate that these architectures provide efficient parameterizations and outperform existing methods by up to 15 dB in gradient field tasks and by up to 11 dB in Hamiltonian dynamics learning tasks.

Gradient Networks

TL;DR

This work introduces Gradient Networks (GradNets) for directly parameterizing and learning gradients of functions, along with Monotone Gradient Networks (mGradNets) that correspond to gradients of convex functions. It provides universal approximation proofs for gradients of broad function classes, including sums of ridge functions and transformed ridge compositions, and presents architecture variants (GradNet-C/M and mGradNet-C/M) with provable symmetry or PSD properties of Jacobians. The authors empirically validate these networks on gradient-field estimation and Hamiltonian dynamics tasks, achieving substantial gains over existing methods in high dimensions and nonconvex settings. The results establish GradNets as a flexible, theoretically grounded framework for gradient-based learning with practical impact in inverse problems, optimal transport, and physics-informed modeling.

Abstract

Directly parameterizing and learning gradients of functions has widespread significance, with specific applications in inverse problems, generative modeling, and optimal transport. This paper introduces gradient networks (GradNets): novel neural network architectures that parameterize gradients of various function classes. GradNets exhibit specialized architectural constraints that ensure correspondence to gradient functions. We provide a comprehensive GradNet design framework that includes methods for transforming GradNets into monotone gradient networks (mGradNets), which are guaranteed to represent gradients of convex functions. Our results establish that our proposed GradNet (and mGradNet) universally approximate the gradients of (convex) functions. Furthermore, these networks can be customized to correspond to specific spaces of potential functions, including transformed sums of (convex) ridge functions. Our analysis leads to two distinct GradNet architectures, GradNet-C and GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and mGradNet-M. Our empirical results demonstrate that these architectures provide efficient parameterizations and outperform existing methods by up to 15 dB in gradient field tasks and by up to 11 dB in Hamiltonian dynamics learning tasks.
Paper Structure (21 sections, 22 theorems, 33 equations, 6 figures, 3 tables)

This paper contains 21 sections, 22 theorems, 33 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Gradient Networks (GradNet)
  4. Monotone Gradient Networks (mGradNet)
  5. Universal Approximation Results
  6. (Monotone) Gradient Functions
  7. Gradients of Sums of (Convex) Ridge Functions
  8. Gradients of (convex monotone) transformations of sums of (convex) ridges
  9. Learning Subgradients of Convex Functions
  10. GradNet Architecture Variants
  11. Modular (Monotone) Gradient Networks (GradNet-M, mGradNet-M)
  12. Cascaded (Monotone) Gradient Networks (GradNet-C, mGradNet-C)
  13. Experiments
  14. Gradient Field
  15. 2D Gradient Field
  16. ...and 6 more sections

Key Result

Lemma 1

A differentiable function $f : \mathbb{R}^d\to \mathbb{R}^d$ has a scalar-valued antiderivative if and only if its Jacobian is symmetric everywhere, i.e., $\forall \boldsymbol{x} \in \mathbb{R}^d\;\boldsymbol{J}_{f}(\boldsymbol{x}) = \boldsymbol{J}_{f}(\boldsymbol{x})^\top$.

Figures (6)

  • Figure 1: Relationships between relevant function classes considered in corresponding theory sections of this paper: A) all functions; B) monotone functions; Sec. \ref{['sec:monotone_gradients']}: C) gradients, D) monotone gradients; Sec. \ref{['sec:transformations']}: E) gradients of transformed sums of ridges, F) gradients of transformed sums of convex ridges; Sec. \ref{['sec:ridges']}: G) gradients of sums of ridges, H) gradients of sums of convex ridges; Sec. \ref{['sec:subgradients']}: respective subgradients.
  • Figure 2: Single module of the modular gradient network (GradNet-M) defined in \ref{['eqn:m-mgn_a']}-\ref{['eqn:m-mgn_b']}.
  • Figure 3: Cascaded gradient network (GradNet-C), defined in \ref{['eqn:cmgn_start']}-\ref{['eqn:cmgn_end']}, with $\boldsymbol{A}_\ell = \mathrm{diag}\left({\boldsymbol{\alpha}_\ell}\right),\boldsymbol{B}_\ell = \mathrm{diag}\left({\boldsymbol{\beta}_\ell}\right)$.
  • Figure 4: Test functions $F$ in \ref{['eq:2d_convex_test_fn']} and $G$ in \ref{['eq:2d_nonconvex_test_fn']}, along with their gradients, for $d=2$ over the unit square.
  • Figure 5: Gradient field learning results for $d=2$
  • ...and 1 more figures

Theorems & Definitions (52)

  • Definition 1
  • Lemma 1: Antiderivatives and Symmetric Jacobians
  • Proposition 1
  • proof
  • Remark 1
  • Remark 2
  • Definition 2
  • Definition 3: Monotonicity
  • Remark 3
  • Proposition 2
  • ...and 42 more