Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rademacher Complexity of Neural ODEs via Chen-Fliess Series

Joshua Hanson, Maxim Raginsky

TL;DR

This work addresses assessing the generalization of neural ODEs by representing the small-time input-to-output map through the Chen--Fliess series, which expresses the output as a linear combination of iterated Lie derivatives weighted by iterated integrals of the control input. By viewing the Chen--Fliess expansion as a single-layer, infinite-width network with signature-based weights, the authors derive a data-dependent Rademacher complexity bound for the corresponding function class, $\mathcal{F}=\{ x \mapsto \langle S(u), \Phi(x) \rangle : u\in\mathcal{U} \}$, in terms of the growth of iterated Lie derivatives $\Lambda_k$. The paper provides concrete instantiations of the bound for bilinear, analytic, and Hopfield-like systems, illustrating how the bound scales with system properties and horizon parameters. While the Chen--Fliess series imposes convergence constraints, the results offer a principled, tractable pathway to quantify generalization for neural ODEs and suggest future directions such as time-slicing and margin-based analyses to broaden applicability.

Abstract

We show how continuous-depth neural ODE models can be framed as single-layer, infinite-width nets using the Chen--Fliess series expansion for nonlinear ODEs. In this net, the output ``weights'' are taken from the signature of the control input -- a tool used to represent infinite-dimensional paths as a sequence of tensors -- which comprises iterated integrals of the control input over a simplex. The ``features'' are taken to be iterated Lie derivatives of the output function with respect to the vector fields in the controlled ODE model. The main result of this work applies this framework to derive compact expressions for the Rademacher complexity of ODE models that map an initial condition to a scalar output at some terminal time. The result leverages the straightforward analysis afforded by single-layer architectures. We conclude with some examples instantiating the bound for some specific systems and discuss potential follow-up work.

Rademacher Complexity of Neural ODEs via Chen-Fliess Series

TL;DR

This work addresses assessing the generalization of neural ODEs by representing the small-time input-to-output map through the Chen--Fliess series, which expresses the output as a linear combination of iterated Lie derivatives weighted by iterated integrals of the control input. By viewing the Chen--Fliess expansion as a single-layer, infinite-width network with signature-based weights, the authors derive a data-dependent Rademacher complexity bound for the corresponding function class, , in terms of the growth of iterated Lie derivatives . The paper provides concrete instantiations of the bound for bilinear, analytic, and Hopfield-like systems, illustrating how the bound scales with system properties and horizon parameters. While the Chen--Fliess series imposes convergence constraints, the results offer a principled, tractable pathway to quantify generalization for neural ODEs and suggest future directions such as time-slicing and margin-based analyses to broaden applicability.

Abstract

We show how continuous-depth neural ODE models can be framed as single-layer, infinite-width nets using the Chen--Fliess series expansion for nonlinear ODEs. In this net, the output ``weights'' are taken from the signature of the control input -- a tool used to represent infinite-dimensional paths as a sequence of tensors -- which comprises iterated integrals of the control input over a simplex. The ``features'' are taken to be iterated Lie derivatives of the output function with respect to the vector fields in the controlled ODE model. The main result of this work applies this framework to derive compact expressions for the Rademacher complexity of ODE models that map an initial condition to a scalar output at some terminal time. The result leverages the straightforward analysis afforded by single-layer architectures. We conclude with some examples instantiating the bound for some specific systems and discuss potential follow-up work.
Paper Structure (13 sections, 2 theorems, 42 equations)

This paper contains 13 sections, 2 theorems, 42 equations.

Table of Contents

  1. Introduction
  2. Chen--Fliess series
  3. Control-affine systems
  4. Chen--Fliess series
  5. Sequence-space embeddings
  6. Main result
  7. Proof:
  8. Proof:
  9. Examples
  10. Example 1
  11. Example 2
  12. Example 3
  13. Conclusion

Key Result

Theorem 1

The empirical Rademacher complexity of $\mathcal{F}$ is bounded by where $\Lambda_k := \sup \left\{ \left| L_w c^{\mathsf T} x \right| : x \in \mathsf{X},\ w \in \mathsf{W},\ |w| = k\right\}$.

Theorems & Definitions (2)

  • Theorem 1
  • Lemma 2