Parameter-Efficient Neural CDEs via Implicit Function Jacobians

TL;DR

The paper addresses the high parameter count of Neural CDEs by introducing a parameter-efficient, implicit-function NCDE framework. It formulates a truly continuous RNN via an implicit update $\mathbf{h}_t = \text{RNN}(\mathbf{X}_t, \mathbf{h}_t)$ and derives its dynamics as $\dot{\mathbf{h}}_t = (I - J_h)^{-1} J_x \dot{\mathbf{X}}_t$, then replaces the expensive inverse with a Taylor expansion to obtain a practical approximation. On CharacterTrajectories, the proposed Jacobian-based NCDE achieves comparable accuracy to the matrix-based NCDE while reducing parameter count by roughly half. This provides a principled, lighter-weight continuous-RNN alternative for sequence modeling with Neural CDEs, reducing computational and memory demands.

Abstract

Neural Controlled Differential Equations (Neural CDEs, NCDEs) are a unique branch of methods, specifically tailored for analysing temporal sequences. However, they come with drawbacks, the main one being the number of parameters, required for the method's operation. In this paper, we propose an alternative, parameter-efficient look at Neural CDEs. It requires much fewer parameters, while also presenting a very logical analogy as the "Continuous RNN", which the Neural CDEs aspire to.