Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Signature Methods in Machine Learning

Terry Lyons, Andrew D. McLeod

TL;DR

This survey consolidates signature-based methods for machine learning on streaming data, emphasizing path signatures as invariant summaries of multi-channel sequences. It covers the mathematical foundations (tensor algebras, Chen’s identity, and rough paths), practical computation (Signature Kernel, Expected Signature, and Log-ODE), and neural extensions (Neural CDEs/RDEs), tying theory to diverse applications. It highlights augmentation strategies (Lead-Lag, Time, Invisibility Reset) and even randomised signatures to manage high-dimensional, irregular data while preserving order information. The work demonstrates broad impact across health, speech, action recognition, and anomaly detection, underscoring the signature’s universality as a feature map and its capacity to unify nonlinear modeling with linear learning tools. Overall, signatures offer a principled, scalable framework for extracting meaningful dynamics from complex time-series, with ready-to-use software for computation and several practical success stories.

Abstract

Signature-based techniques give mathematical insight into the interactions between complex streams of evolving data. These insights can be quite naturally translated into numerical approaches to understanding streamed data, and perhaps because of their mathematical precision, have proved useful in analysing streamed data in situations where the data is irregular, and not stationary, and the dimension of the data and the sample sizes are both moderate. Understanding streamed multi-modal data is exponential: a word in $n$ letters from an alphabet of size $d$ can be any one of $d^n$ messages. Signatures remove the exponential amount of noise that arises from sampling irregularity, but an exponential amount of information still remain. This survey aims to stay in the domain where that exponential scaling can be managed directly. Scalability issues are an important challenge in many problems but would require another survey article and further ideas. This survey describes a range of contexts where the data sets are small enough to remove the possibility of massive machine learning, and the existence of small sets of context free and principled features can be used effectively. The mathematical nature of the tools can make their use intimidating to non-mathematicians. The examples presented in this article are intended to bridge this communication gap and provide tractable working examples drawn from the machine learning context. Notebooks are available online for several of these examples. This survey builds on the earlier paper of Ilya Chevryev and Andrey Kormilitzin which had broadly similar aims at an earlier point in the development of this machinery. This article illustrates how the theoretical insights offered by signatures are simply realised in the analysis of application data in a way that is largely agnostic to the data type.

Signature Methods in Machine Learning

TL;DR

This survey consolidates signature-based methods for machine learning on streaming data, emphasizing path signatures as invariant summaries of multi-channel sequences. It covers the mathematical foundations (tensor algebras, Chen’s identity, and rough paths), practical computation (Signature Kernel, Expected Signature, and Log-ODE), and neural extensions (Neural CDEs/RDEs), tying theory to diverse applications. It highlights augmentation strategies (Lead-Lag, Time, Invisibility Reset) and even randomised signatures to manage high-dimensional, irregular data while preserving order information. The work demonstrates broad impact across health, speech, action recognition, and anomaly detection, underscoring the signature’s universality as a feature map and its capacity to unify nonlinear modeling with linear learning tools. Overall, signatures offer a principled, scalable framework for extracting meaningful dynamics from complex time-series, with ready-to-use software for computation and several practical success stories.

Abstract

Signature-based techniques give mathematical insight into the interactions between complex streams of evolving data. These insights can be quite naturally translated into numerical approaches to understanding streamed data, and perhaps because of their mathematical precision, have proved useful in analysing streamed data in situations where the data is irregular, and not stationary, and the dimension of the data and the sample sizes are both moderate. Understanding streamed multi-modal data is exponential: a word in letters from an alphabet of size can be any one of messages. Signatures remove the exponential amount of noise that arises from sampling irregularity, but an exponential amount of information still remain. This survey aims to stay in the domain where that exponential scaling can be managed directly. Scalability issues are an important challenge in many problems but would require another survey article and further ideas. This survey describes a range of contexts where the data sets are small enough to remove the possibility of massive machine learning, and the existence of small sets of context free and principled features can be used effectively. The mathematical nature of the tools can make their use intimidating to non-mathematicians. The examples presented in this article are intended to bridge this communication gap and provide tractable working examples drawn from the machine learning context. Notebooks are available online for several of these examples. This survey builds on the earlier paper of Ilya Chevryev and Andrey Kormilitzin which had broadly similar aims at an earlier point in the development of this machinery. This article illustrates how the theoretical insights offered by signatures are simply realised in the analysis of application data in a way that is largely agnostic to the data type.
Paper Structure (33 sections, 9 theorems, 149 equations, 24 figures, 19 tables)

This paper contains 33 sections, 9 theorems, 149 equations, 24 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Background Material
  3. Illustrative Toy Regression Problem
  4. Incremental Streamed Data
  5. Mathematical Framework
  6. Simple Regression Approach
  7. Streams of Increments as Paths
  8. Signature Involvement
  9. Tensor Algebra and Signature
  10. Signature of Paths with Finite p-Variation
  11. Rough Paths
  12. Signature as a Feature Map
  13. Controlled Differential Equations and the Log-ODE Method
  14. Computing Signatures
  15. Expected Signature
  16. ...and 18 more sections

Key Result

Lemma 2.5

Consider an element $\textbf{ a} \in \tilde{T}((V))$ and consider $T(V^{\ast})$, the space of formal polynomials over $V^{\ast}$, to be equipped with the shuffle product. Define the evaluation map $\text{ev}_{\textbf{ a}} : T(V^{\ast}) \to {\mathbb R}$ by $\text{ev}_{\textbf{ a}} ( \textbf{ e}^{\ast

Figures (24)

  • Figure 1: Python code to compute the signature of the stream $\textbf{ x}$ truncated to depth $k$ using RoughPy
  • Figure 2: Python code to compute the signature of the stream $\textbf{ x}$ truncated to depth $k$ using RoughPy via a prescribed context.
  • Figure 3: Python code to compute signature of stream $\textbf{ x} \in {\cal S}({\mathbb R}^2)$ truncated to depth $3$. Including the returned output of the commands 'sig' and 'print(sig)'.
  • Figure 4: The outputs resulting from the commands 'logsigA', 'print(logsigA)', 'logsigB' and 'print(logsigB)' where logsigA and logsigB are the results of the python code in \ref{['roughpy_lsigA']} and \ref{['roughpy_lsigB']} respectively.
  • Figure 5: Python code to compute signature of stream $\textbf{ x} \in {\cal S}({\mathbb R}^3)$ truncated to depth $2$ both directly and by computing the exponential of the log signature.
  • ...and 19 more figures

Theorems & Definitions (19)

  • Definition 2.1: Admissible Tensor Norms
  • Definition 2.2: Exponential and Logarithm; Definition 2.20 in CLL04
  • Definition 2.3: Lie formal series over $V$; Definition 2.22 in CLL04
  • Definition 2.4: Group-like elements
  • Lemma 2.5: Equivalent definition of group-like; Variant of Theorem 2.23 in CLL04
  • Definition 2.6: Multiplicative Functional
  • Definition 2.7: Finite $p$-Variation
  • Definition 2.8: $p$-Rough Path
  • Theorem 2.9: Extension Theorem; Variant of Theorem 3.7 in CLL04
  • Definition 2.10: Signature of a $p$-rough path
  • ...and 9 more