Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processes

Zhiheng Chen, Guanhua Fang, Wen Yu

TL;DR

This work addresses non-asymptotic generalization guarantees for recurrent neural network-based temporal point processes (RNN-TPPs). It develops a truncation-based stochastic-error bound and a covering-number complexity analysis for multi-layer RNN-TPPs, together with explicit construction schemes to approximate the intensity functions of Poisson and Hawkes-type processes. The main results show that two-layer RNN-TPPs can achieve vanishing excess risk in Poisson and vanilla Hawkes settings under mild smoothness assumptions, while nonlinear Hawkes requires up to four layers for similar guarantees; rates depend on the smoothness of the baseline intensity and the excitation function. The findings bridge neural network theory with temporal point process modeling, providing practical architecture guidance and theoretical performance guarantees for neural TPP methods.

Abstract

Temporal point process (TPP) is an important tool for modeling and predicting irregularly timed events across various domains. Recently, the recurrent neural network (RNN)-based TPPs have shown practical advantages over traditional parametric TPP models. However, in the current literature, it remains nascent in understanding neural TPPs from theoretical viewpoints. In this paper, we establish the excess risk bounds of RNN-TPPs under many well-known TPP settings. We especially show that an RNN-TPP with no more than four layers can achieve vanishing generalization errors. Our technical contributions include the characterization of the complexity of the multi-layer RNN class, the construction of $\tanh$ neural networks for approximating dynamic event intensity functions, and the truncation technique for alleviating the issue of unbounded event sequences. Our results bridge the gap between TPP's application and neural network theory.

On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processes

TL;DR

This work addresses non-asymptotic generalization guarantees for recurrent neural network-based temporal point processes (RNN-TPPs). It develops a truncation-based stochastic-error bound and a covering-number complexity analysis for multi-layer RNN-TPPs, together with explicit construction schemes to approximate the intensity functions of Poisson and Hawkes-type processes. The main results show that two-layer RNN-TPPs can achieve vanishing excess risk in Poisson and vanilla Hawkes settings under mild smoothness assumptions, while nonlinear Hawkes requires up to four layers for similar guarantees; rates depend on the smoothness of the baseline intensity and the excitation function. The findings bridge neural network theory with temporal point process modeling, providing practical architecture guidance and theoretical performance guarantees for neural TPP methods.

Abstract

Temporal point process (TPP) is an important tool for modeling and predicting irregularly timed events across various domains. Recently, the recurrent neural network (RNN)-based TPPs have shown practical advantages over traditional parametric TPP models. However, in the current literature, it remains nascent in understanding neural TPPs from theoretical viewpoints. In this paper, we establish the excess risk bounds of RNN-TPPs under many well-known TPP settings. We especially show that an RNN-TPP with no more than four layers can achieve vanishing generalization errors. Our technical contributions include the characterization of the complexity of the multi-layer RNN class, the construction of neural networks for approximating dynamic event intensity functions, and the truncation technique for alleviating the issue of unbounded event sequences. Our results bridge the gap between TPP's application and neural network theory.
Paper Structure (33 sections, 23 theorems, 240 equations, 5 figures)

This paper contains 33 sections, 23 theorems, 240 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Framework Specification
  4. RNN Structure
  5. Classical TPPs
  6. Notations
  7. Main Results
  8. Stochastic Error
  9. Main Variance Term
  10. Key Techniques
  11. Probability Bound of Events Number
  12. From Unboundedness to Boundedness
  13. Complexity of the RNN-TPP Class
  14. Approximation Error
  15. Poisson Case
  16. ...and 18 more sections

Key Result

Theorem 1

Under model eq:label:hawkes and RNN-TPP class $\mathcal{F} = \mathcal{F}_{L, D, B_m, l_f, u_f}$ defined as rnn_func_class(main), suppose that assumptions (A1)-(A3) hold, then for $n$ i.i.d. sample series $\{S_i, i \in [n]\}$, with probability at least $1-\delta$, the excess risk eq:def:gen:err of ER (ii) (Vanilla Hawkes case) If $\mu(t) = \alpha\exp(-\beta t)$, for $L = 2$, $D = \tilde{O}(n^{\frac

Figures (5)

  • Figure 1: Left: the classical RNN architecture. Right: the RNN-TPP architecture given in \ref{['eq:form:lam']} - \ref{['eq:form:h:j']}. The blue box represents the interpolation of hidden states.
  • Figure 2: The construction of RNN-TPP for the case of Poisson processes.
  • Figure 3: The construction of RNN-TPP for the case of the vanilla Hawkes process.
  • Figure 4: The construction of RNN-TPP for the case of general linear Hawkes processes.
  • Figure 5: The construction of RNN-TPP for the case of nonlinear Hawkes process.

Theorems & Definitions (51)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Theorem 1
  • Remark 6
  • Theorem 2
  • Remark 7
  • Lemma 1
  • ...and 41 more