Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embedded Variational Neural Stochastic Differential Equations for Learning Heterogeneous Dynamics

Sandeep Kumar Samota, Reema Gupta, Snehashish Chakraverty

Abstract

This study examines the challenges of modeling complex and noisy data related to socioeconomic factors over time, with a focus on data from various districts in Odisha, India. Traditional time-series models struggle to capture both trends and variations together in this type of data. To tackle this, a Variational Neural Stochastic Differential Equation (V-NSDE) model is designed that combines the expressive dynamics of Neural SDEs with the generative capabilities of Variational Autoencoders (VAEs). This model uses an encoder and a decoder. The encoder takes the initial observations and district embeddings and translates them into a Gaussian distribution, which determines the mean and log-variance of the first latent state. Then the obtained latent state initiates the Neural SDE, which utilize neural networks to determine the drift and diffusion functions that govern continuous-time latent dynamics. These governing functions depend on the time index, latent state, and district embedding, which help the model learn the unique characteristics specific to each district. After that, using a probabilistic decoder, the observations are reconstructed from the latent trajectory. The decoder outputs a mean and log-variance for each time step, which follows the Gaussian likelihood. The Evidence Lower Bound (ELBO) training loss improves by adding a KL-divergence regularization term to the negative log-likelihood (nll). The obtained results demonstrate the effective learning of V-NSDE in recognizing complex patterns over time, yielding realistic outcomes that include clear trends and random fluctuations across different areas.

Embedded Variational Neural Stochastic Differential Equations for Learning Heterogeneous Dynamics

Abstract

This study examines the challenges of modeling complex and noisy data related to socioeconomic factors over time, with a focus on data from various districts in Odisha, India. Traditional time-series models struggle to capture both trends and variations together in this type of data. To tackle this, a Variational Neural Stochastic Differential Equation (V-NSDE) model is designed that combines the expressive dynamics of Neural SDEs with the generative capabilities of Variational Autoencoders (VAEs). This model uses an encoder and a decoder. The encoder takes the initial observations and district embeddings and translates them into a Gaussian distribution, which determines the mean and log-variance of the first latent state. Then the obtained latent state initiates the Neural SDE, which utilize neural networks to determine the drift and diffusion functions that govern continuous-time latent dynamics. These governing functions depend on the time index, latent state, and district embedding, which help the model learn the unique characteristics specific to each district. After that, using a probabilistic decoder, the observations are reconstructed from the latent trajectory. The decoder outputs a mean and log-variance for each time step, which follows the Gaussian likelihood. The Evidence Lower Bound (ELBO) training loss improves by adding a KL-divergence regularization term to the negative log-likelihood (nll). The obtained results demonstrate the effective learning of V-NSDE in recognizing complex patterns over time, yielding realistic outcomes that include clear trends and random fluctuations across different areas.

Paper Structure

This paper contains 37 sections, 1 theorem, 58 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Mathematical Preliminaries
  3. Probability Space and Random Variables
  4. Bayes' Rule
  5. Monte Carlo Approximation
  6. Brownian Motion
  7. Itô SDEs
  8. V-NSDE Methodology
  9. Forward Architecture Components
  10. District Embedding
  11. Encoder: $q_{\phi}(\mathbf{z}_0 | \mathbf{y}_0, \mathbf{e}_{d})$
  12. Latent NSDE
  13. Decoder: $p_{\theta}(\mathbf{y}_t | \mathbf{z}_t, \mathbf{e}_{d})$
  14. Backward Model Architecture Components
  15. Evidence Lower Bound (ELBO) Loss Function Derivation
  16. ...and 22 more sections

Key Result

Theorem A.1

Under Assumptions A--E, for each district $d$, the latent NSDE in Eq. eq:latent_nsde admits a unique strong solution such that $Z_t^{(d)}$ is $\mathcal{F}_t$-adapted and almost surely continuous and $\mathbb{E}\left[\sup_{t\in[0,T]} \|Z_t^{(d)}\|^2\right] < \infty$. $\blacktriangleleft$$\blacktriangleleft$

Figures (5)

  • Figure 1: Block diagram of the proposed V-NSDE architecture for district $d$.
  • Figure 2: Training losses over epochs
  • Figure 3: Likelihood predictions for Balangir District
  • Figure 4: Likelihood predictions for Koraput District
  • Figure 5: Likelihood predictions for Kalahandi District

Theorems & Definitions (10)

  • Definition 2.1: rohatgi2015introduction
  • Definition 2.2: rohatgi2015introduction
  • Definition 2.3: rohatgi2015introduction
  • Definition 2.4: rohatgi2015introduction
  • Definition 2.5: rohatgi2015introduction
  • Definition 2.6: kullback1951information
  • Definition 2.7: oksendal2013stochastic
  • Definition 2.8: oksendal2013stochastic
  • Theorem A.1: Existence and Uniqueness of Latent NSDE
  • proof