Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Backward Filtering Forward Guiding

Gefan Yang, Frank van der Meulen, Stefan Sommer

TL;DR

This work tackles smoothing of nonlinear, high-dimensional processes on tree-structured graphs with sparse leaf observations. It introduces Neural Backward Filtering Forward Guiding (NBFFG), a two-phase variational framework that first builds a tractable guided proposal via a linear-Gaussian auxiliary process and then learns a neural residual (via normalizing flows or neural SDEs) to capture nonlinear discrepancies, enabling unbiased path-wise subsampling. The method unifies discrete and continuous dynamics, supports root inference, and achieves scalable inference through path-based amortization across the topology. Empirical results across linear, nonlinear, and high-dimensional phylogenetic tasks demonstrate accurate posterior recovery, improved multimodality handling, and practical applicability to complex evolutionary shape reconstruction.

Abstract

Inference in non-linear continuous stochastic processes on trees is challenging, particularly when observations are sparse (leaf-only) and the topology is complex. Exact smoothing via Doob's $h$-transform is intractable for general non-linear dynamics, while particle-based methods degrade in high dimensions. We propose Neural Backward Filtering Forward Guiding (NBFFG), a unified framework for both discrete transitions and continuous diffusions. Our method constructs a variational posterior by leveraging an auxiliary linear-Gaussian process. This auxiliary process yields a closed-form backward filter that serves as a ``guide'', steering the generative path toward high-likelihood regions. We then learn a neural residual--parameterized as a normalizing flow or a controlled SDE--to capture the non-linear discrepancies. This formulation allows for an unbiased path-wise subsampling scheme, reducing the training complexity from tree-size dependent to path-length dependent. Empirical results show that NBFFG outperforms baselines on synthetic benchmarks, and we demonstrate the method on a high-dimensional inference task in phylogenetic analysis with reconstruction of ancestral butterfly wing shapes.

Neural Backward Filtering Forward Guiding

TL;DR

This work tackles smoothing of nonlinear, high-dimensional processes on tree-structured graphs with sparse leaf observations. It introduces Neural Backward Filtering Forward Guiding (NBFFG), a two-phase variational framework that first builds a tractable guided proposal via a linear-Gaussian auxiliary process and then learns a neural residual (via normalizing flows or neural SDEs) to capture nonlinear discrepancies, enabling unbiased path-wise subsampling. The method unifies discrete and continuous dynamics, supports root inference, and achieves scalable inference through path-based amortization across the topology. Empirical results across linear, nonlinear, and high-dimensional phylogenetic tasks demonstrate accurate posterior recovery, improved multimodality handling, and practical applicability to complex evolutionary shape reconstruction.

Abstract

Inference in non-linear continuous stochastic processes on trees is challenging, particularly when observations are sparse (leaf-only) and the topology is complex. Exact smoothing via Doob's -transform is intractable for general non-linear dynamics, while particle-based methods degrade in high dimensions. We propose Neural Backward Filtering Forward Guiding (NBFFG), a unified framework for both discrete transitions and continuous diffusions. Our method constructs a variational posterior by leveraging an auxiliary linear-Gaussian process. This auxiliary process yields a closed-form backward filter that serves as a ``guide'', steering the generative path toward high-likelihood regions. We then learn a neural residual--parameterized as a normalizing flow or a controlled SDE--to capture the non-linear discrepancies. This formulation allows for an unbiased path-wise subsampling scheme, reducing the training complexity from tree-size dependent to path-length dependent. Empirical results show that NBFFG outperforms baselines on synthetic benchmarks, and we demonstrate the method on a high-dimensional inference task in phylogenetic analysis with reconstruction of ancestral butterfly wing shapes.
Paper Structure (46 sections, 4 theorems, 46 equations, 11 figures)

This paper contains 46 sections, 4 theorems, 46 equations, 11 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Smoothing of state-space models
  4. Sampling methods
  5. Smoothing of diffusions
  6. Preliminaries
  7. Probabilistic models on tree diagrams
  8. Doob's $h$-transform
  9. Discrete dynamics
  10. Continuous dynamics
  11. Backward information filter
  12. Methodology
  13. Guided proposal
  14. Variational posterior
  15. Discrete dynamics
  16. ...and 31 more sections

Key Result

Theorem 4.1

Assume that for each $v \in \mathcal{V}^+$, $\mathbb{Q}_v^{\theta}$ is absolutely continuous with respect to $\mathbb{P}_v$. For each leaf $l \in \mathcal{L}$, let $y_l$ be the observed realization of the leaf state $X_l$, characterized by the conditional likelihood density $\ell(y_l \mid X_l)$. The Then $\mathcal{J}(\theta) = D_{\mathrm{KL}}(\mathbb{Q}^{\theta} \Vert \Pi) - C$, with $C = \log h_{

Figures (11)

  • Figure 1: Validation on Linear Gaussian Benchmarks. We compare the converged training loss against the analytical RTS smoother baseline. (a) Topological Scalability: Relative error decreases as tree complexity ($N_{\mathrm{depth}}, N_{\mathrm{branch}}$) grows, showing that our path-wise amortization effectively leverages larger datasets rather than degrading. (b) Dimensional Scalability: The method remains robust in high dimensions, with relative error bounded below $3\%$ even as state space grows to $d=256$.
  • Figure 2: Empirical distributions of 500 independent samples of the guided proposal (gray) and the learned variational posterior (orange) against the analytical ground truth (RTS, green contours) at different tree depths.
  • Figure 3: Visualization of double-well diffusion conditioned on the leaf observation $[-1, -1, 1, 1]^{\top}$ on a binary tree. The figure displays 20 samples from (a) the prior; (b) the raw guided proposal; (c) the corrected guided proposal with MCMC; (d) the learned variational posterior. Trajectories belonging to the same edge are identified by color. For example, in (a), the orange paths (leading to one of the children of the root) show that the process moves with equal probability to either of the modes at $+1$ and $-1$. In the transition from time $4.0$ to $5.0$, the process remains in the same mode. Green dots at $t=0.0$ represent the root, and stars at $t=5.0$ represent the observed values $\pm 1$ at the leaf nodes. The background green shading illustrates the potential wells.
  • Figure 4: Phylogenetic tree topology and ancestral shape reconstruction. (top) The Papilio family tree topology, where nodes (blue dots) represent ancestral lineages and tips represent extant species. (bottom) Reconstructed posterior mean shape of the ancestors (red) overlaid with the observed morphological variation of their leaf species (gray or colored shade curves).
  • Figure 5: NELBO convergence with and without subsampling. Optimization trajectories are shown for: (a) Linear Gaussian model on a balanced tree and (b) Ornstein-Uhlenbeck process on an irregular tree. The dashed green line represents the analytical ground truth. While the raw subsampling estimator (light blue) exhibits higher stochasticity, its moving average (dark blue) consistently converges to the theoretical optimum alongside the full-tree baseline (orange).
  • ...and 6 more figures

Theorems & Definitions (10)

  • Definition 3.1
  • Definition 3.2
  • Theorem 4.1
  • Corollary 4.2
  • Corollary 4.3
  • Proposition 4.4
  • proof
  • proof
  • proof
  • proof