Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Images

Chen Liu, Ke Xu, Liangbo L. Shen, Guillaume Huguet, Zilong Wang, Alexander Tong, Danilo Bzdok, Jay Stewart, Jay C. Wang, Lucian V. Del Priore, Smita Krishnaswamy

TL;DR

ImageFlowNet addresses the challenge of forecasting disease trajectories from irregularly sampled longitudinal medical images while preserving spatial detail. It constructs multiscale joint patient representations and evolves latent states via a position parameterized neural ODE or SDE, using a UNet backbone and multiple regularizations to stabilize learning. The work provides theoretical insights including an equivalence with original neural ODE dynamics and a dynamic OT interpretation, and demonstrates superior forecasting performance across retinal geographic atrophy, brain multiple sclerosis, and glioblastoma datasets. Practically, ImageFlowNet enables image level trajectory visualization and prediction without heavy feature engineering, with test time optimization offering further gains and a stochastic variant enabling uncertainty quantification.

Abstract

Advances in medical imaging technologies have enabled the collection of longitudinal images, which involve repeated scanning of the same patients over time, to monitor disease progression. However, predictive modeling of such data remains challenging due to high dimensionality, irregular sampling, and data sparsity. To address these issues, we propose ImageFlowNet, a novel model designed to forecast disease trajectories from initial images while preserving spatial details. ImageFlowNet first learns multiscale joint representation spaces across patients and time points, then optimizes deterministic or stochastic flow fields within these spaces using a position-parameterized neural ODE/SDE framework. The model leverages a UNet architecture to create robust multiscale representations and mitigates data scarcity by combining knowledge from all patients. We provide theoretical insights that support our formulation of ODEs, and motivate our regularizations involving high-level visual features, latent space organization, and trajectory smoothness. We validate ImageFlowNet on three longitudinal medical image datasets depicting progression in geographic atrophy, multiple sclerosis, and glioblastoma, demonstrating its ability to effectively forecast disease progression and outperform existing methods. Our contributions include the development of ImageFlowNet, its theoretical underpinnings, and empirical validation on real-world datasets. The official implementation is available at https://github.com/KrishnaswamyLab/ImageFlowNet.

ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Images

TL;DR

ImageFlowNet addresses the challenge of forecasting disease trajectories from irregularly sampled longitudinal medical images while preserving spatial detail. It constructs multiscale joint patient representations and evolves latent states via a position parameterized neural ODE or SDE, using a UNet backbone and multiple regularizations to stabilize learning. The work provides theoretical insights including an equivalence with original neural ODE dynamics and a dynamic OT interpretation, and demonstrates superior forecasting performance across retinal geographic atrophy, brain multiple sclerosis, and glioblastoma datasets. Practically, ImageFlowNet enables image level trajectory visualization and prediction without heavy feature engineering, with test time optimization offering further gains and a stochastic variant enabling uncertainty quantification.

Abstract

Advances in medical imaging technologies have enabled the collection of longitudinal images, which involve repeated scanning of the same patients over time, to monitor disease progression. However, predictive modeling of such data remains challenging due to high dimensionality, irregular sampling, and data sparsity. To address these issues, we propose ImageFlowNet, a novel model designed to forecast disease trajectories from initial images while preserving spatial details. ImageFlowNet first learns multiscale joint representation spaces across patients and time points, then optimizes deterministic or stochastic flow fields within these spaces using a position-parameterized neural ODE/SDE framework. The model leverages a UNet architecture to create robust multiscale representations and mitigates data scarcity by combining knowledge from all patients. We provide theoretical insights that support our formulation of ODEs, and motivate our regularizations involving high-level visual features, latent space organization, and trajectory smoothness. We validate ImageFlowNet on three longitudinal medical image datasets depicting progression in geographic atrophy, multiple sclerosis, and glioblastoma, demonstrating its ability to effectively forecast disease progression and outperform existing methods. Our contributions include the development of ImageFlowNet, its theoretical underpinnings, and empirical validation on real-world datasets. The official implementation is available at https://github.com/KrishnaswamyLab/ImageFlowNet.
Paper Structure (58 sections, 5 theorems, 27 equations, 7 figures, 3 tables)

This paper contains 58 sections, 5 theorems, 27 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Background
  3. Problem Formulation and Notation
  4. Neural Ordinary Differential Equations (Neural ODEs)
  5. Neural Stochastic Differential Equations (Neural SDEs)
  6. UNet
  7. ImageFlowNet
  8. Learning Multiscale Spaces of Joint Patient Representations
  9. Learning Multiscale Flow Fields on Joint Patient Representations
  10. Training and Inference
  11. Training Objectives
  12. ImageFlowNet$_\textrm{SDE}$ as a Stochastic Variant
  13. Theoretical Results
  14. A Position-Parameterized ODE
  15. Connections to Dynamic Optimal Transport
  16. ...and 43 more sections

Key Result

Proposition 4.1

Let $f_{\theta}$ be a continuous function that satisfies the Lipschitz continuity and linear growth conditions. Also, let the initial state $y(t_0) = y_0$ satisfy the finite second moment requirement $(\mathbb{E}[\|y(t_0)\|^2] < \infty)$. Suppose $z(t_0)$ is the latent representation learned by Imag

Figures (7)

  • Figure 1: Advantages of image-level trajectory inferece.
  • Figure 2: Overview of the proposed ImageFlowNet. (A) The model uses an earlier image $x_i$ at time $t_i$ as well as the change in time $t_j - t_i$ to forecast the future image $x_j$ at time $t_j$. (B) For each hidden layer, a separate flow field $f_\theta$ is used to model the joint patient embedding space. Trajectory inference can be performed by integration along this flow field. It should be noted that the change in time $t_j - t_i$ is sufficient for integration in practice, while the exact time values $t_i$ and $t_j$ are included in the integral merely for mathematical clarity. (C) The learning objective has four components. The loss function and modules affected by each component are illustrated.
  • Figure 3: Qualitative comparison of image forecasting results on the retinal geographic atrophy, multiple sclerosis and glioblastoma datasets. "++" means using the 3 regularizations in Eqn \ref{['eqn:loss_total']}.
  • Figure 4: Joint representation space and the effect of contrastive learning regularization. Red dots are the observed disease states and arrows connect adjacent transitions. Normalized time is color coded. (A) Without regularization ($\lambda_c = 0$). (B) With contrastive learning regularization ($\lambda_c = 0.01$).
  • Figure 5: ImageFlowNetSDE alternative trajectories. Multiple inferences predict non-identical disease progressions, and their vectors in the joint representations space indeed follow different trajectories.
  • ...and 2 more figures

Theorems & Definitions (7)

  • Proposition 4.1
  • Proposition 4.2
  • Proposition A.1
  • proof
  • Theorem A.2: Picard-Lindelöf Book_DE
  • Proposition A.3
  • proof