Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatiotemporal Representation Learning for Short and Long Medical Image Time Series

Chengzhi Shen, Martin J. Menten, Hrvoje Bogunović, Ursula Schmidt-Erfurth, Hendrik Scholl, Sobha Sivaprasad, Andrew Lotery, Daniel Rueckert, Paul Hager, Robbie Holland

TL;DR

This work combines clip-level contrastive learning with a novel temporal embedding to adapt to irregular time series and proposes masking and predicting latent frame representations of the temporal sequence, enabling the automated analysis of temporal patterns which are typically overlooked in applications of deep learning to medicine.

Abstract

Analyzing temporal developments is crucial for the accurate prognosis of many medical conditions. Temporal changes that occur over short time scales are key to assessing the health of physiological functions, such as the cardiac cycle. Moreover, tracking longer term developments that occur over months or years in evolving processes, such as age-related macular degeneration (AMD), is essential for accurate prognosis. Despite the importance of both short and long term analysis to clinical decision making, they remain understudied in medical deep learning. State of the art methods for spatiotemporal representation learning, developed for short natural videos, prioritize the detection of temporal constants rather than temporal developments. Moreover, they do not account for varying time intervals between acquisitions, which are essential for contextualizing observed changes. To address these issues, we propose two approaches. First, we combine clip-level contrastive learning with a novel temporal embedding to adapt to irregular time series. Second, we propose masking and predicting latent frame representations of the temporal sequence. Our two approaches outperform all prior methods on temporally-dependent tasks including cardiac output estimation and three prognostic AMD tasks. Overall, this enables the automated analysis of temporal patterns which are typically overlooked in applications of deep learning to medicine.

Spatiotemporal Representation Learning for Short and Long Medical Image Time Series

TL;DR

This work combines clip-level contrastive learning with a novel temporal embedding to adapt to irregular time series and proposes masking and predicting latent frame representations of the temporal sequence, enabling the automated analysis of temporal patterns which are typically overlooked in applications of deep learning to medicine.

Abstract

Analyzing temporal developments is crucial for the accurate prognosis of many medical conditions. Temporal changes that occur over short time scales are key to assessing the health of physiological functions, such as the cardiac cycle. Moreover, tracking longer term developments that occur over months or years in evolving processes, such as age-related macular degeneration (AMD), is essential for accurate prognosis. Despite the importance of both short and long term analysis to clinical decision making, they remain understudied in medical deep learning. State of the art methods for spatiotemporal representation learning, developed for short natural videos, prioritize the detection of temporal constants rather than temporal developments. Moreover, they do not account for varying time intervals between acquisitions, which are essential for contextualizing observed changes. To address these issues, we propose two approaches. First, we combine clip-level contrastive learning with a novel temporal embedding to adapt to irregular time series. Second, we propose masking and predicting latent frame representations of the temporal sequence. Our two approaches outperform all prior methods on temporally-dependent tasks including cardiac output estimation and three prognostic AMD tasks. Overall, this enables the automated analysis of temporal patterns which are typically overlooked in applications of deep learning to medicine.
Paper Structure (22 sections, 3 equations, 2 figures, 2 tables)

This paper contains 22 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-Supervised Video Representation Learning
  4. Representation Learning for Spatiotemporal Medical Data
  5. Materials and Methods
  6. Short and Long Time Scale Datasets
  7. Cardiac MR video
  8. Longitudinal Retinal OCT
  9. Spatiotemporal Encoder
  10. Clip-level Contrastive Learning
  11. Time Embedding
  12. Temporally-Variant Feature Prediction
  13. Experiments and Results
  14. Experimental Setup
  15. Pretraining Protocol
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Clinicians use spatiotemporal data to observe temporal variations over both short and long time scales. To observe dynamic physiological processes such as a beating heart, short cardiac MR videos can be captured (top). Tracking long term developments, such as disease progression in retinal OCT scans, require longitudinal acquisitions that typically occur at irregular intervals of years (bottom). Modeling and extrapolating the trajectory of historical change is crucial for the prognosis of late stage disease.
  • Figure 2: A. Our spatiotemporal encoder extracts representation of the sequence using the CLS token of the temporal Transformer $E_T$. B. Standard contrastive approaches prioritize learning features that persist across different segments of the same sequence (left). Our approaches use a single clip to construct contrastive pairs combined with time embedding (middle) and a frame-level feature predictive approach (right) to model temporal variation over the sequence.