Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatiotemporal Learning with Context-aware Video Tubelets for Ultrasound Video Analysis

Gary Y. Li, Li Chen, Bryson Hicks, Nikolai Schnittke, David O. Kessler, Jeffrey Shupp, Maria Parker, Cristiana Baloescu, Christopher Moore, Cynthia Gregory, Kenton Gregory, Balasundar Raju, Jochen Kruecker, Alvin Chen

TL;DR

This work tackles the challenge of reliable pathology detection in ultrasound videos by preserving global spatial context while learning fine spatiotemporal features. It introduces context-aware video tubelets that embed tubelet location, size, and confidence, and leverages ROI-aligned detector feature maps to form video tubelets for efficient processing. A lightweight spatiotemporal classifier fuses per-frame tubelet features with non-image context to produce robust video-level predictions. Across a large, multi-site LUS dataset, the approach outperforms prior tubelet-based methods and demonstrates potential for real-time clinical workflows.

Abstract

Computer-aided pathology detection algorithms for video-based imaging modalities must accurately interpret complex spatiotemporal information by integrating findings across multiple frames. Current state-of-the-art methods operate by classifying on video sub-volumes (tubelets), but they often lose global spatial context by focusing only on local regions within detection ROIs. Here we propose a lightweight framework for tubelet-based object detection and video classification that preserves both global spatial context and fine spatiotemporal features. To address the loss of global context, we embed tubelet location, size, and confidence as inputs to the classifier. Additionally, we use ROI-aligned feature maps from a pre-trained detection model, leveraging learned feature representations to increase the receptive field and reduce computational complexity. Our method is efficient, with the spatiotemporal tubelet classifier comprising only 0.4M parameters. We apply our approach to detect and classify lung consolidation and pleural effusion in ultrasound videos. Five-fold cross-validation on 14,804 videos from 828 patients shows our method outperforms previous tubelet-based approaches and is suited for real-time workflows.

Spatiotemporal Learning with Context-aware Video Tubelets for Ultrasound Video Analysis

TL;DR

This work tackles the challenge of reliable pathology detection in ultrasound videos by preserving global spatial context while learning fine spatiotemporal features. It introduces context-aware video tubelets that embed tubelet location, size, and confidence, and leverages ROI-aligned detector feature maps to form video tubelets for efficient processing. A lightweight spatiotemporal classifier fuses per-frame tubelet features with non-image context to produce robust video-level predictions. Across a large, multi-site LUS dataset, the approach outperforms prior tubelet-based methods and demonstrates potential for real-time clinical workflows.

Abstract

Computer-aided pathology detection algorithms for video-based imaging modalities must accurately interpret complex spatiotemporal information by integrating findings across multiple frames. Current state-of-the-art methods operate by classifying on video sub-volumes (tubelets), but they often lose global spatial context by focusing only on local regions within detection ROIs. Here we propose a lightweight framework for tubelet-based object detection and video classification that preserves both global spatial context and fine spatiotemporal features. To address the loss of global context, we embed tubelet location, size, and confidence as inputs to the classifier. Additionally, we use ROI-aligned feature maps from a pre-trained detection model, leveraging learned feature representations to increase the receptive field and reduce computational complexity. Our method is efficient, with the spatiotemporal tubelet classifier comprising only 0.4M parameters. We apply our approach to detect and classify lung consolidation and pleural effusion in ultrasound videos. Five-fold cross-validation on 14,804 videos from 828 patients shows our method outperforms previous tubelet-based approaches and is suited for real-time workflows.

Paper Structure

This paper contains 12 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methods
  4. Spatiotemporal framework for ultrasound video analysis based on context-aware video tubelets
  5. Generating video tubelets from detector features
  6. Embedding tubelet location, size, and confidence
  7. Spatiotemporal tubelet classifier
  8. Results
  9. Experimental settings
  10. Comparison methods for video-level classification
  11. Ablation study
  12. Conclusion

Figures (2)

  • Figure 1: Example ultrasound frames containing pathologies misclassified by tubelet classifier li2023weakly but correctly classified by our context-aware tubelet classifier. Orange=tubelets classifier. Green=Context-aware tubelets classifier. White=ground-truth.
  • Figure 2: Spatiotemporal framework for ultrasound video analysis based on context-aware video tubelets. F4 represents one frame's feature maps from the final convolutional layer of the 4-th stage of the YOLOv5 backbone