Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Convolutional Neural Networks for Automated Echocardiography View Recognition: A Holistic Approach

Sarina Thomas, Cristiana Tiago, Børge Solli Andreassen, Svein Arne Aase, Jurica Šprem, Erik Steen, Anne Solberg, Guy Ben-Yosef

TL;DR

This work explores learning 3D heart meshes via graph convolutions, using similar techniques to learn 3D meshes in natural images, such as human pose estimation, to generate synthetic US images from 3D meshes by training an adversarial denoising diffusion model.

Abstract

To facilitate diagnosis on cardiac ultrasound (US), clinical practice has established several standard views of the heart, which serve as reference points for diagnostic measurements and define viewports from which images are acquired. Automatic view recognition involves grouping those images into classes of standard views. Although deep learning techniques have been successful in achieving this, they still struggle with fully verifying the suitability of an image for specific measurements due to factors like the correct location, pose, and potential occlusions of cardiac structures. Our approach goes beyond view classification and incorporates a 3D mesh reconstruction of the heart that enables several more downstream tasks, like segmentation and pose estimation. In this work, we explore learning 3D heart meshes via graph convolutions, using similar techniques to learn 3D meshes in natural images, such as human pose estimation. As the availability of fully annotated 3D images is limited, we generate synthetic US images from 3D meshes by training an adversarial denoising diffusion model. Experiments were conducted on synthetic and clinical cases for view recognition and structure detection. The approach yielded good performance on synthetic images and, despite being exclusively trained on synthetic data, it already showed potential when applied to clinical images. With this proof-of-concept, we aim to demonstrate the benefits of graphs to improve cardiac view recognition that can ultimately lead to better efficiency in cardiac diagnosis.

Graph Convolutional Neural Networks for Automated Echocardiography View Recognition: A Holistic Approach

TL;DR

This work explores learning 3D heart meshes via graph convolutions, using similar techniques to learn 3D meshes in natural images, such as human pose estimation, to generate synthetic US images from 3D meshes by training an adversarial denoising diffusion model.

Abstract

To facilitate diagnosis on cardiac ultrasound (US), clinical practice has established several standard views of the heart, which serve as reference points for diagnostic measurements and define viewports from which images are acquired. Automatic view recognition involves grouping those images into classes of standard views. Although deep learning techniques have been successful in achieving this, they still struggle with fully verifying the suitability of an image for specific measurements due to factors like the correct location, pose, and potential occlusions of cardiac structures. Our approach goes beyond view classification and incorporates a 3D mesh reconstruction of the heart that enables several more downstream tasks, like segmentation and pose estimation. In this work, we explore learning 3D heart meshes via graph convolutions, using similar techniques to learn 3D meshes in natural images, such as human pose estimation. As the availability of fully annotated 3D images is limited, we generate synthetic US images from 3D meshes by training an adversarial denoising diffusion model. Experiments were conducted on synthetic and clinical cases for view recognition and structure detection. The approach yielded good performance on synthetic images and, despite being exclusively trained on synthetic data, it already showed potential when applied to clinical images. With this proof-of-concept, we aim to demonstrate the benefits of graphs to improve cardiac view recognition that can ultimately lead to better efficiency in cardiac diagnosis.
Paper Structure (13 sections, 2 figures, 2 tables)

This paper contains 13 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Model embedding
  5. Data generation
  6. Diffusion model data generation
  7. Graph neural network
  8. Experiments & Results
  9. Data and Implementation:
  10. View recognition:
  11. 2D/3D Reconstruction:
  12. Structure localization:
  13. Discussion & Conclusion

Figures (2)

  • Figure 1: Pipeline overview: A) A diffusion model is trained with real US images and segmentations to generate synthetic images guided by the segmentations of the 3D mesh. B) Synthetic segmentations are sampled from 3D heart meshes along with the 3D coordinates of the model and the view. C) A GCN uses the 3D mesh vertices as nodes and predicts the 3D vertex positions on the 2D image.
  • Figure 2: Left: Confusion matrix of the view prediction and bounding box IoU for 480 synthetic and 248 clinical test cases. Right: Qualitative examples of the projected 3D GCN output applied to clinical cases (best, median, worst based on mIoU). The larger dots indicate where the mesh intersects the image plane (see example in suppl. video).