Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking U-net Skip Connections for Biomedical Image Segmentation

Frauke Wilm, Jonas Ammeling, Mathias Öttl, Rutger H. J. Fick, Marc Aubreville, Katharina Breininger

TL;DR

This work used a synthetic dataset to model different levels of data distribution shifts of U-net-style segmentation networks, quantified the inherent domain susceptibility of each network layer, and confirmed the higher domain susceptibility of earlier network layers.

Abstract

The U-net architecture has significantly impacted deep learning-based segmentation of medical images. Through the integration of long-range skip connections, it facilitated the preservation of high-resolution features. Out-of-distribution data can, however, substantially impede the performance of neural networks. Previous works showed that the trained network layers differ in their susceptibility to this domain shift, e.g., shallow layers are more affected than deeper layers. In this work, we investigate the implications of this observation of layer sensitivity to domain shifts of U-net-style segmentation networks. By copying features of shallow layers to corresponding decoder blocks, these bear the risk of re-introducing domain-specific information. We used a synthetic dataset to model different levels of data distribution shifts and evaluated the impact on downstream segmentation performance. We quantified the inherent domain susceptibility of each network layer, using the Hellinger distance. These experiments confirmed the higher domain susceptibility of earlier network layers. When gradually removing skip connections, a decrease in domain susceptibility of deeper layers could be observed. For downstream segmentation performance, the original U-net outperformed the variant without any skip connections. The best performance, however, was achieved when removing the uppermost skip connection - not only in the presence of domain shifts but also for in-domain test data. We validated our results on three clinical datasets - two histopathology datasets and one magnetic resonance dataset - with performance increases of up to 10% in-domain and 13% cross-domain when removing the uppermost skip connection.

Rethinking U-net Skip Connections for Biomedical Image Segmentation

TL;DR

This work used a synthetic dataset to model different levels of data distribution shifts of U-net-style segmentation networks, quantified the inherent domain susceptibility of each network layer, and confirmed the higher domain susceptibility of earlier network layers.

Abstract

The U-net architecture has significantly impacted deep learning-based segmentation of medical images. Through the integration of long-range skip connections, it facilitated the preservation of high-resolution features. Out-of-distribution data can, however, substantially impede the performance of neural networks. Previous works showed that the trained network layers differ in their susceptibility to this domain shift, e.g., shallow layers are more affected than deeper layers. In this work, we investigate the implications of this observation of layer sensitivity to domain shifts of U-net-style segmentation networks. By copying features of shallow layers to corresponding decoder blocks, these bear the risk of re-introducing domain-specific information. We used a synthetic dataset to model different levels of data distribution shifts and evaluated the impact on downstream segmentation performance. We quantified the inherent domain susceptibility of each network layer, using the Hellinger distance. These experiments confirmed the higher domain susceptibility of earlier network layers. When gradually removing skip connections, a decrease in domain susceptibility of deeper layers could be observed. For downstream segmentation performance, the original U-net outperformed the variant without any skip connections. The best performance, however, was achieved when removing the uppermost skip connection - not only in the presence of domain shifts but also for in-domain test data. We validated our results on three clinical datasets - two histopathology datasets and one magnetic resonance dataset - with performance increases of up to 10% in-domain and 13% cross-domain when removing the uppermost skip connection.
Paper Structure (20 sections, 9 equations, 9 figures, 2 tables)

This paper contains 20 sections, 9 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. U-net for biomedical image segmentation
  4. Domain susceptibility of cnn architectures
  5. Measuring domain shift
  6. Materials
  7. Synthetic malaria dataset
  8. Multi-scanner mitotic figure dataset (MS-MF)
  9. Multi-scanner canine cutaneous tumor dataset (MS-CCT)
  10. Multi-vendor magnetic resonance dataset (MV-MR)
  11. Methods
  12. Domain shift modeling
  13. Hellinger distance for domain shift quantification
  14. Skip connection pruning
  15. Results
  16. ...and 5 more sections

Figures (9)

  • Figure 1: Schematic illustrations of the baseline U-net and the pruned architectures. For the L1-pruned U-net (cyan), the uppermost skip connection was removed. Consecutively removing the layer-wise skip connections resulted in the L2-pruned (orange), L3-pruned (red), and L4-pruned (teal) U-net.
  • Figure 2: Templates used for the creation of the synthetic malaria dataset: background (left column), cells (middle column), and artifacts (right column).
  • Figure 3: Exemplary images and corresponding masks of the synthetically created malaria dataset.
  • Figure 4: Multi-scanner histopathology dataset for mitotic figure segmentation. Each row visualizes a concentric mitotic figure on the same tissue sample digitized with six scanning systems. A: Aperio ScanScope CS2 (Leica), B: NanoZoomer 2.0-HT (Hamamatsu), C: NanoZoomer S360 (Hamamatsu), D: Pannoramic 250 Flash III (3DHISTECH), E: Pannoramic SCAN II (3DHISTECH), F: SG60 (Philips).
  • Figure 5: Multi-scanner histopathology dataset for tumor segmentation. Each row visualizes a patch from the same tissue sample digitized with five scanning systems (white: background, gray: non-tumor, black: tumor). A: Aperio ScanScope CS2 (Leica), B: NanoZoomer S210 (Hamamatsu), C: NanoZoomer 2.0-HT (Hamamatsu), D: Pannoramic 1000 (3DHISTECH), E: Aperio GT450 (Leica).
  • ...and 4 more figures