Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning

Hasindri Watawana, Kanchana Ranasinghe, Tariq Mahmood, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan

TL;DR

A novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images that achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.

Abstract

Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations. In this paper, we explore how the combination of domain specific natural language information with such hierarchical visual representations can benefit rich representation learning for medical image tasks. Building on automated language description generation for features visible in histopathology images, we present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. We explore contrastive objectives and granular language description based text alignment at multiple hierarchies to inject language modality information into the visual representations. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets. Our framework also provides better interpretability with our language aligned representation space. Code is available at https://github.com/Hasindri/HLSS.

Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning

TL;DR

A novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images that achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.

Abstract

Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations. In this paper, we explore how the combination of domain specific natural language information with such hierarchical visual representations can benefit rich representation learning for medical image tasks. Building on automated language description generation for features visible in histopathology images, we present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. We explore contrastive objectives and granular language description based text alignment at multiple hierarchies to inject language modality information into the visual representations. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets. Our framework also provides better interpretability with our language aligned representation space. Code is available at https://github.com/Hasindri/HLSS.
Paper Structure (17 sections, 5 equations, 5 figures, 5 tables)

This paper contains 17 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Background
  5. Architecture
  6. Hierarchical Vision Contrastive Objective
  7. Hierarchical Text-to-Vision Alignment
  8. Experiments
  9. Experimental Setup
  10. Results Comparison
  11. Ablations
  12. Interpretability
  13. Conclusion
  14. Acknowledgment
  15. Appendix
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Overview of HLSS Architecture
  • Figure 2: We illustrate the operations within our proposed Positive Pairing Module (left) and Cross-Modal Alignment Module (right).
  • Figure 3: Cosine similarity of a given sample representation with class specific cancer markers ($M_{i}$). Markers of the ground truth class (GT) are relatively more similar.
  • Figure 4: Cosine similarity of a 'normal' tissue sample representation with class specific cancer markers ($M_{i}$) of all classes is illustrated. Observe how the representation doesn't align well with majority markers in any class.
  • Figure 5: Cosine similarity of a LGG (lower grade glioma) sample representation with class specific cancer markers ($M_{i}$). A proper alignment can only be seen in HGG & LGG classes as they both consider similar characteristics as per the markers we obtained from an expert histopathologist.