Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Tissue Plane to Organ World: A Benchmark Dataset for Multimodal Biomedical Image Registration using Deep Co-Attention Networks

Yifeng Wang, Weipeng Li, Thomas Pearce, Haohan Wang

TL;DR

The paper addresses the challenging problem of histology-to-organ registration by turning it into a learning task that localizes a 2D tissue slice within a 3D organ volume. It introduces ATOM, an automated pipeline that generates diverse 2D–3D image–position pairs from multi-institution data, and RegisMCAN, a deep modular co-attention model that treats the 2D slice as a question and the 3D volume as an answer, predicting the originating block via patch-level co-attention. Key contributions include the ATOM dataset, a tailored RegisMCAN architecture with patch embeddings and an asymmetric loss to handle imbalanced block labels, and extensive experiments across data types, labels, splits, and slice views. The work advances automated, end-to-end tissue-organ registration with potential to improve multimodal disease understanding and non-invasive radiology analyses, while highlighting practical considerations like resolution constraints and memory demands.

Abstract

Correlating neuropathology with neuroimaging findings provides a multiscale view of pathologic changes in the human organ spanning the meso- to micro-scales, and is an emerging methodology expected to shed light on numerous disease states. To gain the most information from this multimodal, multiscale approach, it is desirable to identify precisely where a histologic tissue section was taken from within the organ in order to correlate with the tissue features in exactly the same organ region. Histology-to-organ registration poses an extra challenge, as any given histologic section can capture only a small portion of a human organ. Making use of the capabilities of state-of-the-art deep learning models, we unlock the potential to address and solve such intricate challenges. Therefore, we create the ATOM benchmark dataset, sourced from diverse institutions, with the primary objective of transforming this challenge into a machine learning problem and delivering outstanding outcomes that enlighten the biomedical community. The performance of our RegisMCAN model demonstrates the potential of deep learning to accurately predict where a subregion extracted from an organ image was obtained from within the overall 3D volume. The code and dataset can be found at: https://github.com/haizailache999/Image-Registration/tree/main

From Tissue Plane to Organ World: A Benchmark Dataset for Multimodal Biomedical Image Registration using Deep Co-Attention Networks

TL;DR

The paper addresses the challenging problem of histology-to-organ registration by turning it into a learning task that localizes a 2D tissue slice within a 3D organ volume. It introduces ATOM, an automated pipeline that generates diverse 2D–3D image–position pairs from multi-institution data, and RegisMCAN, a deep modular co-attention model that treats the 2D slice as a question and the 3D volume as an answer, predicting the originating block via patch-level co-attention. Key contributions include the ATOM dataset, a tailored RegisMCAN architecture with patch embeddings and an asymmetric loss to handle imbalanced block labels, and extensive experiments across data types, labels, splits, and slice views. The work advances automated, end-to-end tissue-organ registration with potential to improve multimodal disease understanding and non-invasive radiology analyses, while highlighting practical considerations like resolution constraints and memory demands.

Abstract

Correlating neuropathology with neuroimaging findings provides a multiscale view of pathologic changes in the human organ spanning the meso- to micro-scales, and is an emerging methodology expected to shed light on numerous disease states. To gain the most information from this multimodal, multiscale approach, it is desirable to identify precisely where a histologic tissue section was taken from within the organ in order to correlate with the tissue features in exactly the same organ region. Histology-to-organ registration poses an extra challenge, as any given histologic section can capture only a small portion of a human organ. Making use of the capabilities of state-of-the-art deep learning models, we unlock the potential to address and solve such intricate challenges. Therefore, we create the ATOM benchmark dataset, sourced from diverse institutions, with the primary objective of transforming this challenge into a machine learning problem and delivering outstanding outcomes that enlighten the biomedical community. The performance of our RegisMCAN model demonstrates the potential of deep learning to accurately predict where a subregion extracted from an organ image was obtained from within the overall 3D volume. The code and dataset can be found at: https://github.com/haizailache999/Image-Registration/tree/main
Paper Structure (12 sections, 2 equations, 3 figures, 4 tables)

This paper contains 12 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. ATOM Dataset Preparation
  4. RegisMCAN
  5. Experiments
  6. Data Types Effect
  7. Data Labels Effect
  8. Dataset Splitting Effect
  9. Slice Gathering Effect
  10. Conclusions
  11. Appendix
  12. Discussions of ATOM and RegisMCAN

Figures (3)

  • Figure 1: Overview of ATOM curation pipeline. We collect relevant 3D histopathology volume data in Collect. For 2D Slices extraction, we randomly cut the 2D slices from different views (not only from axial, coronal, and sagittal views) within the collected 3D volume data and record the corresponding coordinates for each corner. In Position Calculation section, we first cut the 3D volume data into multiple small blocks, and then rely on a conventional calculation model to judge which block the 2D slice belongs to. Finally, relevant 3D volume Data, 2D cell slices and positions are paired up to form Data Pairs, which can yield ATOM dataset, a richly annotated image registration dataset for histopathology.
  • Figure 2: Visualization of the tissue-MRI registration problem.
  • Figure 3: Overview of RegisMCAN model. RegisMCAN consists of two layers. It will first use patch embedding layers to embed the whole 3D volume data and 2D tissue slices separately and use SA and GA units to realize the information extraction and relationship simulation. Here to distinguish the embedding of each separate prediction block, we drop the SA unit and only use one GA unit while realizing relationship simulation.