Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Surgical Scene Segmentation by Transformer With Asymmetric Feature Enhancement

Cheng Yuan, Yutong Ban

TL;DR

This work proposes a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE), which enhances local information and then actively fuses the improved feature pyramid into the embeddings from transformer encoders by a multiscale interaction attention strategy.

Abstract

Surgical scene segmentation is a fundamental task for robotic-assisted laparoscopic surgery understanding. It often contains various anatomical structures and surgical instruments, where similar local textures and fine-grained structures make the segmentation a difficult task. Vision-specific transformer method is a promising way for surgical scene understanding. However, there are still two main challenges. Firstly, the absence of inner-patch information fusion leads to poor segmentation performance. Secondly, the specific characteristics of anatomy and instruments are not specifically modeled. To tackle the above challenges, we propose a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE), which enhances local information and then actively fuses the improved feature pyramid into the embeddings from transformer encoders by a multi-scale interaction attention strategy. The proposed method outperforms the SOTA methods in several different surgical segmentation tasks and additionally proves its ability of fine-grained structure recognition. Code is available at https://github.com/cyuan-sjtu/ViT-asym.

Surgical Scene Segmentation by Transformer With Asymmetric Feature Enhancement

TL;DR

This work proposes a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE), which enhances local information and then actively fuses the improved feature pyramid into the embeddings from transformer encoders by a multiscale interaction attention strategy.

Abstract

Surgical scene segmentation is a fundamental task for robotic-assisted laparoscopic surgery understanding. It often contains various anatomical structures and surgical instruments, where similar local textures and fine-grained structures make the segmentation a difficult task. Vision-specific transformer method is a promising way for surgical scene understanding. However, there are still two main challenges. Firstly, the absence of inner-patch information fusion leads to poor segmentation performance. Secondly, the specific characteristics of anatomy and instruments are not specifically modeled. To tackle the above challenges, we propose a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE), which enhances local information and then actively fuses the improved feature pyramid into the embeddings from transformer encoders by a multi-scale interaction attention strategy. The proposed method outperforms the SOTA methods in several different surgical segmentation tasks and additionally proves its ability of fine-grained structure recognition. Code is available at https://github.com/cyuan-sjtu/ViT-asym.

Paper Structure

This paper contains 12 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Framework Overview
  4. Multi-scale Interaction Attention
  5. Asymmetric Feature Enhancement
  6. Experiments and Results
  7. Datasets
  8. Implementation Details
  9. Results and Analysis
  10. Conclusion
  11. Compliance with ethical standards
  12. Acknowledgments

Figures (3)

  • Figure 1: Challenges in surgical scene segmentation: (a) local feature similarity between covered kidney and kidney parenchyma; (b) fine-grained structure complexity of tubular instrument, such as thread.
  • Figure 2: The overall architecture of TAFE. It contains a transformer encoder-decoder backbone injected with the Multi-scale Interaction Attention (MIA) branch and the Asymmetric Feature Enhancement (AFE) module.
  • Figure 3: Visual comparison of the fine-grained structure recognition ability of different methods.