An efficient dual-branch framework via implicit self-texture enhancement for arbitrary-scale histopathology image super-resolution

TL;DR

This work proposes an Implicit Self-Texture Enhancement-based dual-branch framework (ISTE) for arbitrary-scale super-resolution (SR) of histopathology images to address the challenge of unique fine-grained image textures different from natural images.

Abstract

High-quality whole-slide scanning is expensive, complex, and time-consuming, thus limiting the acquisition and utilization of high-resolution histopathology images in daily clinical work. Deep learning-based single-image super-resolution (SISR) techniques provide an effective way to solve this problem. However, the existing SISR models applied in histopathology images can only work in fixed integer scaling factors, decreasing their applicability. Though methods based on implicit neural representation (INR) have shown promising results in arbitrary-scale super-resolution (SR) of natural images, applying them directly to histopathology images is inadequate because they have unique fine-grained image textures different from natural images. Thus, we propose an Implicit Self-Texture Enhancement-based dual-branch framework (ISTE) for arbitrary-scale SR of histopathology images to address this challenge. The proposed ISTE contains a feature aggregation branch and a texture learning branch. We employ the feature aggregation branch to enhance the learning of the local details for SR images while utilizing the texture learning branch to enhance the learning of high-frequency texture details. Then, we design a two-stage texture enhancement strategy to fuse the features from the two branches to obtain the SR images. Experiments on publicly available datasets, including TMA, HistoSR, and the TCGA lung cancer datasets, demonstrate that ISTE outperforms existing fixed-scale and arbitrary-scale SR algorithms across various scaling factors. Additionally, extensive experiments have shown that the histopathology images reconstructed by the proposed ISTE are applicable to downstream pathology image analysis tasks.

Figures (10)

• Figure 1: Motivation of our ISTE. (a) Existing SR methods for histopathology images upadhyay2019mixedmukherjee2018convolutionaljuhong2023superchen2020jointshahidi2021breastli2021singlewu2023mmsrnet can only achieve fixed integer-scale SR and need to retrain the model to achieve different scaling factors; (b) Existing SR algorithms based on implicit neural networks for natural images (exemplified by LIIF chen2021learning) perform SR directly in the spatial domain, and lack attention and enhancement of image texture information; (c) ISTE is an efficient dual-branch framework based on implicit self-texture enhancement for arbitrary-scale histopathology image SR. ISTE further enhances its performance through feature-based and spatial domain-based texture enhancement; (d) We use the Canny operator canny1986computational to extract texture from both natural and histopathology images. It is evident that, in contrast to natural images, histopathology images contain a large amount of fine-grained cell morphology and arrangement information, and they tend to have richer texture information.
• Figure 2: Workflow of our ISTE. The LR image $X_{LR}$ is input into the encoder to get the pre-extracted feature map $F_{LR}$ first. In the feature aggregation branch, we input the feature $F_{LR}$ into the local feature interactor and a convolutional layer to obtain $F_{LFIC}$. In the texture learning branch, we input the feature $F_{LR}$ into the texture learner to obtain the texture feature $F_{TL}$. Then the feature maps from the two branches are input to the self-texture fusion module to accomplish feature-based enhancement. Finally, the enhanced feature $F_{STF}$ output from the STF module and the texture feature $F_{TL}$ output from the texture learner are decoded into RGB values respectively, and added up to accomplish spatial domain-based texture enhancement.
• Figure 3: Local feature interactor.
• Figure 4: (a) Texture learner; (b) Self-texture fusion module; (c) Coordinate diagram of $F_{STF}$ and $F_{TL}$ for the local pixel decoder and local texture decoder.
• Figure 5: Visual comparison with error maps of different methods on the TMA, HistoSR, and TCGA datasets. The error map represents the absolute error value between the output result and the ground truth. The brighter the color, the greater the error.