Hierarchical Salient Patch Identification for Interpretable Fundus Disease Localization

TL;DR

A weakly supervised interpretable fundus disease localization method called hierarchical salient patch identification (HSPI) is proposed that can achieve interpretable disease localization using only image-level labels and a neural network classifier (NNC).

Abstract

With the widespread application of deep learning technology in medical image analysis, the effective explanation of model predictions and improvement of diagnostic accuracy have become urgent problems that need to be solved. Attribution methods have become key tools to help doctors better understand the diagnostic basis of models, and are used to explain and localize diseases in medical images. However, previous methods suffer from inaccurate and incomplete localization problems for fundus diseases with complex and diverse structures. To solve these problems, we propose a weakly supervised interpretable fundus disease localization method called hierarchical salient patch identification (HSPI) that can achieve interpretable disease localization using only image-level labels and a neural network classifier (NNC). First, we propose salient patch identification (SPI), which divides the image into several patches and optimizes consistency loss to identify which patch in the input image is most important for the network's prediction, in order to locate the disease. Second, we propose a hierarchical identification strategy to force SPI to analyze the importance of different areas to neural network classifier's prediction to comprehensively locate disease areas. Conditional peak focusing is then introduced to ensure that the mask vector can accurately locate the disease area. Finally, we propose patch selection based on multi-sized intersections to filter out incorrectly or additionally identified non-disease regions. We conduct disease localization experiments on fundus image datasets and achieve the best performance on multiple evaluation metrics compared to previous interpretable attribution methods. Additional ablation studies are conducted to verify the effectiveness of each method.

Figures (5)

• Figure 1: The principle of weakly supervised interpretable fundus disease localization using SPI. Question 1: Which category does the NNC predict for the input image? Question 2: Which region contributes most to the NNC prediction? NNC and SPI can respectively answer Question 1 and Question 2. When the NNC predicts that the input image is diseased, the region that contributes most to the NNC prediction is identified as the diseased area. SPI uses a mask vector to divide the input image into multiple patches to learn the patch that contributes most to the NNC prediction.
• Figure 2: The overall architecture of HSPI. In Stage 1, the original input image and mask vector are fed into SPI and learned by computing the consistency loss (CL) to produce candidate image and trained vector for NNC prediction. Then, process the mask vector, trained vector, and candidate image from Stage 1 to obtain the input image and mask vector for Stage 2. In Stages 2, 3, and so on, repeat the above process. HSPI finally identifies several candidate patches (yellow patches) for NNC prediction.
• Figure 3: An example of training mask vector in SPI to identify disease area for prediction. The result of SPI training mask vector, as well as the corresponding training period (Epoch), similarity loss $L_{s}$, mask loss $L_{m}$, and maximum-second difference $D_{n}$.
• Figure 4: An example of HSPI identifying disease areas (yellow patches) for prediction. It shows the results of HSPI training mask vectors using SPI at different stages, as well as the corresponding training period (Epoch), similarity loss $L_{s}$, mask loss $L_{m}$, and maximum-second difference $D_{n}$.
• Figure 5: Localization results for ResNet50 using different attribution methods. The saliency maps are visualized using JET colormaps.