Deep spatial context: when attention-based models meet spatial regression

TL;DR

It turns out that the larger the size of the neighborhood taken into account within spatial regression, the less valuable contextual information is.

Abstract

We propose 'Deep spatial context' (DSCon) method, which serves for investigation of the attention-based vision models using the concept of spatial context. It was inspired by histopathologists, however, the method can be applied to various domains. The DSCon allows for a quantitative measure of the spatial context's role using three Spatial Context Measures: $SCM_{features}$, $SCM_{targets}$, $SCM_{residuals}$ to distinguish whether the spatial context is observable within the features of neighboring regions, their target values (attention scores) or residuals, respectively. It is achieved by integrating spatial regression into the pipeline. The DSCon helps to verify research questions. The experiments reveal that spatial relationships are much bigger in the case of the classification of tumor lesions than normal tissues. Moreover, it turns out that the larger the size of the neighborhood taken into account within spatial regression, the less valuable contextual information is. Furthermore, it is observed that the spatial context measure is the largest when considered within the feature space as opposed to the targets and residuals.

Figures (8)

• Figure 1: Simplified scheme of the proposed 'Deep spatial context' method where information extracted from the attention-based model is used within spatial regression models to assess and measure the importance of spatial context.
• Figure 2: Critical distance plotof models' accuracy of classification of normal tissue vs. tumor lesion. For brevity: s denotes Swin, sv2 - Swin v2 and w - window. In the labels, the mean accuracy (percentage) and the standard deviation over 5 trained models during the cross-validation are provided. The fact that the horizontal line below the ranking axis connects all the models means that there is no significant difference in their performance based on the Wilcoxon-Holm method used to detect pairwise significance.
• Figure 3: Stacked kernel distribution estimation plots where the proportion of values from normal tissues and tumor lesions for a given spatial context measure is shown. The results are for the sv2_base_w8 model with k=24 nearest neighbors within the spatial weights matrix. The appearance of the plots is representative of other model-$k$ combinations. The differences between the models are summarized in Table \ref{['tab:ranking']}.
• Figure 4: The mean value of spatial context measures for models with different feature extractors with respect to the size of neighborhood in the spatial weights matrix within spatial regression models over all images with tumor.
• Figure 5: Mean of $SCM_{features}$ computed on all tumor images with all model-$k$ combinations where $k=99$ within $W$ is used. The images are characterized by the measure of tumor spread.