DBAT: Dynamic Backward Attention Transformer for Material Segmentation with Cross-Resolution Patches

TL;DR

This work tackles dense material segmentation, where material appearance varies across scenes, by introducing Dynamic Backward Attention Transformer (DBAT), a single-branch transformer that learns from cross-resolution image patches. DBAT uses a backbone encoder with patch merging, a Dynamic Backward Attention (DBA) module to fuse cross-resolution features via per-pixel attention, and a feature merging module to inject aggregated information back into the backbone before decoding. It achieves state-of-the-art real-time Pixel Acc on the Local Material Database (86.85%) and strong performance on OpenSurfaces, while offering interpretability through Centered Kernel Alignment (CKA) heatmaps and network dissection, which reveal a focus on texture and material-specific features and robustness to initialization. The work demonstrates that cross-resolution patch features can improve material segmentation, provides detailed analyses of attention behavior and feature alignment, and suggests future directions toward densely labeled or synthetic datasets to further enhance interpretability and reliability.

Abstract

The objective of dense material segmentation is to identify the material categories for every image pixel. Recent studies adopt image patches to extract material features. Although the trained networks can improve the segmentation performance, their methods choose a fixed patch resolution which fails to take into account the variation in pixel area covered by each material. In this paper, we propose the Dynamic Backward Attention Transformer (DBAT) to aggregate cross-resolution features. The DBAT takes cropped image patches as input and gradually increases the patch resolution by merging adjacent patches at each transformer stage, instead of fixing the patch resolution during training. We explicitly gather the intermediate features extracted from cross-resolution patches and merge them dynamically with predicted attention masks. Experiments show that our DBAT achieves an accuracy of 86.85%, which is the best performance among state-of-the-art real-time models. Like other successful deep learning solutions with complex architectures, the DBAT also suffers from lack of interpretability. To address this problem, this paper examines the properties that the DBAT makes use of. By analysing the cross-resolution features and the attention weights, this paper interprets how the DBAT learns from image patches. We further align features to semantic labels, performing network dissection, to infer that the proposed model can extract material-related features better than other methods. We show that the DBAT model is more robust to network initialisation, and yields fewer variable predictions compared to other models. The project code is available at https://github.com/heng-yuwen/Dynamic-Backward-Attention-Transformer.

Figures (11)

• Figure 1: The architecture of the Dynamic Backward Attention Transformer (DBAT). It comprises three primary components: an encoder backbone that generates cross-resolution features by merging adjacent patches at each transformer stage, a backward attention module for aggregating these features, and a feature merging module leveraging a residual connection to learn complementary features.
• Figure 2: Structure of the DBA module. It is formulated to execute a weighted sum across the feature maps, $Map_{1,2,3,4}$, producing the aggregated features. These weights are dynamically predicted in accordance with the input image through the attention module, which utilises the fourth feature map, $Map_4$, as its input.
• Figure 3: The feature merging module. This module integrates pertinent cross-resolution information from the aggregated patch feature into $Map_4$ by employing the window attention mechanism and a residual connection.
• Figure 4: Boxplot of the performance on the LMD across five runs.
• Figure 5: Boxplot of the performance on the OpenSurfaces across five runs.