Revisiting Context Aggregation for Image Matting

TL;DR

This paper investigates context aggregation in image matting and finds that conventional context-aggregation modules are sensitive to scale mismatches between training and inference. Through systematic experiments, it shows that a basic encoder-decoder architecture can learn universal context aggregation, especially when trained with large image patches and expanded receptive fields. The authors propose AEMatter, a simple matting network with a Hybrid-Transformer encoder and appearance-enhanced axis-wise learning (AEAL) blocks, plus a Swin-based decoder and large-patch training. Extensive experiments on five matting datasets demonstrate that AEMatter delivers state-of-the-art performance and strong generalization, representing a practical advance for matting applications.

Abstract

Traditional studies emphasize the significance of context information in improving matting performance. Consequently, deep learning-based matting methods delve into designing pooling or affinity-based context aggregation modules to achieve superior results. However, these modules cannot well handle the context scale shift caused by the difference in image size during training and inference, resulting in matting performance degradation. In this paper, we revisit the context aggregation mechanisms of matting networks and find that a basic encoder-decoder network without any context aggregation modules can actually learn more universal context aggregation, thereby achieving higher matting performance compared to existing methods. Building on this insight, we present AEMatter, a matting network that is straightforward yet very effective. AEMatter adopts a Hybrid-Transformer backbone with appearance-enhanced axis-wise learning (AEAL) blocks to build a basic network with strong context aggregation learning capability. Furthermore, AEMatter leverages a large image training strategy to assist the network in learning context aggregation from data. Extensive experiments on five popular matting datasets demonstrate that the proposed AEMatter outperforms state-of-the-art matting methods by a large margin.

Figures (6)

• Figure 1: Illustration of a basic matting network and context aggregation modules. (a) The basic matting network uses an encoder to extract context features from inputs, and a decoder to predict alpha mattes. Our AEMatter also follows this scheme. (b) Pooling-based context aggregation module uses pooling operations to aggregate contexts from surrounding regions. (c) Affinity-based context aggregation module uses affinity operations to aggregate contexts from globally related regions.
• Figure 2: Inference Patch Size vs Prediction Errors. As the inference patch size increases, the prediction errors of the compared matting methods first decrease and then show different trends.
• Figure 3: Trimap Dilation Distance vs Prediction Error. Note that, * denotes the network does not incorporate context aggregation modules. As the trimap dilation distance increases, the prediction errors (MSE) of all compared matting methods increase.
• Figure 4: Visualization of the receptive field of matting networks trained on image patches of different sizes. (a) Untrained network. (b) Network trained on $256 \times 256$ patches. (c) Network trained on $512 \times 512$ patches. (d) Network trained on $768 \times 768$ patches. (e) Network trained on $1024 \times 1024$ patches.
• Figure 5: Overview of AEMatter. The encoder adopts a Hybrid-Transformer backbone with appearance-enhanced axis-wise learning blocks to extract context features. The decoder adopts Swin blocks to refine the context features and estimate the alpha matte.