Deformable Convolutional Networks

TL;DR

The paper tackles CNNs' difficulty in modeling geometric transformations by introducing deformable convolution and deformable RoI pooling. These modules learn dense spatial offsets from data, enabling adaptive receptive fields and deformable sampling that can replace standard components in existing architectures. Across semantic segmentation and object detection benchmarks (VOC, CityScapes, COCO), the approach yields meaningful accuracy gains with only modest computational overhead. Ablation studies confirm that learned offsets correlate with object size and that stacking deformable layers enhances performance, validating the method's effectiveness for complex visual tasks.

Abstract

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation. The code would be released.

Figures (8)

• Figure 1: Illustration of the sampling locations in $3\times 3$ standard and deformable convolutions. (a) regular sampling grid (green points) of standard convolution. (b) deformed sampling locations (dark blue points) with augmented offsets (light blue arrows) in deformable convolution. (c)(d) are special cases of (b), showing that the deformable convolution generalizes various transformations for scale, (anisotropic) aspect ratio and rotation.
• Figure 2: Illustration of $3\times 3$ deformable convolution.
• Figure 3: Illustration of $3\times 3$ deformable RoI pooling.
• Figure 4: Illustration of $3\times 3$ deformable PS RoI pooling.
• Figure 5: Illustration of the fixed receptive field in standard convolution (a) and the adaptive receptive field in deformable convolution (b), using two layers. Top: two activation units on the top feature map, on two objects of different scales and shapes. The activation is from a $3\times 3$ filter. Middle: the sampling locations of the $3\times 3$ filter on the preceding feature map. Another two activation units are highlighted. Bottom: the sampling locations of two levels of $3\times 3$ filters on the preceding feature map. Two sets of locations are highlighted, corresponding to the highlighted units above.