LDConv: Linear deformable convolution for improving convolutional neural networks

TL;DR

LDConv introduces Linear Deformable Convolution, a plug-and-play operation that decouples kernel size and sampling shape from quadratic parameter growth by generating initial arbitrary sampling coordinates and learning per-position offsets to deform the sampling grid. This enables irregular, arbitrarily shaped kernels with linear growth in parameters, and it supports multiple feature-extraction strategies to realize irregular convolutions. Extensive object-detection experiments on COCO2017, VOC 7+12, and VisDrone-DET2021 demonstrate improved accuracy with controllable overhead when replacing standard convolutions in backbone blocks and detection heads, outperforming or matching Deformable Conv while offering broader shape flexibility. The work highlights practical benefits for hardware-aware model design and opens avenues for task-specific shape design, with code available for reproducibility.

Abstract

Neural networks based on convolutional operations have achieved remarkable results in the field of deep learning, but there are two inherent flaws in standard convolutional operations. On the one hand, the convolution operation is confined to a local window, so it cannot capture information from other locations, and its sampled shapes is fixed. On the other hand, the size of the convolutional kernel are fixed to k $\times$ k, which is a fixed square shape, and the number of parameters tends to grow squarely with size. Although Deformable Convolution (Deformable Conv) address the problem of fixed sampling of standard convolutions, the number of parameters also tends to grow in a squared manner. In response to the above questions, the Linear Deformable Convolution (LDConv) is explored in this work, which gives the convolution kernel an arbitrary number of parameters and arbitrary sampled shapes to provide richer options for the trade-off between network overhead and performance. In LDConv, a novel coordinate generation algorithm is defined to generate different initial sampled positions for convolutional kernels of arbitrary size. To adapt to changing targets, offsets are introduced to adjust the shape of the samples at each position. LDConv corrects the growth trend of the number of parameters for standard convolution and Deformable Conv to a linear growth. Moreover, it completes the process of efficient feature extraction by irregular convolutional operations and brings more exploration options for convolutional sampled shapes. Object detection experiments on representative datasets COCO2017, VOC 7+12, and VisDrone-DET2021 fully demonstrate the advantages of LDConv. LDConv is a plug-and-play convolutional operation that can replace the convolutional operation to improve network performance. The code for the relevant tasks can be found at https://github.com/CV-ZhangXin/LDConv.

Figures (9)

• Figure 1: The initial sampled coordinates for arbitrary convolutional kernel sizes are generated by a generation algorithm. It provides initial sampled shapes for irregular convolution kernel sizes.
• Figure 2: It shows a detailed schematic of the structure of LDConv. It assigns initial sampled coordinates to a convolution of arbitrary size and adjusts the sample shape with the learnable offsets. Compared to the original sampled shape, the sampled shape at each position is changed by resampling.
• Figure 3: Three methods to extract features corresponding to the irregular convolutional kernels. The features boxed in red and black represent the convolutional kernel sampling features on the different channels. (a) represents the Conv3d to complete it. (b) displays the 1 $\times$ 1 Conv2d to finish it. (c) shows the column Conv2d to solve it.
• Figure 4: It shows the initial sampled shape of size 5. LDConv can achieve arbitrary sampled shapes by designing different initial sampled shapes.
• Figure 5: It shows the ImprovedFasterBlock. The LDConv (N = 2) is used to replace two 1 $\times$ 1 convolutional operations in FasterBlock.