DeGMix: Efficient Multi-Task Dense Prediction with Deformable and Gating Mixer

TL;DR

DeGMix addresses efficient multi-task dense prediction by combining deformable CNNs with a query-based Transformer, enabling simultaneous learning across semantic segmentation, depth, normal, boundary, and related tasks. The approach introduces a deformable mixer encoder and a task-aware gating transformer decoder with a shared spatial gating layer, enabling task-specific feature selection and cross-task interactions. Empirical results on NYUD-v2, PASCAL-Context, and Cityscapes demonstrate superior accuracy with reduced parameter counts compared with CNN- and Transformer-based baselines, and show robust generalization to unseen tasks. The method offers a practical, scalable solution for dense, multi-task understanding in real-world scenes.

Abstract

Convolution neural networks and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL). Existing studies typically employ either CNNs (effectively capture local spatial patterns) or Transformers (capturing long-range dependencies) independently, but integrating their strengths may yield more robust models. In this work, we present an efficient MTL model that combines the adaptive capabilities of deformable CNN and query-based Transformer with shared gating for MTL of dense prediction. This combination may offer a simple and efficient solution owing to its powerful and flexible task-specific learning and the advantages of lower cost, less complexity, and smaller parameters than traditional MTL methods. We introduce an efficient multi-task dense prediction with deformable and gating mixer (DeGMix). First, the deformable mixer encoder contains two types of operators: the channel-aware mixing operator leveraged to allow communication among different channels, and the spatial-aware deformable operator with deformable convolution applied to efficiently sample more informative spatial locations. Second, the task-aware gating transformer decoder is used to perform task-specific predictions, in which task interaction block integrated with self-attention is applied to capture task interaction features, and the task query block integrated with gating attention is leveraged to dynamically select the corresponding task-specific features. Furthermore, the results of the experiment demonstrate that the proposed DeGMix uses fewer GFLOPs and significantly outperforms current Transformer-based and CNN-based competitive models on a variety of metrics on three dense prediction datasets. Our code and models are available at https://github.com/yangyangxu0/DeMTG.

Figures (8)

• Figure 1: Accuracy-to-Parameter trade-off on NYUD-v2 dataset. Our DeGMix model can outperform existing MTL methods by a large margin while maintaining an optimal balance between accuracy and model parameters.
• Figure 2: Illustration of our feature extractor. Our feature extractor can be compatible with three kinds of backbone networks (i.e., HRNet, Swin-Transformer (Swin) and Vision Transformer (ViT)). Note that when ViT is used as backbone, we select four block-specific features. We conduct the scale number ablation study in Table \ref{['tab:scales']}. In this figure, we set the feature map channel $15c=C$.
• Figure 3: An overview of our model jointly handles multiple tasks with a unified encoder-decoder architecture. The DeGMix model consists of the deformable mixer encoder and task-aware gating transformer decoder. The core blocks of our DeGMix are (a) the deformable mixer (DM) that generates task-relevant deformable features, (b) the task interaction block that performs the interactions among tasks, and (c) the task query gating block that can dynamically select the task-aware features. The depth $d$ refers to the number of repetitions of the Deformable Mixer (see ablation on $d$ in Table \ref{['tab:depth']}).
• Figure 4: Illustration of our Deformable Mixer Encoder. For simplicity, we assume there is only one task ($T = 1$) in this figure.
• Figure 5: Illustration of our Task-aware Gating Transformer Decoder. The $dim$ denotes the input dimension. Conv1d denotes 1D convolution operation. For simplicity, we assume there is two tasks in the task interaction block and one task in the task query gating block.