Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MTLSI-Net: A Linear Semantic Interaction Network for Parameter-Efficient Multi-Task Dense Prediction

Chen Liu, Hengyu Man, Xiaopeng Fan, Debin Zhao

Abstract

Multi-task dense prediction aims to perform multiple pixel-level tasks simultaneously. However, capturing global cross-task interactions remains non-trivial due to the quadratic complexity of standard self-attention on high-resolution features. To address this limitation, we propose a Multi-Task Linear Semantic Interaction Network (MTLSI-Net), which facilitates cross-task interaction through linear attention. Specifically, MTLSI-Net incorporates three key components: a Multi-Task Multi-scale Query Linear Fusion Block, which captures cross-task dependencies across multiple scales with linear complexity using a shared global context matrix; a Semantic Token Distiller that compresses redundant features into compact semantic tokens, distilling essential cross-task knowledge; and a Cross-Window Integrated attention Block that injects global semantics into local features via a dual-branch architecture, preserving both global consistency and spatial precision. These components collectively enable the network to capture comprehensive cross-task interactions at linear complexity with reduced parameters. Extensive experiments on NYUDv2 and PASCAL-Context demonstrate that MTLSI-Net achieves state-of-the-art performance, validating its effectiveness and efficiency in multi-task learning.

MTLSI-Net: A Linear Semantic Interaction Network for Parameter-Efficient Multi-Task Dense Prediction

Abstract

Multi-task dense prediction aims to perform multiple pixel-level tasks simultaneously. However, capturing global cross-task interactions remains non-trivial due to the quadratic complexity of standard self-attention on high-resolution features. To address this limitation, we propose a Multi-Task Linear Semantic Interaction Network (MTLSI-Net), which facilitates cross-task interaction through linear attention. Specifically, MTLSI-Net incorporates three key components: a Multi-Task Multi-scale Query Linear Fusion Block, which captures cross-task dependencies across multiple scales with linear complexity using a shared global context matrix; a Semantic Token Distiller that compresses redundant features into compact semantic tokens, distilling essential cross-task knowledge; and a Cross-Window Integrated attention Block that injects global semantics into local features via a dual-branch architecture, preserving both global consistency and spatial precision. These components collectively enable the network to capture comprehensive cross-task interactions at linear complexity with reduced parameters. Extensive experiments on NYUDv2 and PASCAL-Context demonstrate that MTLSI-Net achieves state-of-the-art performance, validating its effectiveness and efficiency in multi-task learning.

Paper Structure

This paper contains 13 sections, 11 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Framework Overview
  4. Multi-task Multi-scale Query Linear Fusion Block
  5. Semantic Token Distiller
  6. Cross-Window Integrated attention Block
  7. Experiments
  8. Experimental Setup
  9. Comparison with State-of-the-art Methods
  10. Effect of token count $K$ in Semantic Token Distiller
  11. Ablation of multi-scale queries in MT-MQLFB
  12. Qualitative Study
  13. Conclusion

Figures (4)

  • Figure 1: Overview of the proposed coarse-to-fine multi-task learning framework. Given an input image, a shared backbone first extracts multi-scale features. Subsequently, preliminary decoders generate initial task-specific representations and coarse predictions. Next, to enable cross-task interaction, the MT-MQLFB processes the concatenated features and predictions. Finally, semantic decoders, incorporating the Semantic Token Distiller and Cross-Window Integrated Attention Block (CWIB), refine these features to produce fine-grained predictions for all tasks.
  • Figure 2: Architecture of the Multi-task Multi-scale Query Linear Fusion Block.
  • Figure 3: Ablation study on the number of semantic tokens $K$ on NYUDv2.
  • Figure 4: Visual comparison on the NYUDv2 dataset.