Unprejudiced Training Auxiliary Tasks Makes Primary Better: A Multi-Task Learning Perspective

TL;DR

The paper tackles the problem that auxiliary tasks in multi-task learning are often under-trained, limiting their benefit to the primary task. It proposes Impartial Auxiliary Learning (IAL), a two-stage framework that uses uncertainty-based weights to train decoders impartially and gradient-normalization coupled with a task-uncertainty–driven function to balance encoder training in favor of the primary task. Through extensive experiments on NYUv2, Cityscapes, Pascal Context, and a multi-domain CIFAR-100 setting, IAL achieves state-of-the-art or competitive results and demonstrates robustness to noisy pseudo-tasks. The method also includes thorough ablations and analyses of balancing strategies, and highlights its potential applicability to large pre-trained model–driven auxiliary tasks. Overall, IAL advances practical multi-task learning by ensuring high-quality training of auxiliary tasks to bolster the primary task while maintaining stability and efficiency.

Abstract

Human beings can leverage knowledge from relative tasks to improve learning on a primary task. Similarly, multi-task learning methods suggest using auxiliary tasks to enhance a neural network's performance on a specific primary task. However, previous methods often select auxiliary tasks carefully but treat them as secondary during training. The weights assigned to auxiliary losses are typically smaller than the primary loss weight, leading to insufficient training on auxiliary tasks and ultimately failing to support the main task effectively. To address this issue, we propose an uncertainty-based impartial learning method that ensures balanced training across all tasks. Additionally, we consider both gradients and uncertainty information during backpropagation to further improve performance on the primary task. Extensive experiments show that our method achieves performance comparable to or better than state-of-the-art approaches. Moreover, our weighting strategy is effective and robust in enhancing the performance of the primary task regardless the noise auxiliary tasks' pseudo labels.

Figures (14)

• Figure 1: Visual comparison between our impartial training framework and OL-AUX ol_aux. By sufficiently training the auxiliary task (disparity estimation), our method achieves noticeably better results, as evidenced by the visualizations on the bottom row. Furthermore, we observe that by leveraging the knowledge provided by better auxiliary task, the predictions of the primary task (semantic segmentation) are also improved.
• Figure 2: An comparison between our equally training framework and previous methods. (a) Baseline methods, use a weight estimator to produce auxiliary weights according to their gradient norms, directions or loss magnitudes. All parameters are trained under one group of weights, and the auxiliary tasks are always set to lower weights leading to poor training problem. (b) An overview of our framework. For the decoder stage (parameters surrounding by orange dash lines), we update the task-specific decoders with uncertainty weights uw which are changed only according to their corresponding losses. In this stage, the uncertainty $\sigma_t$ is estimated automatically. For the encoder stage (shared encoder surrounding by blue dash lines), we use both gradients and uncertainty information to weight the auxiliary tasks. More details will be reported on Sec. \ref{['sec:details']}.
• Figure 3: Visualization on NYUv2 nyuv2 with semantic segmentation as the primary task and other two tasks (depth estimation and normal prediction) as auxiliary. The impressive improvements are marked with a purple box.
• Figure 4: Visualization on NYUv2 nyuv2 with depth estimation as the primary task and other two tasks (semantic segmentation and normal prediction) as auxiliary. The impressive improvements are marked with a purple box.
• Figure 5: Visualization on NYUv2 nyuv2 with surface normal prediction as the primary task and other two tasks (semantic segmentation and depth estimation) as auxiliary. The impressive improvements are marked with a purple box.