RobuMTL: Enhancing Multi-Task Learning Robustness Against Weather Conditions

TL;DR

This work tackles the problem of robustness in multi-task learning under adverse weather by introducing RobuMTL, a parameter-efficient framework that adaptively selects and aggregates perturbation-specific hierarchical LoRA experts. A lightweight Dynamic Modular LoRA Selector (DMLS) classifies input perturbations and informs a top-K fusion mechanism, Multiple Expert Parameter Fusion (MEPF), to modulate the backbone with the appropriate experts. Two training/inference regimes are proposed: RobuMTL and RobuMTL+ (the latter including per-perturbation adaptation squads), augmented with a consistency loss to stabilize predictions across clean and perturbed inputs. Empirical results on PASCAL and NYUD-v2 show notable robustness gains under single and mixed perturbations, with up to +44.4% improvement in adverse conditions and maintained or improved clean-data performance, while achieving reduced parameters and competitive FPS. This approach offers a scalable, real-time capable solution for perturbation-aware, multi-task perception in autonomous systems.

Abstract

Robust Multi-Task Learning (MTL) is crucial for autonomous systems operating in real-world environments, where adverse weather conditions can severely degrade model performance and reliability. In this paper, we introduce RobuMTL, a novel architecture designed to adaptively address visual degradation by dynamically selecting task-specific hierarchical Low-Rank Adaptation (LoRA) modules and a LoRA expert squad based on input perturbations in a mixture-of-experts fashion. Our framework enables adaptive specialization based on input characteristics, improving robustness across diverse real-world conditions. To validate our approach, we evaluated it on the PASCAL and NYUD-v2 datasets and compared it against single-task models, standard MTL baselines, and state-of-the-art methods. On the PASCAL benchmark, RobuMTL delivers a +2.8% average relative improvement under single perturbations and up to +44.4% under mixed weather conditions compared to the MTL baseline. On NYUD-v2, RobuMTL achieves a +9.7% average relative improvement across tasks. The code is available at GitHub.

Figures (8)

• Figure 1: In our pipeline, the input image is first processed by DMLS to estimate contribution scores based on its characteristics, and MEPF then uses these scores to selectively aggregate the relevant LoRA experts and update the MTL weights.The number of aggregated experts is at most equal to the number of perturbations.
• Figure 2: DMLS: Dynamic Modular LoRA Selector.
• Figure 3: Illustration of SE contribution within DMLS.
• Figure 4: DMLS expert activations under single perturbations show diagonal dominance(high activation) for the matching expert and minimal activation for others.
• Figure 5: Performance comparison of RobuMTL+ against the baseline and MTLoRA (LoRA rank 16) across multiple datasets. The symbols C, S, R, F, N, B represent the clean, snow, rain, fog, noise, and blur datasets, respectively.