UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception

TL;DR

UV-M3TL introduces a unified multimodal multi-task learning framework for ADAS perception that jointly models driver state and traffic context. It couples a MARNet+3D-CNN-based encoder with DB-SCME to separate task-shared and task-specific features, and a novel AFD-Loss that dynamically balances task learning and decouples shared vs. task-specific representations. The approach achieves state-of-the-art results on AIDE across four tasks and demonstrates strong generalization to NYUD-v2, PASCAL-Context, BDD100K, and CityScapes, supported by comprehensive ablations. This work provides a versatile baseline for multimodal MTL in driving perception, with implications for more holistic and robust ADAS systems.

Abstract

Advanced Driver Assistance Systems (ADAS) need to understand human driver behavior while perceiving their navigation context, but jointly learning these heterogeneous tasks would cause inter-task negative transfer and impair system performance. Here, we propose a Unified and Versatile Multimodal Multi-Task Learning (UV-M3TL) framework to simultaneously recognize driver behavior, driver emotion, vehicle behavior, and traffic context, while mitigating inter-task negative transfer. Our framework incorporates two core components: dual-branch spatial channel multimodal embedding (DB-SCME) and adaptive feature-decoupled multi-task loss (AFD-Loss). DB-SCME enhances cross-task knowledge transfer while mitigating task conflicts by employing a dual-branch structure to explicitly model salient task-shared and task-specific features. AFD-Loss improves the stability of joint optimization while guiding the model to learn diverse multi-task representations by introducing an adaptive weighting mechanism based on learning dynamics and feature decoupling constraints. We evaluate our method on the AIDE dataset, and the experimental results demonstrate that UV-M3TL achieves state-of-the-art performance across all four tasks. To further prove the versatility, we evaluate UV-M3TL on additional public multi-task perception benchmarks (BDD100K, CityScapes, NYUD-v2, and PASCAL-Context), where it consistently delivers strong performance across diverse task combinations, attaining state-of-the-art results on most tasks.

Figures (12)

• Figure 1: Traffic context and driver states interaction. Tasks (a), (b), (c), and (d) represent traffic context recognition, vehicle behavior recognition, driver behavior recognition, and driver emotion recognition, respectively. These tasks comprehensively demonstrate the complex and closely interconnected relationships between the driver and traffic.
• Figure 2: Comparison results of different methods on five benchmark datasets. (a)–(e) present the performance comparisons between our method and various state-of-the-art approaches on the AIDE, PASCAL-Context, BDD100K, NYUD-v2, and CityScapes datasets, respectively. To facilitate a more intuitive comparison of performance across different methods, for evaluation metrics where lower values indicate better performance—such as mErr, RMSE, Absolute Error (Abs), and Relative Error (Rel)—we take their negative values (denoted by “-” in the figure), so that all metrics follow a unified convention where higher values correspond to better performance. Detailed Comparison results of all methods are reported in Section \ref{['Experiments']}.
• Figure 3: The overall pipeline of UV-M$^3$TL with three primary components: Multimodal Encoder, DB-SCME, and AFD-Loss.
• Figure 4: Encoder architecture of UV-M$^3$TL with the single-modality input, where multi-level features are extracted from different network layers.
• Figure 5: The flowchart of the MARNet architecture, including the processes for horizontal-vertical attention and region attention.