RepVF: A Unified Vector Fields Representation for Multi-task 3D Perception

TL;DR

This work tackles the inefficiencies of conventional multi-task learning in autonomous driving 3D perception by introducing RepVF, a task-agnostic vector-field representation that unifies diverse targets (e.g., 3D objects and lanes) within a single geometric framework. Building on RepVF, the authors propose RFTR, a single-head transformer-based architecture that uses set-level perception queries to jointly predict unified vector fields and then convert them differentiably into task-specific outputs, eliminating task-specific heads and reducing gradient conflicts. The model is trained using existing labels by differentiable conversions and evaluated on a fusion of Waymo Open Dataset with OpenLane lane labels, demonstrating strong 3D lane and competitive 3D object detection performance while achieving improved gradient balance and computational efficiency. Overall, RepVF and RFTR offer a principled, efficient path toward true multi-task perception in autonomous driving, with practical impact in reducing model complexity and enhancing convergence across tasks.

Abstract

Concurrent processing of multiple autonomous driving 3D perception tasks within the same spatiotemporal scene poses a significant challenge, in particular due to the computational inefficiencies and feature competition between tasks when using traditional multi-task learning approaches. This paper addresses these issues by proposing a novel unified representation, RepVF, which harmonizes the representation of various perception tasks such as 3D object detection and 3D lane detection within a single framework. RepVF characterizes the structure of different targets in the scene through a vector field, enabling a single-head, multi-task learning model that significantly reduces computational redundancy and feature competition. Building upon RepVF, we introduce RFTR, a network designed to exploit the inherent connections between different tasks by utilizing a hierarchical structure of queries that implicitly model the relationships both between and within tasks. This approach eliminates the need for task-specific heads and parameters, fundamentally reducing the conflicts inherent in traditional multi-task learning paradigms. We validate our approach by combining labels from the OpenLane dataset with the Waymo Open dataset. Our work presents a significant advancement in the efficiency and effectiveness of multi-task perception in autonomous driving, offering a new perspective on handling multiple 3D perception tasks synchronously and in parallel. The code will be available at: https://github.com/jbji/RepVF

Figures (4)

• Figure 1: RepVF is proposed as a unified representation for 3D perception to achieve single-head multi-tasking. It consists of a set of vector fields learned to represent road elements, and is trained utilizing existing labels through differentiable conversions. The proposed single-head multi-tasking paradigm reduces task conflict and competition.
• Figure 2: Overview of our RFTR (Representative Vector Fields Transformer) built upon RepVF. Multi-view image features with 3D position embeds are extracted by image backbone and fed into the decoder. We generate set-level query embeds from 3D space sampling sets, each representing a perception target. One single unified task head is then used to predict unified representative vector fields. Finally, predictions are transformed in a differentiable manner into task specific representations to utilize existing labels for supervision.
• Figure 3: Our single-head RFTR model shows a reduced gradient imbalance and better disparity stability (mean: 2.47, variance: 0.90) compared to the multi-head baseline (mean: 2.98 variance: 1.86) or the multi-head baseline with RepVF (mean: 2.72, variance: 1.57). Ideally balanced gradient disparity through training iterations should approach 2. For ease of trend observation, we have clipped the top 2% values to the mean, with curves smoothed through a window size of 50 1-d convolution.
• Figure 4: Visualization of the RFTR model's performance and the learned RepVF.