Large Multimodal Models for Embodied Intelligent Driving: The Next Frontier in Self-Driving?

TL;DR

This work tackles the limitations of modular autonomous driving in open-world settings by proposing a semantics and policy dual-driven hybrid framework that fuses Large Multimodal Models (LMMs) with Deep Reinforcement Learning (DRL) to enable continuous learning and joint decision-making in Embodied Intelligent (EI) driving. The framework comprises a three-stage pipeline: semantic reasoning via LMMs, real-time policy optimization via DRL formulated as an MDP over states $s$, actions $a$, and reward $R(s,a)$ with $R(s_t,a_t)=R_t^{sfty}+R_t^{de}+R_t^{comf}$, and a fusion mechanism to align reasoning with policy and enable backward learning. A lane-change planning case study demonstrates superior convergence and higher average rewards compared with baselines, validating the approach in a mixed traffic scenario. The authors also outline future directions, including virtual-real collaborative training, endogenous security defenses, and AGI-enabled multi-agent collaboration, to advance practical deployment of EI driving.

Abstract

The advent of Large Multimodal Models (LMMs) offers a promising technology to tackle the limitations of modular design in autonomous driving, which often falters in open-world scenarios requiring sustained environmental understanding and logical reasoning. Besides, embodied artificial intelligence facilitates policy optimization through closed-loop interactions to achieve the continuous learning capability, thereby advancing autonomous driving toward embodied intelligent (El) driving. However, such capability will be constrained by relying solely on LMMs to enhance EI driving without joint decision-making. This article introduces a novel semantics and policy dual-driven hybrid decision framework to tackle this challenge, ensuring continuous learning and joint decision. The framework merges LMMs for semantic understanding and cognitive representation, and deep reinforcement learning (DRL) for real-time policy optimization. We starts by introducing the foundational principles of EI driving and LMMs. Moreover, we examine the emerging opportunities this framework enables, encompassing potential benefits and representative use cases. A case study is conducted experimentally to validate the performance superiority of our framework in completing lane-change planning task. Finally, several future research directions to empower EI driving are identified to guide subsequent work.

Figures (5)

• Figure 1: An illustration of the EI driving scenarios enabled by the joint decision approach based on LMMs and DRL. Such approach shows great promise in three illustrative application scenarios: i) Urban robotaxi, which is on-demand and driverless transport, faces tremendous challenges in negotiating highly dynamic traffic scenarios. Examples would be predicting a pedestrians intent, and avoiding a sudden obstacle in a narrow corridor. ii) Autonomous freight operates in trunk-line logistics, ports, and industrial parks, where good performance is needed in complex geometrical road situations. Representative scenarios include container handling, and narrow space delivery robots. iii) Structured shuttle, driving in structured and semi-open environments, must achieve precision docking. Typical scenarios include airport shuttle service, and micro-circulation bus.
• Figure 2: The architecture of EI vehicle, which consists of the embodied perception module, embodied decision module, and embodied execution module. The EI vehicle first achieves the autonomous environmental perception through multimodal sensors, and collects heterogeneous data sources, such as vision, radar, velocity, and textual data. Based on the comprehensively fused perception, the EI vehicle proceeds to interpret the surrounding environment and perform the scenario-based decision-making, thereby formulating the high-level policy intentions. Finally, these policy intentions are translated into the concrete control actions for EI vehicle. These modules operate as a closed-loop pipeline, including the perception, decision, and execution.
• Figure 3: An overview of the overall framework of LMMs. In particular, the LMMs achieve unified cross-modal understanding and support any-to-any modality inputs and outputs through multimodal encoders, multimodal alignment learning, multimodal generation, and multimodal instruction tuning.
• Figure 4: Illustration of the proposed semantics and policy dual-driven hybrid decision framework to implement the LMM-empowered EI driving. Three successive stages are designed and incorporated into this framework: semantic pipeline, policy pipeline, and fusion pipeline.
• Figure 5: Performance evaluation of the proposed dual-driven hybrid decision framework, compared with the baseline schemes including the state-of-the-art DRL algorithms and the particular case of W/O LMM: (a) Convergence performance of the adopted D3QN algorithm integrated with LMMs in the proposed framework; (b) Average reward versus the number of human-driven vehicles.