LORD: Large Models based Opposite Reward Design for Autonomous Driving

TL;DR

The paper tackles reward design for reinforcement learning in autonomous driving, where specifying a concrete, safe driving goal is inherently ambiguous. It introduces LORD, a framework that uses undesired linguistic goals (e.g., 'a collision is happening') and large pretrained models to produce step-wise rewards via embeddings, enabling zero-shot reward signals. LORD supports image, video, and text observations and is trained with PPO in a closed-loop Highway-env environment, demonstrating strong generalization to unseen, denser traffic scenarios and outperforming baseline methods and target-goal designs. This approach offers a scalable, generalizable reward design paradigm for embodied AI, reducing manual reward engineering by leveraging existing large-model capabilities. {$<S,O,\theta,A,T,\R,\gamma>$} is used to formalize the problem and the cosine-distance reward is computed from embedding similarities, underscoring the method's mathematically grounded design.

Abstract

Reinforcement learning (RL) based autonomous driving has emerged as a promising alternative to data-driven imitation learning approaches. However, crafting effective reward functions for RL poses challenges due to the complexity of defining and quantifying good driving behaviors across diverse scenarios. Recently, large pretrained models have gained significant attention as zero-shot reward models for tasks specified with desired linguistic goals. However, the desired linguistic goals for autonomous driving such as "drive safely" are ambiguous and incomprehensible by pretrained models. On the other hand, undesired linguistic goals like "collision" are more concrete and tractable. In this work, we introduce LORD, a novel large models based opposite reward design through undesired linguistic goals to enable the efficient use of large pretrained models as zero-shot reward models. Through extensive experiments, our proposed framework shows its efficiency in leveraging the power of large pretrained models for achieving safe and enhanced autonomous driving. Moreover, the proposed approach shows improved generalization capabilities as it outperforms counterpart methods across diverse and challenging driving scenarios.

Figures (8)

• Figure 1: An overview of our LORD powered reinforcement learning framework for closed-loop autonomous driving task. LORD firstly measures cosine similarity between agent's state and undesired goal state using large pretrained models. Followingly, it returns cosine distance as the reward to the agent.
• Figure 2: The insight of using an opposite goal. In autonomous driving tasks, desired goal states such as "drive safely" are ambiguous to grasp, whereas undesired goal states such as "collision" are tractable more comprehensible to humans and large pretrained models.
• Figure 3: Illustrations of the original and the modified Highway-env. In the modified environment, white car denotes the ego vehicle and blue cars depict the npc vehicles.
• Figure 4: Illustrations of the reward values generated by our LORD under various observation representations and GRADxi2022graph for different states. We distinguish different states in terms of the ego vehicle's distance to its nearest front vehicle and their speed difference. In this way, time to collision can be roughly estimated. Blue points denote collision-free states while red points indicate the ego vehicle collides with other vehicles.
• Figure 5: Illustrations of how the driving policy learned by our LORD with image, video and text based observation performs in the lane-5-density-3 setting of Highway-env. The ego vehicle is colored in green with a line depicting its past trajectory. The ego vehicle behaves properly in the congested traffic scenarios.