Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning

TL;DR

This paper addresses the challenge of learning rich, task-relevant representations for visual reinforcement learning. It proposes Semore, a dual-stream framework guided by vision-language models to separately learn semantic and motion representations from RGB sequences, with feature-level supervision and VLM-driven interaction. A combination of semantic alignment, motion enhancement, and a reward-predictive head yields superior performance in autonomous driving scenarios, validated on CARLA benchmarks. The approach distills knowledge from VLMs into the encoders, improving data efficiency and decision quality, and the authors release the code.

Abstract

The growing exploration of Large Language Models (LLM) and Vision-Language Models (VLM) has opened avenues for enhancing the effectiveness of reinforcement learning (RL). However, existing LLM-based RL methods often focus on the guidance of control policy and encounter the challenge of limited representations of the backbone networks. To tackle this problem, we introduce Enhanced Semantic Motion Representations (Semore), a new VLM-based framework for visual RL, which can simultaneously extract semantic and motion representations through a dual-path backbone from the RGB flows. Semore utilizes VLM with common-sense knowledge to retrieve key information from observations, while using the pre-trained clip to achieve the text-image alignment, thereby embedding the ground-truth representations into the backbone. To efficiently fuse semantic and motion representations for decision-making, our method adopts a separately supervised approach to simultaneously guide the extraction of semantics and motion, while allowing them to interact spontaneously. Extensive experiments demonstrate that, under the guidance of VLM at the feature level, our method exhibits efficient and adaptive ability compared to state-of-art methods. All codes are released.

Figures (8)

• Figure 1: (a) In the first row, the overly large sampling space of RL leads to difficulty in capturing key objectives in extreme scenarios; (b) In the second row, due to the complex high-dimensional feature space and the back propagation, guidance at the policy level cannot ensure that the encoder extracts reliable features; (c) In contrast, our method can fully take advantage of the capability of VLMs to enhance the task-specific representations.
• Figure 2: The overall VLM-guided learning framework. It integrates two key modules: 1) the VLM-guided semantics module employs the similarity loss to explicitly supervise the extraction of semantic representations, while the motion supervision module introduces knowledge-aware features into the motion extraction using bidirectional cross-attention.
• Figure 3: The illustration of VLM-generated knowledge-aware representations. (a) and (d) show the road scenarios of rural and urban areas, respectively. The text prompts are: (b): The right trees; (c): The pedestrian; (e): The white van on the left side of the black car ahead; (f): The left white car.
• Figure 4: Interaction of motion and semantic features. (Fusion is concat and knowledge-aware features are only used in training stage.).
• Figure 5: VLM-based selective replay buffer.