MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Driving

TL;DR

The paper addresses the high computational cost and single-view limitations of existing vision-language models for autonomous driving. It introduces MiniDrive, a lightweight framework that maps multi-view 2D image features into text tokens using a Feature Engineering Mixture of Experts and dynamically adapts these tokens via a Dynamic Instruction Adapter, while employing a compact vision backbone and language model. With 83M parameters, MiniDrive achieves strong efficiency and competitive accuracy on Drive-LM and CODA-LM benchmarks, demonstrating that multi-view perception can be effectively integrated into language models for real-time autonomous driving. The work offers a practical path toward deployment on resource-constrained hardware and provides open-source resources to foster further development.

Abstract

Vision-language models (VLMs) serve as general-purpose end-to-end models in autonomous driving, performing subtasks such as prediction, planning, and perception through question-and-answer interactions. However, most existing methods rely on computationally expensive visual encoders and large language models (LLMs), making them difficult to deploy in real-world scenarios and real-time applications. Meanwhile, most existing VLMs lack the ability to process multiple images, making it difficult to adapt to multi-camera perception in autonomous driving. To address these issues, we propose a novel framework called MiniDrive, which incorporates our proposed Feature Engineering Mixture of Experts (FE-MoE) module and Dynamic Instruction Adapter (DI-Adapter). The FE-MoE effectively maps 2D features into visual token embeddings before being input into the language model. The DI-Adapter enables the visual token embeddings to dynamically change with the instruction text embeddings, resolving the issue of static visual token embeddings for the same image in previous approaches. Compared to previous works, MiniDrive achieves state-of-the-art performance in terms of parameter size, floating point operations, and response efficiency, with the smallest version containing only 83M parameters.

Figures (8)

• Figure 1: (a) shows the input format of MiniDrive and the tasks it can perform. (b) compares the average evaluation of multiple-image inputs on the Drive-LM evaluation system with related models. (c) compares the average evaluation of single-image inputs on the CODA-LM evaluation system with related models. Minidrive outperforms open-source models larger than 7B and approaches the performance of commercial models.
• Figure 2: Comparison of the MiniDrive architecture with mainstream architectures. (a) Existing vision-language models primarily use a Transformer-based visual encoder to learn image patches as visual tokens. These visual tokens remain unchanged regardless of the user's questions. (b) Our architecture employs a more efficient large convolutional kernel as the visual encoder, learning 2D features of the image as visual tokens. These visual tokens change in response to different user questions.
• Figure 3: MiniDrive Structural Details. In Figure (a), the overall architecture of MiniDrive is presented. The image features from the vision encoder input are processed by the FE-MoE and DI-Adapter with residual connections, resulting in visual token embeddings. These embeddings, along with text embeddings, are then fed into the T5-Small language model, producing the output. In Figure (b), the specific framework of FE-MoE is shown. The image is input into UniRepLKNet, producing feature maps at different levels. These feature maps are then fed into the FE-MoE module, where the Gate network generates weights. The 2D visual features are further assigned to different experts for feature mapping and weighted summation.
• Figure 4: UniRepLKNet generates feature maps. We obtain the set of feature maps from each image propagated to the final stage.
• Figure 5: Examples of MiniDrive’s Response. The color blue represents the user command querying for multi-image input. The color red represents the activation response generated by MiniDrive corresponding to the text.