BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving

TL;DR

BeLLA introduces an end-to-end framework that grounds a unified 360° BEV representation in a large language model for autonomous-driving question answering. By freezing a BEV encoder and learning a BEV→text projector in pretraining, followed by end-to-end finetuning of an LLM with LoRA adapters, BeLLA achieves competitive and, in some cases, state-of-the-art performance on spatial reasoning tasks across NuScenes-QA and DriveLM. The approach highlights the value of BEV-language alignment for reasoning about object relations, motions, and behaviors in driving scenes, while revealing limitations in appearance-sensitive queries and temporal dynamics. The work suggests practical avenues for enhancing BEV-based reasoning, such as incorporating raw camera cues and extending to video inputs to capture temporal context. Overall, BeLLA advances BEV-grounded multimodal reasoning in autonomous driving with a scalable, end-to-end training paradigm.

Abstract

The rapid development of Vision-Language models (VLMs) and Multimodal Language Models (MLLMs) in autonomous driving research has significantly reshaped the landscape by enabling richer scene understanding, context-aware reasoning, and more interpretable decision-making. However, a lot of existing work often relies on either single-view encoders that fail to exploit the spatial structure of multi-camera systems or operate on aggregated multi-view features, which lack a unified spatial representation, making it more challenging to reason about ego-centric directions, object relations, and the wider context. We thus present BeLLA, an end-to-end architecture that connects unified 360° BEV representations with a large language model for question answering in autonomous driving. We primarily evaluate our work using two benchmarks - NuScenes-QA and DriveLM, where BeLLA consistently outperforms existing approaches on questions that require greater spatial reasoning, such as those involving relative object positioning and behavioral understanding of nearby objects, achieving up to +9.3% absolute improvement in certain tasks. In other categories, BeLLA performs competitively, demonstrating the capability of handling a diverse range of questions.

Figures (2)

• Figure 1: Overview of the BeLLA framework. A frozen BEV encoder processes multi-view camera images to obtain a unified BEV feature map $\mathbf{B} \in \mathbb{R}^{H \times W \times C}$. Part (a) of the figure illustrates the general workflow shared across both pretraining and finetuning stages. During the pretraining phase, $\mathbf{B}$ is passed through a trainable projector $f_\theta$ to produce the projected embedding $\mathbf{E}_{\text{BEV}} \in \mathbb{R}^{1 \times d}$, which is used for aligning the BEV text with its description using the frozen LLM. During the finetuning phase, a textual prompt $\mathbf{\textit{q}}$ is embedded and fused with $\mathbf{E}_{\text{BEV}}$ as input to the LLM, which is fine-tuned using parameter-efficient LoRA adapters for answer generation. While $\mathbf{\textit{q}}$ corresponds to a natural language question during finetuning, it contains only a placeholder token in the pretraining stage, which is replaced by $\mathbf{E}_{\text{BEV}}$.
• Figure 2: Qualitative results from the DriveLM benchmark: each example includes the rendered BEV map, the corresponding multi-view camera images, along with the natural language question, ground-truth answer, and the model's predicted answer. In some cases, the question refers to scene elements via spatial coordinates. To aid interpretation, we have manually mentioned those elements in the 'reference', as well as added a bounding box on the camera images. These cues are not provided to the model and are included solely for visualization purposes. Examples with a blue border denote correct (positive) responses, while the red-bordered example shows a failure case.