CarFormer: Self-Driving with Learned Object-Centric Representations

TL;DR

CarFormer introduces learned object-centric slot representations for self-driving, extracting slots from BEV sequences via slot attention and using a GPT-2–style autoregressive transformer with block attention to drive and forecast scene dynamics. The method jointly learns driving policies and future slot predictions, enabling both action prediction and world-model capabilities. Empirical results on CARLA Longest6 show that slot-based representations outperform scene-level approaches and can match or exceed exact-attribute object models, with superior robustness and forecasting performance. The work demonstrates that object-centric slots capture essential vehicle dynamics (position, heading, speed) through spatio-temporal context without explicit attributes, offering a scalable and generalizable pathway for object-aware autonomous driving with strong potential for end-to-end BEV-slot extraction and multi-step planning.

Abstract

The choice of representation plays a key role in self-driving. Bird's eye view (BEV) representations have shown remarkable performance in recent years. In this paper, we propose to learn object-centric representations in BEV to distill a complex scene into more actionable information for self-driving. We first learn to place objects into slots with a slot attention model on BEV sequences. Based on these object-centric representations, we then train a transformer to learn to drive as well as reason about the future of other vehicles. We found that object-centric slot representations outperform both scene-level and object-level approaches that use the exact attributes of objects. Slot representations naturally incorporate information about objects from their spatial and temporal context such as position, heading, and speed without explicitly providing it. Our model with slots achieves an increased completion rate of the provided routes and, consequently, a higher driving score, with a lower variance across multiple runs, affirming slots as a reliable alternative in object-centric approaches. Additionally, we validate our model's performance as a world model through forecasting experiments, demonstrating its capability to predict future slot representations accurately. The code and the pre-trained models can be found at https://kuis-ai.github.io/CarFormer/.

Figures (5)

• Figure 1: Overview of CarFormer. Given a trajectory $\tau_t$ consisting of discrete and continuous inputs, we first embed these tokens to the same hidden dimension $H$. For scalar inputs such as target point ($g_t^x$, $g_t^y$), traffic light flag $l_t$, speed $v_t$, and waypoints $q_t^i$, we discretize them, using k-means if not discrete, before we look them up from an embedding matrix. For continuous inputs like the $K$ slot features $\{\mathbf{z}_t^i\}_{i=1}^K$ and the desired route $\mathbf{r}_t^1, \mathbf{r}_t^2$, we project them using an MLP. Conditioned on this context, CarFormer learns to jointly predict future slot features $\{\mathbf{z}_{t+1}^i\}_{i=1}^K$ and the waypoints autoregressively using the backbone ($\mathbf{q}_t$) as well as the GRU head ($\mathbf{w}_t$). The K slot features $\{\mathbf{z}_t^i\}_{i=1}^K$ are extracted from a pre-trained, frozen, SAVi model, shown on the left. The K slot features, along with the desired route, are considered a block, and block attention is applied in the attention layers as shown on the top.
• Figure 2: Visualization of Slot Forecasting Results. Each sub-figure shows an example of input (dark grey)-output (light grey) objects in the first column, SAVi reconstructions in the second column, and our model's predictions in the third column. The top left corner of each column shows the mIoU compared to the ground truth. For comparison, we overlay the three in the last column where the red channel (R) is the ground-truth location, the green channel (G) is SAVi reconstruction, and the blue channel (B) is our prediction. In the case of perfect alignment between the three, we see the vehicles in white, and different errors for our model can be seen in a unique color such as yellow (R+G) indicating misses and blue indicating false positives (B).
• Figure 3: Ablation of Forecasting Weight. We visualize the effect of varying the hyper-parameter $\alpha$, the weight of forecasting in the loss function, on driving performance. While initially increasing the contribution of forecasting improves the driving performance, it peaks around 40. We use CarFormer with SAVi-30-light as the encoder backbone for all experiments.
• Figure 4: Comparison of Slot Extraction by Varying the Number of Slots. Within each sub-figure, the columns correspond to ground truth BEV followed by SAVi-light reconstructions with 7, 14, and 30 slots, respectively.
• Figure 5: Additional Visualization of Slot Forecasting Results for CarFormer with SAVi-7-base (left half) and CarFormer with SAVi-14-light (right half). Each sub-figure shows an example of input (dark grey)-output (light grey) objects in the first column, SAVi reconstructions in the second column, and our model's predictions in the third column. The top left corner of each column shows the mIoU compared to the ground truth. For comparison, we overlay the three in the last column where the red channel (R) is the ground-truth location, the green channel (G) is SAVi reconstruction, and the blue channel (B) is our prediction. In the case of perfect alignment between the three, we see the vehicles in white, and different errors can be seen from combinations of R-G-B colors such as yellow (R+G) indicating misses and blue indicating false positives for our model (B).