Vision-Conditioned Variational Bayesian Last Layer Dynamics Models

TL;DR

This work tackles the challenge of proactive dynamics modeling for autonomous racing under rapidly changing road conditions. It introduces VcVBLL, a vision-conditioned variational Bayesian last-layer dynamics model, that uses FiLM-based conditioning to incorporate visual context into predictive dynamics and an MPC controller for proactive racing. The approach employs a two-stage training scheme: a base VBLL learned on nominal dry data, followed by fine-tuning the visual conditioning path with limited wet-condition data, achieving interpretable, context-aware predictions. Hardware validation on a Lexus LC500 racing through water puddles shows that vision-conditioned control enables complete laps under varying conditions, while baselines without vision fail, highlighting the practical impact of exteroceptive-informed dynamics for safety and performance.

Abstract

Agile control of robotic systems often requires anticipating how the environment affects system behavior. For example, a driver must perceive the road ahead to anticipate available friction and plan actions accordingly. Achieving such proactive adaptation within autonomous frameworks remains a challenge, particularly under rapidly changing conditions. Traditional modeling approaches often struggle to capture abrupt variations in system behavior, while adaptive methods are inherently reactive and may adapt too late to ensure safety. We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment. The model first learns nominal vehicle dynamics and is then fine-tuned with feature-wise affine transformations of latent features, enabling context-aware dynamics prediction. The resulting model is integrated into an optimal controller for vehicle racing. We validate our method on a Lexus LC500 racing through water puddles. With vision-conditioning, the system completed all 12 attempted laps under varying conditions. In contrast, all baselines without visual context consistently lost control, demonstrating the importance of proactive dynamics adaptation in high-performance applications.

Figures (7)

• Figure 1: A learned vehicle model conditioned on visual context is used for predictive control. With the visual context (orange), the car autonomously races through the water (blue), while models void of context spin out (red).
• Figure 2: Vehicle geometry with the bicycle model states $\boldsymbol{x}$ (black), control inputs $\boldsymbol{u}$ (blue), tire forces (red), and the reference path $\boldsymbol{x}_{\mathrm{ref}}(s)$ (orange).
• Figure 3: Conceptual overview of VcVBLL and its training. We begin by training a base VBLL dynamics model (green path). To incorporate vision-based conditioning, we first extract semantic classes using a fine-tuned Segman-t backbone from the limited available data with water interaction, from which we compute a short 2 time series of past water scores corresponding to $\boldsymbol{c}_t$. In a subsequent fine-tuning phase, this time series is encoded with an LSTM and used in a FiLM conditioning path to obtain our vision-conditioned VcVBLL dynamics model (orange path).
• Figure 4: Slices through the VcVBLL posterior when driving over a puddle. After fine-tuning, the VcVBLL model can effectively provide conditional prediction based on the provided visual context, as seen by the difference in posterior predictive distributions for the selected states. Moreover, the model’s behavior is physically plausible: a wet surface results in higher wheel speeds and increased side-slip dynamics, as well as increased uncertainty.
• Figure 5: Performance of VBLL dynamics model on the dry track. With a learned dynamics model (VBLL and VcVBLL) the MPC is able to achieve higher velocities resulting in faster lap times (cf. Table \ref{['tab:laptimes']}).

Theorems & Definitions (2)

• Remark 1
• Remark 2