HanoiWorld : A Joint Embedding Predictive Architecture BasedWorld Model for Autonomous Vehicle Controller

TL;DR

HanoiWorld introduces a joint embedding predictive architecture-based world model for autonomous driving, replacing pixel reconstruction with predictive latent embeddings to enable long-horizon planning under partial observability. It combines a strong, pretrained V-JEPA-2 encoder with an EMA teacher, a recurrent state-space memory (RSSM), and a latent-space actor-critic trained in the world-model latent space, following DreamerV3-inspired objectives. Empirical results on HighwayEnv show HanoiWorld achieving safer planning than strong baselines in Highway and Roundabout tasks, but facing challenges in the more demanding Merge scenario, highlighting both robustness and failure modes of latent imagination in driving. The work demonstrates the practicality of predictive embedding-based world models for efficient, safety-aware driving control and points to future enhancements through multi-modal conditioning and richer environment representations.

Abstract

Current attempts of Reinforcement Learning for Autonomous Controller are data-demanding while the results are under-performed, unstable, and unable to grasp and anchor on the concept of safety, and over-concentrating on noise features due to the nature of pixel reconstruction. While current Self-Supervised Learningapproachs that learning on high-dimensional representations by leveraging the JointEmbedding Predictive Architecture (JEPA) are interesting and an effective alternative, as the idea mimics the natural ability of the human brain in acquiring new skill usingimagination and minimal samples of observations. This study introduces Hanoi-World, a JEPA-based world model that using recurrent neural network (RNN) formaking longterm horizontal planning with effective inference time. Experimentsconducted on the Highway-Env package with difference enviroment showcase the effective capability of making a driving plan while safety-awareness, with considerablecollision rate in comparison with SOTA baselines

Figures (27)

• Figure 1: The proposed HanoiWorld World Model (left) include an visual encoder based on V-JEPA-2 checkpoint proposed by assran_2025_vjepa and the RSSM-backbone suggested by hafner_2019_learning for making long-term planning on the next possible transition of enviroment - the green block. The overal system arrchitectural of the enviroment on the (right) which been design aiming for both effective rolling-out while model training from the enviroment by creating such feedback loop on the agent interaction with data-scarcity
• Figure 2: Overview of the proposed HanoiWorld encoder architecture. A pretrained and frozen V-JEPA 2 encoder assran_2025_vjepa is used as a high-quality representation backbone to improve training efficiency and embedding robustness under limited-data settings. A downstream bottleneck Multi-Layer Perceptron (MLP) is trained to project the high-dimensional representations into a compact and task-compatible latent space of size $1024 \times 128$. In parallel, the student encoder branch incorporates an additional 2D convolutional neural network (CNN) module to predict spatial representations. Both branches are jointly optimized using an $\ell_1$ alignment loss as Eq. \ref{['eq:jepa_loss']} and the VICReg regularization objective bardes_2022_vicreg.
• Figure 3: The highway-v0 Enviroment - a snapshot taken from faramafoundation_2025_gymnasium
• Figure 4: The roundabout-v0 Enviroment - a snapshot taken from faramafoundation_2025_gymnasium
• Figure 5: The roundabout-v0 Enviroment - a snapshot taken from faramafoundation_2025_gymnasium