Online Monitoring Framework for Automotive Time Series Data using JEPA Embeddings

TL;DR

The paper tackles the challenge of online anomaly monitoring for autonomous vehicles without relying on anomaly labels. It introduces a three-component framework that combines perception, JEPA-based self-supervised embeddings, and latent-space anomaly detection to monitor object lists in real time. The method is validated on the nuScenes dataset, showing that latent embeddings enable more reliable anomaly detection under strict false-alarm budgets compared to a baseline in the original object space. The approach supports operation-phase safety validation and data collection for continuous learning, with open-source code to facilitate adoption and further research.

Abstract

As autonomous vehicles are rolled out, measures must be taken to ensure their safe operation. In order to supervise a system that is already in operation, monitoring frameworks are frequently employed. These run continuously online in the background, supervising the system status and recording anomalies. This work proposes an online monitoring framework to detect anomalies in object state representations. Thereby, a key challenge is creating a framework for anomaly detection without anomaly labels, which are usually unavailable for unknown anomalies. To address this issue, this work applies a self-supervised embedding method to translate object data into a latent representation space. For this, a JEPA-based self-supervised prediction task is constructed, allowing training without anomaly labels and the creation of rich object embeddings. The resulting expressive JEPA embeddings serve as input for established anomaly detection methods, in order to identify anomalies within object state representations. This framework is particularly useful for applications in real-world environments, where new or unknown anomalies may occur during operation for which there are no labels available. Experiments performed on the publicly available, real-world nuScenes dataset illustrate the framework's capabilities.

Figures (4)

• Figure 1: Framework overview: In the perception system mot is performed to obtain object state estimations, which are encoded by $E_{\theta}$ into latent embeddings $\bm{z}$ and used for anomaly detection.
• Figure 2: Architecture of the self-supervised embedding method based on jepa LeCun2022.
• Figure 3: Plot of two variants of the roc curve, from a single run using abod as $\varphi$.
• Figure 4: Visualization of the anomaly scores from the lof for normal objects and altered objects using different means $\mu_{\text{v}} \in \{2.5, 5, 7.5\}$$m/s$.