Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet

Arian Mehrfard, Bharanidhar Duraisamy, Stefan Haag, Florian Geiss

TL;DR

The results demonstrate that KalmanNet is outperformed by the IMM filter and indicate that while data-driven methods such as KalmanNet show promise, their current lack of reliability and robustness makes them unsuited for safety-critical applications.

Abstract

Kalman Filters (KF) are fundamental to real-time state estimation applications, including radar-based tracking systems used in modern driver assistance and safety technologies. In a linear dynamical system with Gaussian noise distributions the KF is the optimal estimator. However, real-world systems often deviate from these assumptions. This deviation combined with the success of deep learning across many disciplines has prompted the exploration of data driven approaches that leverage deep learning for filtering applications. These learned state estimators are often reported to outperform traditional model based systems. In this work, one prevalent model, KalmanNet, was selected and evaluated on automotive radar data to assess its performance under real-world conditions and compare it to an interacting multiple models (IMM) filter. The evaluation is based on raw and normalized errors as well as the state uncertainty. The results demonstrate that KalmanNet is outperformed by the IMM filter and indicate that while data-driven methods such as KalmanNet show promise, their current lack of reliability and robustness makes them unsuited for safety-critical applications.

Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet

TL;DR

The results demonstrate that KalmanNet is outperformed by the IMM filter and indicate that while data-driven methods such as KalmanNet show promise, their current lack of reliability and robustness makes them unsuited for safety-critical applications.

Abstract

Kalman Filters (KF) are fundamental to real-time state estimation applications, including radar-based tracking systems used in modern driver assistance and safety technologies. In a linear dynamical system with Gaussian noise distributions the KF is the optimal estimator. However, real-world systems often deviate from these assumptions. This deviation combined with the success of deep learning across many disciplines has prompted the exploration of data driven approaches that leverage deep learning for filtering applications. These learned state estimators are often reported to outperform traditional model based systems. In this work, one prevalent model, KalmanNet, was selected and evaluated on automotive radar data to assess its performance under real-world conditions and compare it to an interacting multiple models (IMM) filter. The evaluation is based on raw and normalized errors as well as the state uncertainty. The results demonstrate that KalmanNet is outperformed by the IMM filter and indicate that while data-driven methods such as KalmanNet show promise, their current lack of reliability and robustness makes them unsuited for safety-critical applications.

Paper Structure

This paper contains 14 sections, 19 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Relevant Work
  4. Interacting Multiple Model Algorithm
  5. KalmanNet
  6. Training Setup & Data
  7. Dataset
  8. KalmanNet Initialization
  9. Training
  10. Results
  11. Discussion
  12. The $8$-drive scenario
  13. The follow-drive scenario
  14. Conclusion

Figures (5)

  • Figure 1: Estimated position of OV from KalmanNet. Input was averaged over clusters at each $k$.
  • Figure 2: RMSE comparison between KalmanNet and IMM on the $8$-drive scenario. The plots shows the position (top) and velocity (bottom) errors.
  • Figure 3: RMSE comparison between KalmanNet and IMM on the follow-drive scenario. The plots shows the position (top) and velocity (bottom) errors.
  • Figure 4: Comparison of the state covariance volume between KalmanNet and IMM on the $8$-drive scenario. The volume is computed for the position and velocity variance and covariance components.
  • Figure 5: NEES and NIS comparison between KalmanNet and IMM on the $8$-drive scenario. Dotted lines represent the upper and lower limits of the chi-squared distributions.