Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Deep Learning for Remaining Useful Life Estimation via Stein Variational Gradient Descent

Luca Della Libera, Jacopo Andreoli, Davide Dalle Pezze, Mirco Ravanelli, Gian Antonio Susto

TL;DR

This work tackles the challenge of estimating remaining useful life (RUL) while quantifying predictive uncertainty. It advocates Bayesian neural networks trained with Stein variational gradient descent (SVGD) as a scalable, expressive alternative to traditional Bayes by Backprop and MCMC methods. On simulated run-to-failure data from NASA's C-MAPSS turbofan benchmarks, SVGD-based models consistently outperform baselines and a simple uncertainty-informed prediction heuristic further reduces risk from late predictions. The results highlight the practical value of SVGD for uncertainty-aware RUL prognostics and suggest avenues for future exploration, including larger particle counts, kernel choices, and transformer-based architectures, with code released for reproducibility.

Abstract

A crucial task in predictive maintenance is estimating the remaining useful life of physical systems. In the last decade, deep learning has improved considerably upon traditional model-based and statistical approaches in terms of predictive performance. However, in order to optimally plan maintenance operations, it is also important to quantify the uncertainty inherent to the predictions. This issue can be addressed by turning standard frequentist neural networks into Bayesian neural networks, which are naturally capable of providing confidence intervals around the estimates. Several methods exist for training those models. Researchers have focused mostly on parametric variational inference and sampling-based techniques, which notoriously suffer from limited approximation power and large computational burden, respectively. In this work, we use Stein variational gradient descent, a recently proposed algorithm for approximating intractable distributions that overcomes the drawbacks of the aforementioned techniques. In particular, we show through experimental studies on simulated run-to-failure turbofan engine degradation data that Bayesian deep learning models trained via Stein variational gradient descent consistently outperform with respect to convergence speed and predictive performance both the same models trained via parametric variational inference and their frequentist counterparts trained via backpropagation. Furthermore, we propose a method to enhance performance based on the uncertainty information provided by the Bayesian models. We release the source code at https://github.com/lucadellalib/bdl-rul-svgd.

Bayesian Deep Learning for Remaining Useful Life Estimation via Stein Variational Gradient Descent

TL;DR

This work tackles the challenge of estimating remaining useful life (RUL) while quantifying predictive uncertainty. It advocates Bayesian neural networks trained with Stein variational gradient descent (SVGD) as a scalable, expressive alternative to traditional Bayes by Backprop and MCMC methods. On simulated run-to-failure data from NASA's C-MAPSS turbofan benchmarks, SVGD-based models consistently outperform baselines and a simple uncertainty-informed prediction heuristic further reduces risk from late predictions. The results highlight the practical value of SVGD for uncertainty-aware RUL prognostics and suggest avenues for future exploration, including larger particle counts, kernel choices, and transformer-based architectures, with code released for reproducibility.

Abstract

A crucial task in predictive maintenance is estimating the remaining useful life of physical systems. In the last decade, deep learning has improved considerably upon traditional model-based and statistical approaches in terms of predictive performance. However, in order to optimally plan maintenance operations, it is also important to quantify the uncertainty inherent to the predictions. This issue can be addressed by turning standard frequentist neural networks into Bayesian neural networks, which are naturally capable of providing confidence intervals around the estimates. Several methods exist for training those models. Researchers have focused mostly on parametric variational inference and sampling-based techniques, which notoriously suffer from limited approximation power and large computational burden, respectively. In this work, we use Stein variational gradient descent, a recently proposed algorithm for approximating intractable distributions that overcomes the drawbacks of the aforementioned techniques. In particular, we show through experimental studies on simulated run-to-failure turbofan engine degradation data that Bayesian deep learning models trained via Stein variational gradient descent consistently outperform with respect to convergence speed and predictive performance both the same models trained via parametric variational inference and their frequentist counterparts trained via backpropagation. Furthermore, we propose a method to enhance performance based on the uncertainty information provided by the Bayesian models. We release the source code at https://github.com/lucadellalib/bdl-rul-svgd.
Paper Structure (19 sections, 15 equations, 3 figures, 4 tables)

This paper contains 19 sections, 15 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Bayesian Neural Networks
  5. Bayes by Backprop
  6. Stein Variational Gradient Descent
  7. Uncertainty-Informed Predictions
  8. Experimental Setup
  9. Dataset
  10. Preprocessing
  11. Feature selection
  12. Normalization
  13. Sliding window segmentation
  14. Target rectification
  15. Deep Learning Models
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Min-max normalized sample from FD001 test set with $T = 30$ and $F = 14$.
  • Figure 2: Comparison between $\text{RMSE}$, $\text{MAE}$, and score function with respect to different average error values, where the error is the difference between the prediction and the target. The score function is asymmetric: it penalizes late predictions ($\text{error} > 0$) more than early ones ($\text{error} < 0$).
  • Figure 3: Prior and posterior distributions (first column) of the first weight of the first hidden layer and corresponding posterior predictive distribution (second column) of a test sample of Dense3 (D3) trained on FD001 via backpropagation (BP), Bayes by Backprop (BBB), and Stein variational gradient descent (SVGD). Prediction$\ast$ is computed using the heuristic described in \ref{['subsec:uncertainty_informed_predictions']}.