Science based AI model certification for new operational environments with application in traffic state estimation

TL;DR

This work tackles the challenge of certifying AI deployed in new operational environments with limited data and interpretability concerns. It introduces a science-based certification framework that fuses physics-based traffic flow models (conservation laws, the Greenshields diagram, and the LWR model) with data-driven AI, and introduces physics-oriented metrics such as Normalized Physics Loss (NPL). Synthetic datasets generated via the Lax-Hopf method, aligned with conservation laws, enable evaluation of predictions under domain shifts and a categorization into reuse, refine, or discard. The findings suggest that physics-based certification can enhance the safety and reliability of traffic state estimation and point to future work on physics-regularized deep learning and validation with real-world data.

Abstract

The expanding role of Artificial Intelligence (AI) in diverse engineering domains highlights the challenges associated with deploying AI models in new operational environments, involving substantial investments in data collection and model training. Rapid application of AI necessitates evaluating the feasibility of utilizing pre-trained models in unobserved operational settings with minimal or no additional data. However, interpreting the opaque nature of AI's black-box models remains a persistent challenge. Addressing this issue, this paper proposes a science-based certification methodology to assess the viability of employing pre-trained data-driven models in new operational environments. The methodology advocates a profound integration of domain knowledge, leveraging theoretical and analytical models from physics and related disciplines, with data-driven AI models. This novel approach introduces tools to facilitate the development of secure engineering systems, providing decision-makers with confidence in the trustworthiness and safety of AI-based models across diverse environments characterized by limited training data and dynamic, uncertain conditions. The paper demonstrates the efficacy of this methodology in real-world safety-critical scenarios, particularly in the context of traffic state estimation. Through simulation results, the study illustrates how the proposed methodology efficiently quantifies physical inconsistencies exhibited by pre-trained AI models. By utilizing analytical models, the methodology offers a means to gauge the applicability of pre-trained AI models in new operational environments. This research contributes to advancing the understanding and deployment of AI models, offering a robust certification framework that enhances confidence in their reliability and safety across a spectrum of operational conditions.

Figures (5)

• Figure 1: Speed against density diagram in traffic flow
• Figure 2: Certification process
• Figure 3: Dataset generated with a $v_f$ =25
• Figure 4: Classification based on Normalized Physics Loss.
• Figure 5: Certification classification table