Explainable AI: Learning from the Learners

TL;DR

Focusing on discovery, optimization and certification, it is shown how the combination of foundation models and explainability methods allows the extraction of causal mechanisms, guides robust design and control, and supports trust and accountability in high-stakes applications.

Abstract

Artificial intelligence now outperforms humans in several scientific and engineering tasks, yet its internal representations often remain opaque. In this Perspective, we argue that explainable artificial intelligence (XAI), combined with causal reasoning, enables {\it learning from the learners}. Focusing on discovery, optimization and certification, we show how the combination of foundation models and explainability methods allows the extraction of causal mechanisms, guides robust design and control, and supports trust and accountability in high-stakes applications. We discuss challenges in faithfulness, generalization and usability of explanations, and propose XAI as a unifying framework for human-AI collaboration in science and engineering.

Figures (4)

• Figure 1: Summary of the application areas for explainable AI discussed in this study. The three main areas of discovery, optimization and certification (generally associated with science, engineering and auditing) are ranked in terms of the risk of the application and of how easy it is to obtain explanations in that context. For each area we list relevant methods (in bold face) and an example of application in the context of fluid mechanics for vehicle aerodynamic design.
• Figure 2: Autoencoders learn coordinates for parsimonious dynamics. Discovering an effective coordinate system and a parsimonious description for mechanistic behavior is a goal in XAI for scientific discovery. In this schematic, a video of a swinging pendulum is the input to the autoencoder, which parameterizes the kinematics with two latent variables, $\theta$ and $\dot{\theta}$. It is then possible to learn parsimonious models of the dynamics in this latent space, for example using SINDy Champion2019pnas.
• Figure 3: Possible XAI-based framework for discovery and optimization. We illustrate this framework with a fluid-mechanics problem. High-quality data from a range of geometries and flow conditions are used to train a foundation model bommasani2022, which can basically produce flow realizations in new geometries/conditions. The instantaneous fields for the various geometries are expressed on a common reference domain, and the geometry is defined based on a common parameterization, which is provided as an input to the encoder (potentially enabling generalizing to other geometries by properly choosing the parameters). The physical dimensions are encoded into disentangled latent variables via $\beta$-VAEs solera2024beta, and time is predicted by transformers ref_easy, such that each flow case maps into a 2D slice of the latent space. Then, each new flow condition is encoded into a new slice of the 3D latent space. Conditional latent diffusion Du2024CoNFiLD is used to produce new slices in the latent space for the various flow conditions, including unseen ones. When these get decoded, instantaneous realizations of those new conditions are obtained in physical space. Causal analysis martinez-sanchez2024surd can be used to understand physical phenomena for a particular flow case in the latent space, and in this case SHAP can connect physical regions with each of the latent variables Cremades2024Cremades2025, thus providing physical understanding in an interpretable manner. An LLM-based agentic-AI system can be used to explore new phenomena by deciding which new flow conditions need to be investigated with the foundation model, thereby understanding the effect of various flow regimes, scaling laws and the impact of geometry. This will then be again mapped into physical space via SHAP, providing a complete framework for autonomous discovery very relevant for optimization in engineering and science.
• Figure 4: Possible XAI-based framework for certification. a) Shows a certification and auditing process obtained via the performance of the AI system (i.e., "trust by performance"). This pathway to certification and auditing has critical issues, namely limited validity bounds, and understanding of failure modes, unknown biases, and limited liability attribution. b) Shows a certification and auditing process obtained through both performance and explainability (i.e., "trust by understanding"), that addresses some of the issues of "trust by performance". Certification and auditing by performance and explainability allows for traceability and therefore auditing, and enables a better understanding of why a model may fail or not behave as expected, therefore giving the opportunity to address potential biases or unsafe behavior prior of it occurring.