Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the definition and importance of interpretability in scientific machine learning

Conor Rowan, Alireza Doostan

TL;DR

The paper addresses the ambiguity of interpretability in scientific machine learning (SciML) and argues that the field has overly equated interpretability with sparsity. It surveys how interpretability has been defined in equation discovery and symbolic regression and contrasts this with broader interpretable ML literature, concluding that sparsity alone is insufficient for scientific understanding. The authors propose a mechanistic, principle-based definition: a model is interpretable if it can be derived from basic physical principles or as an empirical component of a model built from those principles, and they discuss the implications for unifying disparate phenomena under common mechanisms. They emphasize that true interpretability requires prior knowledge and principled grounding, while sparsity can aid discovery but does not guarantee interpretability, suggesting a path toward more meaningful data-driven scientific insights. The work lays groundwork for focusing SciML research on mechanisms and principled integration with established theory rather than pursuing sparsity as a universal proxy for understanding.

Abstract

Though neural networks trained on large datasets have been successfully used to describe and predict many physical phenomena, there is a sense among scientists that, unlike traditional scientific models comprising simple mathematical expressions, their findings cannot be integrated into the body of scientific knowledge. Critics of machine learning's inability to produce human-understandable relationships have converged on the concept of "interpretability" as its point of departure from more traditional forms of science. As the growing interest in interpretability has shown, researchers in the physical sciences seek not just predictive models, but also to uncover the fundamental principles that govern a system of interest. However, clarity around a definition of interpretability and the precise role that it plays in science is lacking in the literature. In this work, we argue that researchers in equation discovery and symbolic regression tend to conflate the concept of sparsity with interpretability. We review key papers on interpretable machine learning from outside the scientific community and argue that, though the definitions and methods they propose can inform questions of interpretability for scientific machine learning (SciML), they are inadequate for this new purpose. Noting these deficiencies, we propose an operational definition of interpretability for the physical sciences. Our notion of interpretability emphasizes understanding of the mechanism over mathematical sparsity. Innocuous though it may seem, this emphasis on mechanism shows that sparsity is often unnecessary. It also questions the possibility of interpretable scientific discovery when prior knowledge is lacking. We believe a precise and philosophically informed definition of interpretability in SciML will help focus research efforts toward the most significant obstacles to realizing a data-driven scientific future.

On the definition and importance of interpretability in scientific machine learning

TL;DR

The paper addresses the ambiguity of interpretability in scientific machine learning (SciML) and argues that the field has overly equated interpretability with sparsity. It surveys how interpretability has been defined in equation discovery and symbolic regression and contrasts this with broader interpretable ML literature, concluding that sparsity alone is insufficient for scientific understanding. The authors propose a mechanistic, principle-based definition: a model is interpretable if it can be derived from basic physical principles or as an empirical component of a model built from those principles, and they discuss the implications for unifying disparate phenomena under common mechanisms. They emphasize that true interpretability requires prior knowledge and principled grounding, while sparsity can aid discovery but does not guarantee interpretability, suggesting a path toward more meaningful data-driven scientific insights. The work lays groundwork for focusing SciML research on mechanisms and principled integration with established theory rather than pursuing sparsity as a universal proxy for understanding.

Abstract

Though neural networks trained on large datasets have been successfully used to describe and predict many physical phenomena, there is a sense among scientists that, unlike traditional scientific models comprising simple mathematical expressions, their findings cannot be integrated into the body of scientific knowledge. Critics of machine learning's inability to produce human-understandable relationships have converged on the concept of "interpretability" as its point of departure from more traditional forms of science. As the growing interest in interpretability has shown, researchers in the physical sciences seek not just predictive models, but also to uncover the fundamental principles that govern a system of interest. However, clarity around a definition of interpretability and the precise role that it plays in science is lacking in the literature. In this work, we argue that researchers in equation discovery and symbolic regression tend to conflate the concept of sparsity with interpretability. We review key papers on interpretable machine learning from outside the scientific community and argue that, though the definitions and methods they propose can inform questions of interpretability for scientific machine learning (SciML), they are inadequate for this new purpose. Noting these deficiencies, we propose an operational definition of interpretability for the physical sciences. Our notion of interpretability emphasizes understanding of the mechanism over mathematical sparsity. Innocuous though it may seem, this emphasis on mechanism shows that sparsity is often unnecessary. It also questions the possibility of interpretable scientific discovery when prior knowledge is lacking. We believe a precise and philosophically informed definition of interpretability in SciML will help focus research efforts toward the most significant obstacles to realizing a data-driven scientific future.
Paper Structure (13 sections, 7 equations, 5 figures, 1 table)

This paper contains 13 sections, 7 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. The concept of interpretability in equation discovery and symbolic regression
  3. Brief literature review
  4. Discussion
  5. Interpretable ML
  6. Brief literature review
  7. Discussion
  8. The definition and importance of interpretability
  9. Failure of the sparsity definition
  10. Novel definition of interpretability
  11. Definition
  12. Discussion
  13. Conclusion

Figures (5)

  • Figure 1: A thought experiment to show that sparsity does not guarantee interpretability in the absence of prior knowledge. In science, we propose that the goal of interpretation is to make a connection between a particular system of interest and the relevant set of underlying principles.
  • Figure 2: Interpretable equations are guaranteed when prior knowledge of mechanisms restricts the choice of basis used in the regression problem.
  • Figure 3: The path to deriving the governing equations for the mechanics of a continuum.
  • Figure 4: Interpreting equations reduces them to fundamental physical principles. The physical principle is analogous to a latent variable that decodes to different equations for different physical systems.
  • Figure 5: Conclusions drawn from investigating the concept of interpretability in SciML. We equate interpretation with the ability to connect terms in an equation to known physical mechanisms.