Discovering Symbolic Models from Deep Learning with Inductive Biases

TL;DR

The paper tackles the challenge of interpreting deep models by distilling symbolic laws from structured neural architectures. It introduces a framework that imposes strong inductive biases via Graph Networks and then applies symbolic regression to the network’s internal components to extract explicit equations, including force laws and Hamiltonians. Across Newtonian dynamics, Hamiltonian dynamics, and cosmology, the approach recovers known physical relationships and uncovers new analytic expressions, with symbolic models often generalizing better to out-of-distribution data than the original networks. By enforcing compact latent representations and using separable internal functions, the method provides interpretable, data-driven discoveries of physical principles and new analytic insights in astrophysics. The work thus offers a principled path to interpretability and principled discovery in neural representations learned from complex, high-dimensional data.

Abstract

We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn.

Figures (6)

• Figure 1: A cartoon depicting how we extract physical equations from a dataset.
• Figure 2: An illustration of the internal structure of the graph neural network we use in some of our experiments. Note that the comparison to Newtonian mechanics is purely for explanatory purposes, but is not explicit. Differences include: the "forces" (messages) are often high dimensional, the nodes need not be physical particles, $\phi^e$ and $\phi^v$ are arbitrary learned functions, and the output need not be an updated state. However, the rough equivalency between this architecture and physical frameworks allows us to interpret learned formulas in terms of existing physics.
• Figure 3: A diagram showing how we implement and exploit our inductive bias on GNs. A video of this figure during training can be seen by going to the URL \video.
• Figure 4: Examples of a selection of simulations, for 4 nodes and two dimensions. Decreasing transparency shows increasing time, and size of points shows mass.
• Figure 5: The most significant message components of each model compared with a linear combination of the force components: this example, the spring simulation in 2D with eight nodes for training. These plots demonstrate that the GN's messages have learned to be linear transformations of the vector components of the true force, in this case a springlike force, after applying an inductive bias to the messages.