Distilling Symbolic Priors for Concept Learning into Neural Networks

TL;DR

The paper addresses how to impart human-like inductive biases to neural networks to enable rapid concept learning from few examples. It accomplishes this by distilling a Bayesian prior over symbolic concepts, defined by a probabilistic CFG and instantiated by Goodman’s Rational Rules, into a neural network using Model-Agnostic Meta-Learning (MAML). It generates 10,000 concepts by Dirichlet-sampled CFG rules, trains a prior-trained MLP, and shows close alignment with human data and the Rational Rules model across Medin–Schaffer, Shepard–Hovland–Jenkins, and Medin82 tasks, outperforming a standard network. This demonstrates a practical route to combine symbolic priors with deep learning, enabling structured, data-efficient concept learning and bridging Bayesian and connectionist accounts of cognition.

Abstract

Humans can learn new concepts from a small number of examples by drawing on their inductive biases. These inductive biases have previously been captured by using Bayesian models defined over symbolic hypothesis spaces. Is it possible to create a neural network that displays the same inductive biases? We show that inductive biases that enable rapid concept learning can be instantiated in artificial neural networks by distilling a prior distribution from a symbolic Bayesian model via meta-learning, an approach for extracting the common structure from a set of tasks. By generating the set of tasks used in meta-learning from the prior distribution of a Bayesian model, we are able to transfer that prior into a neural network. We use this approach to create a neural network with an inductive bias towards concepts expressed as short logical formulas. Analyzing results from previous behavioral experiments in which people learned logical concepts from a few examples, we find that our meta-trained models are highly aligned with human performance.

Figures (12)

• Figure 1: Concept learning from examples. The concept underlying the labels is green or triangle: objects are labeled yes if they are green or a triangle and no otherwise. In our experiments, learners are given a set of labeled examples such as these and are then required to predict the labels for an additional set of examples.
• Figure 2: Learning a prior from a Bayesian model and distilling it into a neural network
• Figure 3: DNF grammar from goodman. $i \in \{1,2,... \text{number of features}\}$
• Figure 4: Predictions of prior-trained and standard neural networks vs humans and vs Rational Rules (data from Table \ref{['tab: medinschaffer']}).
• Figure : (a) Humans