Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Symbolic Optimization: Reinforcement Learning for Symbolic Mathematics

Conor F. Hayes, Felipe Leno Da Silva, Jiachen Yang, T. Nathan Mundhenk, Chak Shing Lee, Jacob F. Pettit, Claudio Santiago, Sookyung Kim, Joanne T. Kim, Ignacio Aravena Solis, Ruben Glatt, Andre R. Goncalves, Alexander Ladd, Ahmet Can Solak, Thomas Desautels, Daniel Faissol, Brenden K. Petersen, Mikel Landajuela

TL;DR

DSO reframes symbolic optimization as a sequential decision problem, employing an autoregressive token model to generate symbolic expressions while leveraging in-situ priors, constraints, and continuous constant optimization. Through reinforcement learning, imitation learning (priority queue training and GP seeding), and large-scale pre-training, DSO achieves strong performance on symbolic regression benchmarks, culminating in the unified uDSO framework that integrates multiple components for state-of-the-art results. The work demonstrates that combining neural-guided search with symbolic reasoning yields interpretable, physically meaningful models and identifies clear directions for scalable, multi-objective, and data-driven symbolic discovery. Overall, DSO provides a versatile, modular approach to automate symbolic discovery with practical impact across scientific domains.

Abstract

Deep Symbolic Optimization (DSO) is a novel computational framework that enables symbolic optimization for scientific discovery, particularly in applications involving the search for intricate symbolic structures. One notable example is equation discovery, which aims to automatically derive mathematical models expressed in symbolic form. In DSO, the discovery process is formulated as a sequential decision-making task. A generative neural network learns a probabilistic model over a vast space of candidate symbolic expressions, while reinforcement learning strategies guide the search toward the most promising regions. This approach integrates gradient-based optimization with evolutionary and local search techniques, and it incorporates in-situ constraints, domain-specific priors, and advanced policy optimization methods. The result is a robust framework capable of efficiently exploring extensive search spaces to identify interpretable and physically meaningful models. Extensive evaluations on benchmark problems have demonstrated that DSO achieves state-of-the-art performance in both accuracy and interpretability. In this chapter, we provide a comprehensive overview of the DSO framework and illustrate its transformative potential for automating symbolic optimization in scientific discovery.

Deep Symbolic Optimization: Reinforcement Learning for Symbolic Mathematics

TL;DR

DSO reframes symbolic optimization as a sequential decision problem, employing an autoregressive token model to generate symbolic expressions while leveraging in-situ priors, constraints, and continuous constant optimization. Through reinforcement learning, imitation learning (priority queue training and GP seeding), and large-scale pre-training, DSO achieves strong performance on symbolic regression benchmarks, culminating in the unified uDSO framework that integrates multiple components for state-of-the-art results. The work demonstrates that combining neural-guided search with symbolic reasoning yields interpretable, physically meaningful models and identifies clear directions for scalable, multi-objective, and data-driven symbolic discovery. Overall, DSO provides a versatile, modular approach to automate symbolic discovery with practical impact across scientific domains.

Abstract

Deep Symbolic Optimization (DSO) is a novel computational framework that enables symbolic optimization for scientific discovery, particularly in applications involving the search for intricate symbolic structures. One notable example is equation discovery, which aims to automatically derive mathematical models expressed in symbolic form. In DSO, the discovery process is formulated as a sequential decision-making task. A generative neural network learns a probabilistic model over a vast space of candidate symbolic expressions, while reinforcement learning strategies guide the search toward the most promising regions. This approach integrates gradient-based optimization with evolutionary and local search techniques, and it incorporates in-situ constraints, domain-specific priors, and advanced policy optimization methods. The result is a robust framework capable of efficiently exploring extensive search spaces to identify interpretable and physically meaningful models. Extensive evaluations on benchmark problems have demonstrated that DSO achieves state-of-the-art performance in both accuracy and interpretability. In this chapter, we provide a comprehensive overview of the DSO framework and illustrate its transformative potential for automating symbolic optimization in scientific discovery.
Paper Structure (39 sections, 2 theorems, 20 equations, 11 figures, 4 tables)

This paper contains 39 sections, 2 theorems, 20 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The Deep Symbolic Optimization Framework
  3. Defining the Discovery Problem
  4. Autoregressive Modeling for Combinatorial Search
  5. Integrating Priors and Constraints to Prune the Search Space
  6. The Challenge of Constant Optimization in Symbolic Expressions
  7. Training Strategies for Deep Symbolic Optimization
  8. Reinforcement Learning for Sequence Generation
  9. Framing Discrete Optimization as a Markov Decision Process
  10. Policy Gradient Methods
  11. Risk-Seeking Policy Gradient Approaches
  12. Leveraging Expert Demonstrations through Imitation Learning
  13. Priority Queue Training for Enhanced Learning
  14. Incorporating Genetic Programming
  15. Scaling Up: Large-Scale Pre-Training and Synthetic Demonstrations
  16. ...and 24 more sections

Key Result

Lemma 1

(Dual Problem) Assume that for all $\tau \in \mathcal{T}$, there exists $\theta_{\tau} \in \mathbb{R}^M$ such that $p(\tau | \theta_{\tau})$ is the Dirac delta distribution $\delta_{\tau}$. Assume also that there exists a unique global maximum $\tau^{\star} = \mathop{\mathrm{arg\,max}}\limits_{\tau Then, we have that $\tau^{\star} = \tau^{\star}_p$, i.e., the optimization problem (eq:hard_problem

Figures (11)

  • Figure 1: DSO overview. A. Example of sampling an expression from the generative model. In the figure, we use a Recurrent Neural Network (RNN). For each token, the RNN emits a categorical distribution over tokens, a token is sampled, and the parent and sibling of the next token are used as the next input to the RNN. In this example, the sampled expression is $\sin(cx)/\log(y)$, where the value of the constant $c$ is optimized with respect to an input dataset. Starting at the root node, a token is sampled from the emitted categorical distribution. Subsequent tokens are sampled autoregressively until the tree is complete (i.e., all tree branches reach terminal nodes). The resulting sequence of tokens is the tree's pre-order traversal, which can be used to reconstruct the tree and instantiate its corresponding expression. Colors correspond to the number of children for each token. White circles represent empty tokens. B. The library of tokens. C. The expression tree sampled in A. Figure reproduced from petersen2019deep.
  • Figure 2: Pruning the search tree in neural-guided search via in situ constraints. The AR model (RNN in this case) produces logits for each token in the library. Priors and constraints can be learned or user defined, and, are used to adjust the logits guiding the search towards well-performing sequences. The adjusted logits are used to sample the next token in the sequence. The process is repeated until the sequence is complete. Figure reproduced from petersen2021incorporating.
  • Figure 3: DisCo-DSO overview (application to decision tree search). A. DisCo-DSO augments the autoregressive model with a continuous distribution over floating-point values that is conditioned on the discrete tokens. The observations are also augmented with the continuous values. B. Resulting decision tree sampled by DisCo-DSO. The continuous values are optimized jointly with the discrete tokens. Figure reproduced from pettit2024discodsocouplingdiscretecontinuous. To be compared with \ref{['fig:dso']}.
  • Figure 4: Relaxation and Dual Formulation of the Discrete Optimization Problem. This figure illustrates the dual formulation of a discrete optimization problem in a toy example where the search space consists of four possible solutions: $\mathcal{T}=\{(0,0),(1,0),(0,1),(1,1)\}$. (a) The black-box discrete objective function $R(\tau)$ assigns rewards to each point. (b) The generative model $p(\tau|\theta)=\mathcal{B}(\tau_1|\theta_1)\,\mathcal{B}(\tau_2|\theta_2)$ with $\theta_1=0.8,\theta_2=0.2$. (c) Relaxation via expected reward under the model. (d) Entropy regularization (negative prior divergence). (e) $(1-\epsilon)$-quantile constraint on the reward. (f) Expectation conditioned on that quantile.
  • Figure 5: Training DSO with genetic programming population seeding. The AR model (a RNN in this case) generates $N$ samples, e.g. expression for symbolic regression. These samples are used as the start population for a GP component. GP then runs for $S$ generations. The top $M$ samples from GP are extracted, combined with the $N$ samples from the RNN, and are used to train the RNN. Since GP is stateless, it runs in a random restart-like fashion each time the RNN is sampled. Figure reproduced from mundhenk2021symbolic.
  • ...and 6 more figures

Theorems & Definitions (4)

  • Lemma 1
  • proof
  • Lemma 2
  • proof