Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CircuitBuilder: From Polynomials to Circuits via Reinforcement Learning

Weikun K. Zhang, Rohan Pandey, Bhaumik Mehta, Kaijie Jin, Naomi Morato, Archit Ganapule, Michael Ruofan Zeng, Jarod Alper

Abstract

Motivated by auto-proof generation and Valiant's VP vs. VNP conjecture, we study the problem of discovering efficient arithmetic circuits to compute polynomials, using addition and multiplication gates. We formulate this problem as a single-player game, where an RL agent attempts to build the circuit within a fixed number of operations. We implement an AlphaZero-style training loop and compare two approaches: Proximal Policy Optimization with Monte Carlo Tree Search (PPO+MCTS) and Soft Actor-Critic (SAC). SAC achieves the highest success rates on two-variable targets, while PPO+MCTS scales to three variables and demonstrates steady improvement on harder instances. These results suggest that polynomial circuit synthesis is a compact, verifiable setting for studying self-improving search policies.

CircuitBuilder: From Polynomials to Circuits via Reinforcement Learning

Abstract

Motivated by auto-proof generation and Valiant's VP vs. VNP conjecture, we study the problem of discovering efficient arithmetic circuits to compute polynomials, using addition and multiplication gates. We formulate this problem as a single-player game, where an RL agent attempts to build the circuit within a fixed number of operations. We implement an AlphaZero-style training loop and compare two approaches: Proximal Policy Optimization with Monte Carlo Tree Search (PPO+MCTS) and Soft Actor-Critic (SAC). SAC achieves the highest success rates on two-variable targets, while PPO+MCTS scales to three variables and demonstrates steady improvement on harder instances. These results suggest that polynomial circuit synthesis is a compact, verifiable setting for studying self-improving search policies.
Paper Structure (21 sections, 10 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 10 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Arithmetic circuits
  4. AlphaZero-Style Search and Policy Optimization
  5. Monte Carlo Tree Search
  6. Proximal Policy Optimization
  7. Soft Actor-Critic
  8. Methodology
  9. Environment and State Encoding
  10. Game-Board Generation
  11. PPO + MCTS
  12. Supervised Pretraining
  13. PPO Fine-Tuning
  14. MCTS Integration
  15. MCTS-Guided Expert Iteration
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Two arithmetic circuits for the polynomial $x^2 + 2xy+y^2$.
  • Figure 2: PPO+MCTS training plots for complexity 5.
  • Figure 3: PPO+MCTS training plots for complexity 6.
  • Figure 4: Soft Actor-Critic Training Metrics Over Time (MA50) at Complexity level 1-4
  • Figure 5: Soft Actor-Critic Training Metrics Over Time (MA50) at Complexity 5