Autonomous Discovery of the Ising Model's Critical Parameters with Reinforcement Learning

TL;DR

The paper tackles autonomous discovery of the Ising model's critical parameters, including $T_c$ and exponents $(\beta,\gamma,\nu)$, addressing human bias and perturbations that plague traditional finite-size scaling. It introduces AMPPI, a physics-guided reinforcement learning framework that uses an ensemble of RNN dynamic models, Path Integral Control, and Reward-Error Adaptive Variance Control to navigate parameter space and maximize data-collapse rewards. A global data-collapse reward plus an auxiliary $\nu$-targeted reward enable simultaneous estimation of $(T_c,\beta,\gamma,\nu)$ with high precision, while staged optimization improves convergence. The approach yields superior accuracy and robustness compared to CEM and pyfssa and demonstrates transferable learning across lattice structures with sample fine-tuning, suggesting a new AI-driven paradigm for autonomous discovery in critical phenomena and potential extensions to non-equilibrium and quantum systems.

Abstract

Traditional methods for determining critical parameters are often influenced by human factors. This research introduces a physics-inspired adaptive reinforcement learning framework that enables agents to autonomously interact with physical environments, simultaneously identifying both the critical temperature and various types of critical exponents in the Ising model with precision. Interestingly, our algorithm exhibits search behavior reminiscent of phase transitions, efficiently converging to target parameters regardless of initial conditions. Experimental results demonstrate that this method significantly outperforms traditional approaches, particularly in environments with strong perturbations. This study not only incorporates physical concepts into machine learning to enhance algorithm interpretability but also establishes a new paradigm for scientific exploration, transitioning from manual analysis to autonomous AI discovery.

Figures (9)

• Figure 1: Basic framework of the AMPPI algorithm. The framework includes three main components: First, the physical environment defined by the Hamiltonian; Second, the dynamic model that extracts $(O, \Delta T)$ from state-action pairs $(s, a)$ as RNN inputs to predict observable changes, then processes the results to generate new states $s'$; Finally, the agent that samples $K$ action sequences from $\mathcal{N}(\mu, \Sigma_\mathrm{action})$ and generates optimal action $a^*$ through Path Integral Control combined with REAVC mechanism.
• Figure 2: Training convergence of two types of RNN dynamic models showing validation loss evolution for global models (solid line) and local models (dashed line).
• Figure 3: (a) Performance curves of AMPPI and CEM algorithms in the two-dimensional square lattice Ising model (sizes 32 and 64). (b) Critical temperature determination results of AMPPI and pyfssa methods in the two-dimensional square lattice Ising model with different lattice sizes, where the horizontal axis represents initial temperature ($T_{\mathrm{init}}$), the vertical axis represents final temperature ($T_{\mathrm{terminal}}$), and the horizontal dashed line indicates the theoretical critical temperature $T_c = 2/\ln(1+\sqrt{2})$. The inset compares critical temperatures obtained by both algorithms at different lattice sizes ($L$).
• Figure 4: Critical parameter optimization using the AMPPI algorithm in the two-dimensional square lattice Ising model ($L=(32, 64)$). (a-b) Three-parameter optimization with fixed $\nu=1$: (a) parameter search trajectories in $(T_c, \beta, \gamma)$ space, (b) evolution of parameters and reward values over iterations. (c) Evolution of critical exponents $(\gamma, \nu)$, auxiliary reward $R_{\mathrm{aux}}$, and composite reward over iterations at fixed critical temperature, with $R_{\mathrm{global}} = \frac{1}{2}\left[e^{-d(\mathcal{N}(U))} + e^{-d(\mathcal{N}(\tilde{\chi}))}\right]$.
• Figure 5: Transfer learning results of the AMPPI algorithm in finding critical parameters ($T_c$, $\beta$) in the two-dimensional triangular lattice Ising model ($L=(32, 64)$). Blue circles in (a) represent the distribution of critical parameter values found after direct transfer, (b) shows a locally magnified area, where orange triangles represent the distribution of critical parameter values found after 20 iterations of fine-tuning based on physical environment sample data, and black crosses mark theoretical values ($T_c = \frac{4}{\ln(3)}$, $\beta = 0.125$).