Refined Gradient-Based Temperature Optimization for the Replica-Exchange Monte-Carlo Method

TL;DR

The paper tackles the challenge of choosing optimal temperatures in replica-exchange Monte-Carlo to sample multi-modal distributions efficiently. It introduces a refined online gradient-based temperature optimization that reparameterizes the inverse-temperature differences as $L_k=\log\Delta\beta_k$, ensuring monotonic ordering and fixed endpoints, and uses the loss $f_{\text{uni}}(\bm{\beta})$ (the variance of adjacent acceptance rates) with a score-function gradient estimator. The method yields robust convergence to uniform acceptance rates and reduces round-trip times across Ising, XY, and Edwards-Anderson spin systems, outperforming policy-gradient approaches that struggle with constraint violations and tuning. This approach is applicable to both discrete and continuous spin models and offers a practical, automated way to adapttemperatures during RXMC simulations, with potential extensions to spin-glass instances and variance-reduction techniques.

Abstract

The replica-exchange Monte-Carlo (RXMC) method is a powerful Markov-chain Monte-Carlo algorithm for sampling from multi-modal distributions, which are challenging for conventional methods. The sampling efficiency of the RXMC method depends highly on the selection of the temperatures, and finding optimal temperatures remains a challenge. In this study, we propose a refined online temperature selection method by extending the gradient-based optimization framework proposed previously. Building upon the existing temperature update approach, we introduce a reparameterization technique to strictly enforce physical constraints, such as the monotonic ordering of inverse temperatures, which were not explicitly addressed in the original formulation. The proposed method defines the variance of acceptance rates between adjacent replicas as a loss function, estimates its gradient using differential information from the sampling process, and optimizes the temperatures via gradient descent. We demonstrate the effectiveness of our method through experiments on benchmark spin systems, including the two-dimensional ferromagnetic Ising model, the two-dimensional ferromagnetic XY model, and the three-dimensional Edwards-Anderson model. Our results show that the method successfully achieves uniform acceptance rates and reduces round-trip times across the temperature space. Furthermore, our proposed method offers a significant advantage over recently proposed policy gradient method that require careful hyperparameter tuning, while simultaneously preventing the constraint violations that destabilize optimization.

Figures (13)

• Figure 1: Evolution of acceptance rates between adjacent replicas during the optimization process for the 2D Ising model ($L=20$). (a) Results using the policy gradient method over 3000 epochs. (b) Results using the proposed method over 300 epochs.
• Figure 2: Learning curves for the 2D Ising model ($L=20$). (a) Policy gradient method: The negative reward equivalent to the loss function is plotted against the epoch. (b) Proposed method: The loss function is plotted against the epoch.
• Figure 3: The inverse temperatures at different optimization steps in the 2D Ising model ($L=20$). Starting from the initial geometric progression (epoch 0), the optimization process progressively concentrates the inverse temperatures around the value indicated by the dashed line, where the specific heat is maximal. The results are shown every 25 steps.
• Figure 4: Distribution of round-trip times for the 2D Ising model ($L=20$) under four different temperature selection strategies: from above, inverse linear, geometric progression, the strategy resulting from the policy gradient method, the proposed strategy. The y-axis is presented on a logarithmic scale. The density is estimated from $10^7$ MCS evaluation runs for each strategy.
• Figure 5: Box plot of the round-trip time distributions for the 2D Ising model ($L=20$) under four strategies. Outliers, defined as data points falling beyond 1.5 times the IQR from the box, are omitted for clarity.