Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization

Hyeonah Kim, Minsu Kim, Sungsoo Ahn, Jinkyoo Park

TL;DR

This work tackles the inefficiency of DRL in combinatorial optimization when reward evaluations are costly. It introduces symmetric replay training (SRT), a generic two-step method that first performs reward-maximizing training and then imitates symmetric, high-reward trajectories to explore under-explored regions without additional reward calls. By employing symmetric trajectory transformations through maximum-entropy, adversarial, or importance-sampling policies, SRT increases replay usefulness and reduces overfitting, yielding consistent gains across TSP, hardware design, and molecular optimization benchmarks. The approach harmonizes well with both on-policy and off-policy DRL methods and relates to GFlowNets, offering practical, scalable improvements for real-world CO tasks. Overall, SRT significantly enhances sample efficiency and broadens the applicability of DRL in expensive-evaluation CO domains.

Abstract

Deep reinforcement learning (DRL) has significantly advanced the field of combinatorial optimization (CO). However, its practicality is hindered by the necessity for a large number of reward evaluations, especially in scenarios involving computationally intensive function assessments. To enhance the sample efficiency, we propose a simple but effective method, called symmetric replay training (SRT), which can be easily integrated into various DRL methods. Our method leverages high-reward samples to encourage exploration of the under-explored symmetric regions without additional online interactions - free. Through replay training, the policy is trained to maximize the likelihood of the symmetric trajectories of discovered high-rewarded samples. Experimental results demonstrate the consistent improvement of our method in sample efficiency across diverse DRL methods applied to real-world tasks, such as molecular optimization and hardware design.

Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization

TL;DR

This work tackles the inefficiency of DRL in combinatorial optimization when reward evaluations are costly. It introduces symmetric replay training (SRT), a generic two-step method that first performs reward-maximizing training and then imitates symmetric, high-reward trajectories to explore under-explored regions without additional reward calls. By employing symmetric trajectory transformations through maximum-entropy, adversarial, or importance-sampling policies, SRT increases replay usefulness and reduces overfitting, yielding consistent gains across TSP, hardware design, and molecular optimization benchmarks. The approach harmonizes well with both on-policy and off-policy DRL methods and relates to GFlowNets, offering practical, scalable improvements for real-world CO tasks. Overall, SRT significantly enhances sample efficiency and broadens the applicability of DRL in expensive-evaluation CO domains.

Abstract

Deep reinforcement learning (DRL) has significantly advanced the field of combinatorial optimization (CO). However, its practicality is hindered by the necessity for a large number of reward evaluations, especially in scenarios involving computationally intensive function assessments. To enhance the sample efficiency, we propose a simple but effective method, called symmetric replay training (SRT), which can be easily integrated into various DRL methods. Our method leverages high-reward samples to encourage exploration of the under-explored symmetric regions without additional online interactions - free. Through replay training, the policy is trained to maximize the likelihood of the symmetric trajectories of discovered high-rewarded samples. Experimental results demonstrate the consistent improvement of our method in sample efficiency across diverse DRL methods applied to real-world tasks, such as molecular optimization and hardware design.
Paper Structure (65 sections, 2 theorems, 17 equations, 12 figures, 13 tables, 1 algorithm)

This paper contains 65 sections, 2 theorems, 17 equations, 12 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Symmetries in DRL for CO
  4. Data Augmentation
  5. Replay Ratio Scaling
  6. Background
  7. Methodology
  8. Reward-maximizing Training
  9. Symmetric Replay Training
  10. Symmetric transformation policy
  11. Maximum entropy transformation policy.
  12. Adversarial transformation policy.
  13. Importance sampling transformation policy.
  14. Symmetric replay training
  15. Interplay of Reward-maximizing and Symmetric Replay Training
  16. ...and 50 more sections

Key Result

Theorem 4.1

Consider a distribution $\pi_{\theta}(\tau)$ over the trajectory, and $\pi_{\theta}(x)$ over its corresponding solutions. Let $U(\tau_{\rightarrow x} | x)$ denote an uniform distribution. Then the entropy of $\pi_\theta(\tau)$ can be decomposed and upper bounded as follows:

Figures (12)

  • Figure 1: Illustration for two-step training strategies: reward-maximizing training and symmetric replay training.
  • Figure 2: Overview of iterative training
  • Figure 3: Optimization curves on TSP ($N=50$)
  • Figure 4: Increasing replay loop
  • Figure 5: The optimization curve (left) and difference between symmetric trajectories (right) with different transformation polices.
  • ...and 7 more figures

Theorems & Definitions (4)

  • Theorem 4.1
  • Proposition 1.1
  • proof : Proof of Statement 1
  • proof : Proof of Statement 2