Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparative Analysis of Parameterized Action Actor-Critic Reinforcement Learning Algorithms for Web Search Match Plan Generation

Ubayd Bapoo, Clement N Nyirenda

TL;DR

The paper tackles efficient policy learning in parameterized action spaces for match plan generation in web-scale search tasks. It systematically compares three modern PA-AC algorithms—SAC, GAC, and TQC—across fully observable Platform-v0 and Goal-v0 benchmarks, with hyperparameter optimization via Microsoft NNI. PAGAC emerges as the top performer, delivering the fastest convergence and strongest evaluation returns while maintaining stability in high-dimensional action spaces. The results advocate PAGAC for rapid, robust decision-making in complex hybrid action settings and point to future work combining entropy-regularization with truncation-based methods to further boost stability and generalizability.

Abstract

This study evaluates the performance of Soft Actor Critic (SAC), Greedy Actor Critic (GAC), and Truncated Quantile Critics (TQC) in high-dimensional decision-making tasks using fully observable environments. The focus is on parametrized action (PA) spaces, eliminating the need for recurrent networks, with benchmarks Platform-v0 and Goal-v0 testing discrete actions linked to continuous action-parameter spaces. Hyperparameter optimization was performed with Microsoft NNI, ensuring reproducibility by modifying the codebase for GAC and TQC. Results show that Parameterized Action Greedy Actor-Critic (PAGAC) outperformed other algorithms, achieving the fastest training times and highest returns across benchmarks, completing 5,000 episodes in 41:24 for the Platform game and 24:04 for the Robot Soccer Goal game. Its speed and stability provide clear advantages in complex action spaces. Compared to PASAC and PATQC, PAGAC demonstrated superior efficiency and reliability, making it ideal for tasks requiring rapid convergence and robust performance. Future work could explore hybrid strategies combining entropy-regularization with truncation-based methods to enhance stability and expand investigations into generalizability.

Comparative Analysis of Parameterized Action Actor-Critic Reinforcement Learning Algorithms for Web Search Match Plan Generation

TL;DR

The paper tackles efficient policy learning in parameterized action spaces for match plan generation in web-scale search tasks. It systematically compares three modern PA-AC algorithms—SAC, GAC, and TQC—across fully observable Platform-v0 and Goal-v0 benchmarks, with hyperparameter optimization via Microsoft NNI. PAGAC emerges as the top performer, delivering the fastest convergence and strongest evaluation returns while maintaining stability in high-dimensional action spaces. The results advocate PAGAC for rapid, robust decision-making in complex hybrid action settings and point to future work combining entropy-regularization with truncation-based methods to further boost stability and generalizability.

Abstract

This study evaluates the performance of Soft Actor Critic (SAC), Greedy Actor Critic (GAC), and Truncated Quantile Critics (TQC) in high-dimensional decision-making tasks using fully observable environments. The focus is on parametrized action (PA) spaces, eliminating the need for recurrent networks, with benchmarks Platform-v0 and Goal-v0 testing discrete actions linked to continuous action-parameter spaces. Hyperparameter optimization was performed with Microsoft NNI, ensuring reproducibility by modifying the codebase for GAC and TQC. Results show that Parameterized Action Greedy Actor-Critic (PAGAC) outperformed other algorithms, achieving the fastest training times and highest returns across benchmarks, completing 5,000 episodes in 41:24 for the Platform game and 24:04 for the Robot Soccer Goal game. Its speed and stability provide clear advantages in complex action spaces. Compared to PASAC and PATQC, PAGAC demonstrated superior efficiency and reliability, making it ideal for tasks requiring rapid convergence and robust performance. Future work could explore hybrid strategies combining entropy-regularization with truncation-based methods to enhance stability and expand investigations into generalizability.

Paper Structure

This paper contains 11 sections, 4 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Match Plan Generation Overview
  3. Overview on Actor-critic Reinforcement Learning
  4. RL Fundamentals
  5. Actor-critic Techniques
  6. Emerging Deep Learning based Actor-Critic Techniques
  7. Parameterised Action Actor-Critic Algorithms
  8. Simulation Setup and Results
  9. Platform Game Description and Results
  10. Robot Soccer Goal Game Description and Results
  11. Conclusion and Future Works

Figures (1)

  • Figure 1: Illustration of the telescoping design used in Bing's retrieval mechanism.