A Comparative Study of Deep Reinforcement Learning Models: DQN vs PPO vs A2C

TL;DR

Addressing how three prominent DRL algorithms—DQN, PPO, and A2C—perform in a controlled Atari BreakOut setting, the paper investigates learning efficiency, strategy development, and adaptability under different hyperparameters including the discount factor $\gamma$. It uses Stable Baselines3 implementations and a fixed frame-based training budget of approximately 20 million frames to ensure fair comparisons while varying learning rates and $\gamma$ values. The results show DQN is robust and efficient across a broad hyperparameter range, whereas PPO and A2C exhibit greater sensitivity but can excel in long-horizon strategy development, depending on $\gamma$ and task complexity. The work provides practical guidance on model selection for DRL tasks and improves reproducibility by releasing code and data on GitHub.

Abstract

This study conducts a comparative analysis of three advanced Deep Reinforcement Learning models: Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and Advantage Actor-Critic (A2C), within the BreakOut Atari game environment. Our research assesses the performance and effectiveness of these models in a controlled setting. Through rigorous experimentation, we examine each model's learning efficiency, strategy development, and adaptability under dynamic game conditions. The findings provide critical insights into the practical applications of these models in game-based learning environments and contribute to the broader understanding of their capabilities. The code is publicly available at github.com/Neilus03/DRL_comparative_study.

Figures (1)

• Figure 1: Performance metrics for DQN, PPO, and A2C models across different settings.