Learning Tennis Strategy Through Curriculum-Based Dueling Double Deep Q-Networks

TL;DR

The paper develops a reinforcement learning framework to optimize tennis strategy within a full-match, fatigue-aware simulator using a Dueling Double Deep Q-Network and curriculum learning. It demonstrates strong win rates across opponents and provides detailed ablations showing the necessity of the dueling architecture and curriculum, while revealing a pronounced defensive bias driven by reward design and environmental simplifications. The study emphasizes that high in-simulation win rates do not imply human-like play or transfer to real coaching, and argues for multi-objective optimization, improved opponent modeling, and more realistic physics to bridge the gap to real-world applicability. Overall, the work provides a comprehensive methodology and critical insights into the challenges of applying deep RL to sports, and outlines concrete future directions to enhance realism and practical impact.

Abstract

Tennis strategy optimization is a challenging sequential decision-making problem involving hierarchical scoring, stochastic outcomes, long-horizon credit assignment, physical fatigue, and adaptation to opponent skill. I present a reinforcement learning framework that integrates a custom tennis simulation environment with a Dueling Double Deep Q-Network(DDQN) trained using curriculum learning. The environment models complete tennis scoring at the level of points, games, and sets, rally-level tactical decisions across ten discrete action categories, symmetric fatigue dynamics, and a continuous opponent skill parameter. The dueling architecture decomposes action-value estimation into state-value and advantage components, while double Q-learning reduces overestimation bias and improves training stability in this long-horizon stochastic domain. Curriculum learning progressively increases opponent difficulty from 0.40 to 0.50, enabling robust skill acquisition without the training collapse observed under fixed opponents. Across extensive evaluations, the trained agent achieves win rates between 98 and 100 percent against balanced opponents and maintains strong performance against more challenging opponents. Serve efficiency ranges from 63.0 to 67.5 percent, and return efficiency ranges from 52.8 to 57.1 percent. Ablation studies demonstrate that both the dueling architecture and curriculum learning are necessary for stable convergence, while a standard DQN baseline fails to learn effective policies. Despite strong performance, tactical analysis reveals a pronounced defensive bias, with the learned policy prioritizing error avoidance and prolonged rallies over aggressive point construction. These results highlight a limitation of win-rate driven optimization in simplified sports simulations and emphasize the importance of reward design for realistic sports reinforcement learning.

Figures (10)

• Figure 1: Training curve for the Vanilla DQN baseline. Rewards remain negative and unstable, and no consistent improvement emerges.
• Figure 2: Vanilla DQN action distribution as a bar chart. The policy is dominated by defensive_lob, revealing a degenerate survival-oriented strategy.
• Figure 3: Training curve for the Dueling DDQN agent. Rewards and win rate climb steadily and stabilise at high values.
• Figure 4: Direct comparison of training performance for Vanilla DQN vs. Dueling DDQN. The Vanilla DQN remains unstable and negative, whereas Dueling DDQN converges to consistently high rewards.
• Figure 5: Win rate of the Dueling DDQN agent across opponent skill levels. Performance remains near-perfect even as the opponent becomes stronger.