Sample-efficient Reinforcement Learning in Robotic Table Tennis

TL;DR

This work tackles the challenge of sample-efficient reinforcement learning for a real-world robotic table tennis task by embedding learning within a KUKA-based robot system and framing the problem as a one-step MDP. The authors introduce APRG, an actor–critic method where the critic outputs reward parameters for a given ball state and racket action, and the actor is trained via the critic’s gradient to maximize the parametrized reward. They demonstrate strong performance both in a realistic simulator and on the physical robot, achieving accurate returns with fewer than 200 ball contacts and showing robustness to system noise and varied human opponents. The approach leverages trajectory prediction, a parametric reward, and post-optimization of actions, pointing to broad applicability in robotic control problems where low-fidelity models and safety constraints limit extensive exploration.

Abstract

Reinforcement learning (RL) has achieved some impressive recent successes in various computer games and simulations. Most of these successes are based on having large numbers of episodes from which the agent can learn. In typical robotic applications, however, the number of feasible attempts is very limited. In this paper we present a sample-efficient RL algorithm applied to the example of a table tennis robot. In table tennis every stroke is different, with varying placement, speed and spin. An accurate return therefore has to be found depending on a high-dimensional continuous state space. To make learning in few trials possible the method is embedded into our robot system. In this way we can use a one-step environment. The state space depends on the ball at hitting time (position, velocity, spin) and the action is the racket state (orientation, velocity) at hitting. An actor-critic based deterministic policy gradient algorithm was developed for accelerated learning. Our approach performs competitively both in a simulation and on the real robot in a number of challenging scenarios. Accurate results are obtained without pre-training in under $200$ episodes of training. The video presenting our experiments is available at https://youtu.be/uRAtdoL6Wpw.

Figures (8)

• Figure 1: Table tennis robot system with a KUKA Agilus robot. The goal is to learn the orientation and the velocity of the racket at hitting time.
• Figure 2: The figure shows several simulated ball tracks with different starting angles, viewed from the side. The ambiguity is evident by the two black trajectories with the same achieved goal position.
• Figure 3: Simulated example trajectory. The racket is represented by the blue plane.
• Figure 4: Process on the real robot.
• Figure 5: Modified actor-critic model using a parameterized reward.