Robotic Table Tennis: A Case Study into a High Speed Learning System

TL;DR

The paper presents a real-world, high-speed robotic table tennis platform that integrates fast perception, low-latency control, a flexible Gym-like simulator, and autonomous real-world resets to enable hours of continuous, autonomous training. It introduces latency-aware simulation, robust ball tracking from Bayer images via temporal convolutional detectors, and task-space policy representations trained with Blackbox Gradient Sensing, achieving notable zero-shot transfer from simulation to the physical robot. Through extensive system studies, the work demonstrates the critical role of latency modeling, environment parameterization, and perception robustness in end-to-end performance, while also showing that automatic resets and modular software enable rapid experimentation. The results offer a practical blueprint for scalable, real-world, high-speed robotic learning systems and provide insights applicable to other agile robotic tasks beyond table tennis.

Abstract

We present a deep-dive into a real-world robotic learning system that, in previous work, was shown to be capable of hundreds of table tennis rallies with a human and has the ability to precisely return the ball to desired targets. This system puts together a highly optimized perception subsystem, a high-speed low-latency robot controller, a simulation paradigm that can prevent damage in the real world and also train policies for zero-shot transfer, and automated real world environment resets that enable autonomous training and evaluation on physical robots. We complement a complete system description, including numerous design decisions that are typically not widely disseminated, with a collection of studies that clarify the importance of mitigating various sources of latency, accounting for training and deployment distribution shifts, robustness of the perception system, sensitivity to policy hyper-parameters, and choice of action space. A video demonstrating the components of the system and details of experimental results can be found at https://youtu.be/uFcnWjB42I0.

Figures (16)

• Figure 1: The physical robotic table tennis system. Images from left to right show (I) ball thrower, (II) entire system (thrower, arm, gantry), (III) automatic ball refill, (inlay) simulator, and (IV) robot mid-swing.
• Figure 2: Overview of the components for running simulated and real environments. The diagram on the left shows how the various software components fit to form the environment: in simulation, everything runs in a single process, but the real environment splits the work among several. The diagram on the right shows the components of the real hardware system. A custom MPI manages communication between the parts and logging of all data.
• Figure 3: Quantification of triangulation bias over the length of playing area (y-position) at a height of 250mm above the center line. The more orthogonal viewpoints offered by placing cameras on opposite sides of the tables lead to an order of magnitude reduction in triangulation bias.
• Figure 4: Ball Detection. These synchronized images (cropped to approximately 50% normal size) show the temporal convolutional network detecting the ball (detected ball center in pixels) independently from cameras on both sides of the table. These detections are triangulated and used for 3D tracking.
• Figure 5: Effect of simulator parameters on zero-shot sim-to-real transfer. Policies are sensitive to latency and physical parameter values, yet surprisingly robust to ball observation noise and changes in the ball distribution. Charts show the mean (with 95% CIs) zero-shot sim-to-real transfer. 2.0 is a perfect score with a policy returning all balls. R = restitution coefficient.