An Event-Based Perception Pipeline for a Table Tennis Robot

TL;DR

This work tackles the challenge of fast, accurate perception for table tennis robots by introducing the first real-time, fully event-based perception pipeline that relies on two synchronized event cameras and an Exponential Reduced Ordinal Surface (EROS). The system uses a two-thread design to update the EROS surface event-by-event while performing as-fast-as-possible ball detection with a fast Hough circle detector, yielding an update rate about an order of magnitude higher than frame-based methods and robust ball detection independent of motion. The approach achieves comparable 2D localization accuracy to frame-based pipelines and significantly improves the reliability of 3D trajectory estimation, velocity, and spin through higher update rates feeding an EKF-based trajectory predictor. The combination of high temporal resolution, low latency, and improved state uncertainty has direct implications for faster and more reliable robot control in rally scenarios, and the authors make their pipeline publicly accessible for further research.

Abstract

Table tennis robots gained traction over the last years and have become a popular research challenge for control and perception algorithms. Fast and accurate ball detection is crucial for enabling a robotic arm to rally the ball back successfully. So far, most table tennis robots use conventional, frame-based cameras for the perception pipeline. However, frame-based cameras suffer from motion blur if the frame rate is not high enough for fast-moving objects. Event-based cameras, on the other hand, do not have this drawback since pixels report changes in intensity asynchronously and independently, leading to an event stream with a temporal resolution on the order of us. To the best of our knowledge, we present the first real-time perception pipeline for a table tennis robot that uses only event-based cameras. We show that compared to a frame-based pipeline, event-based perception pipelines have an update rate which is an order of magnitude higher. This is beneficial for the estimation and prediction of the ball's position, velocity, and spin, resulting in lower mean errors and uncertainties. These improvements are an advantage for the robot control, which has to be fast, given the short time a table tennis ball is flying until the robot has to hit back.

Figures (8)

• Figure 1: Background: The industrial robot arm of our table tennis robot setup for which the proposed perception pipeline is designed. The two event-based cameras, indicated with orange circles, are mounted on the ceiling. Foreground: The event streams of the two event-based cameras with detected balls on the EROS event surface, visualized in green, the triangulation process (using the ball detections from both cameras and calculating the 3D position given the camera calibration), indicated in violet, and the resulting triangulated 3D trajectory, shown in blue.
• Figure 2: Our camera setup consists of four frame-based cameras (in blue) and two event-based cameras (in red) with baselines of $3$m to $5$m. Only the event-based cameras are used in this work. Schematic is up to scale.
• Figure 3: The first thread of our event-based perception pipeline. We process the incoming event and update the EROS event surface event-by-event.
• Figure 4: The second thread of our event-based perception pipeline. We initialize the ROI of the ball and, afterward, detect the ball within the ROI as-fast-as-possible.
• Figure 5: Example surface using the EROS algorithm to enable velocity-independent representation without temporal parameter tuning: (a) the input events over time for an example scene, (b) a visualization of the EROS surface after integrating event information over the entire time period of the dataset.