Investigating Event-Based Cameras for Video Frame Interpolation in Sports

TL;DR

This work investigates the use of event-based cameras for generating sports slow-motion via video frame interpolation. By building a bi-camera setup combining an RGB sensor and an event-based sensor, and aligning the modalities, the study demonstrates that the off-the-shelf TimeLens model can produce plausible slow-motion sequences for indoor tennis. The approach highlights the practical potential of event-based VFI to lower barriers to high-speed sports footage and establishes a foundation for future improvements in alignment, domain-specific interpolation, and downstream sports analytics. Overall, the paper provides a first step toward integrating neuromorphic sensing with deep learning-based VFI to enhance accessibility and analysis of sports video content.

Abstract

Slow-motion replays provide a thrilling perspective on pivotal moments within sports games, offering a fresh and captivating visual experience. However, capturing slow-motion footage typically demands high-tech, expensive cameras and infrastructures. Deep learning Video Frame Interpolation (VFI) techniques have emerged as a promising avenue, capable of generating high-speed footage from regular camera feeds. Moreover, the utilization of event-based cameras has recently gathered attention as they provide valuable motion information between frames, further enhancing the VFI performances. In this work, we present a first investigation of event-based VFI models for generating sports slow-motion videos. Particularly, we design and implement a bi-camera recording setup, including an RGB and an event-based camera to capture sports videos, to temporally align and spatially register both cameras. Our experimental validation demonstrates that TimeLens, an off-the-shelf event-based VFI model, can effectively generate slow-motion footage for sports videos. This first investigation underscores the practical utility of event-based cameras in producing sports slow-motion content and lays the groundwork for future research endeavors in this domain.

Figures (4)

• Figure 1: Bi-camera recording setup for event-based video frame interpolation in sports. We propose a two camera recording setup, including an RGB camera and an event-based camera to capture sports videos, and temporally align and spatially register both cameras. We record video footage of racquet sports characterized by high-speed movements of the ball and racquets, utilizing our specialized setup, and demonstrate the effectiveness of off-the-shelf event-based video frame interpolation techniques in producing slow-motion footage.
• Figure 2: Camera setup. Picture of our bi-camera setup used for sports data collection. We align both the RGB camera and event-based camera on a tripod.
• Figure 3: SSIM values for different time shifts calculated for three interleaved sequences at a frame rate of 40 FPS.
• Figure 4: Qualitative results obtained by Timelens with different input frame rates. The first line shows the original $120$ FPS video, the second the $40$ FPS with two intermediate frame interpolated, the third line a $20$ FPS subsampled with $5$ out of $6$ frames interpolated, and finally a $10$ FPS input video with $11$ frames interpolated (only $6$ shown). It can be seen that the tennis ball is well-placed on all interpolated frames. However, for the racquet, the fast movement is only well interpolated with an initial frame rate of $20$ and $40$ while $10$ FPS does not provide satisfactory results. Original frames are shown in red, while interpolated ones are shown in green.