Motion-Aware Optical Camera Communication with Event Cameras

TL;DR

This work tackles the bottlenecks of optical camera communication in dynamic scenes by substituting CMOS cameras with event cameras and introducing a dynamic marker designed for displays. The authors present a full end-to-end pipeline—pre-processing, detection, tracking, and decoding—that leverages the high temporal resolution of event streams to achieve robust localization and data streaming despite motion and screen refresh. Key contributions include the dynamic marker design, event-based detection/tracking rules, and a decoding strategy that sustains throughput up to 114 Kbps with centimeter-level localization accuracy and about 1% bit error rate under motion, demonstrated through real-world experiments and AR scenarios. The results indicate significant improvements over frame-based baselines, underscoring the method's potential for private, high-rate OCC in dynamic environments and highlighting the need for faster displays to further scale throughput.

Abstract

As the ubiquity of smart mobile devices continues to rise, Optical Camera Communication systems have gained more attention as a solution for efficient and private data streaming. This system utilizes optical cameras to receive data from digital screens via visible light. Despite their promise, most of them are hindered by dynamic factors such as screen refreshing and rapid camera motion. CMOS cameras, often serving as the receivers, suffer from limited frame rates and motion-induced image blur, which degrade overall performance. To address these challenges, this paper unveils a novel system that utilizes event cameras. We introduce a dynamic visual marker and design event-based tracking algorithms to achieve fast localization and data streaming. Remarkably, the event camera's unique capabilities mitigate issues related to screen refresh rates and camera motion, enabling a high throughput of up to 114 Kbps in static conditions, and a 1 cm localization accuracy with 1% bit error rate under various camera motions.

Figures (6)

• Figure 1: The overview of the proposed event-based optical camera communication system. The original message is encoded and displayed on media. The modulated visible lights are captured by the event camera as the receiver. Through our designed detector, tracker and decoder, the camera pose and received message can be recovered at the same time.
• Figure 2: Dynamic Marker Design. The payload cells (in green) carry the transmitted information. Cells marked by purple and teal serve as the interior and exterior locators, respectively. We alternately display data frames and blank frames on a black background like a slide show. Note that the effective payload and the interior locator are dynamic, while the exterior locator remains static. Colors and each cell's boundary are used only for clarity.
• Figure 3: Schematic diagram of the process of detector, tracker and decoder. Blue arrows stand for negative events while red ones are positive events.
• Figure 4: The horizontal axis is the payload size on the marker and the vertical axis illustrates the two evaluation metrics. Different colors correspond to various lighting conditions. The box plot outlines the range between the first quartile to the third, where the median is marked by a line.
• Figure 5: Comparison between estimated camera poses and ground truth for the trajectory of seq. 3.