JSTR: Joint Spatio-Temporal Reasoning for Event-based Moving Object Detection

TL;DR

This work discovers that the moving object has a complete columnar structure in the point cloud composed of motion-compensated events along the timestamp, and proposes a novel joint spatio-temporal reasoning method that can effectively detect the moving object from motion confidence and geometric structure.

Abstract

Event-based moving object detection is a challenging task, where static background and moving object are mixed together. Typically, existing methods mainly align the background events to the same spatial coordinate system via motion compensation to distinguish the moving object. However, they neglect the potential spatial tailing effect of moving object events caused by excessive motion, which may affect the structure integrity of the extracted moving object. We discover that the moving object has a complete columnar structure in the point cloud composed of motion-compensated events along the timestamp. Motivated by this, we propose a novel joint spatio-temporal reasoning method for event-based moving object detection. Specifically, we first compensate the motion of background events using inertial measurement unit. In spatial reasoning stage, we project the compensated events into the same image coordinate, discretize the timestamp of events to obtain a time image that can reflect the motion confidence, and further segment the moving object through adaptive threshold on the time image. In temporal reasoning stage, we construct the events into a point cloud along timestamp, and use RANSAC algorithm to extract the columnar shape in the cloud for peeling off the background. Finally, we fuse the results from the two reasoning stages to extract the final moving object region. This joint spatio-temporal reasoning framework can effectively detect the moving object from motion confidence and geometric structure. Moreover, we conduct extensive experiments on various datasets to verify that the proposed method can improve the moving object detection accuracy by 13\%.

Figures (6)

• Figure 1: Illustration of different moving object detection frameworks. (a) Traditional method. (b) The proposed joint spatio-temporal reasoning framework. (c) Corresponding segmentation results. Traditional method reasons the motion confidence distribution from spatial dimension for moving object, while suffering from the incomplete contour. Joint spatio-temporal reasoning can further exploit the structure characteristic of moving object from the temporal event point cloud for structure integrity of moving object.
• Figure 2: Visualization of event frame and point cloud. (a) Compensated event frame. (b) Compensated event point cloud. Moving object has an obvious tailing effect in the event frame, while has a significant columnar structure in the event cloud. This motivates us to improve moving object detection from the structure integrity of moving object.
• Figure 3: The framework of joint spatio-temporal reasoning (JSTR) mainly contains confidence-based spatial reasoning and structure-based temporal reasoning. Motion compensation aligns the background events. Confidence-based spatial reasoning segments the moving object region via adaptive threshold segmentation on time image (namely motion confidence map). Structure-based temporal reasoning further extracts the complete contour of the moving object from the event cloud.
• Figure 4: Visual comparison of moving object detection on public (first row) and self-collected (last two row) datasets.
• Figure 5: Effectiveness of different modules. (a) Event frames with and without motion compensation. (b) Detection results with only threshold segmentation (TS). (c) Detection results with threshold segmentation and structure extraction (SE).