Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing

TL;DR

This work tackles the challenge of reconstructing high-speed video from snapshot compressive imaging by modeling non-local spatial-temporal dependencies with a motion-aware dynamic graph, MadyGraph. It integrates an initial data-adaptive predictor with a graph module that uses motion-guided dynamic sampling, cross-scale node sampling, and global knowledge integration to capture long-range relations efficiently (achieving near-linear $O(N)$ complexity relative to the naive $O(N^2)$ non-local approaches). The approach yields state-of-the-art results on grayscale and color simulated data, as well as convincing results on real SCI measurements, while offering fast inference and broad compatibility with existing backbones. This demonstrates a practical pathway to robust, scalable video SCI reconstruction and informs future work on flexible graph-based non-local modeling in computational imaging.

Abstract

Video snapshot compressive imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Various reconstruction methods have been developed to recover the high-speed video frames from the snapshot measurement. However, most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies, which are critical for video processing. In this paper, we propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance. Specifically, we develop a motion-aware dynamic GNN for better video representation, i.e., represent each node as the aggregation of relative neighbors under the guidance of frame-by-frame motions, which consists of motion-aware dynamic sampling, cross-scale node sampling, global knowledge integration, and graph aggregation. Extensive results on both simulation and real data demonstrate both the effectiveness and efficiency of the proposed approach, and the visualization illustrates the intrinsic dynamic sampling operations of our proposed model for boosting the video SCI reconstruction results. The code and model will be released.

Figures (12)

• Figure 1: (a) An illustration of the spatial-temporal non-local correlations between the central point (yellow point) and related points (red points, dynamically selected by our model) in adjacent three frames, and the motion directions represented by optical flow surrounding the central point. (b) The comparisons of reconstruction quality and inference time by appending our proposed Graph module to various backbones. The same color represents the same backbone; '$\circ$'/ '$\qedsymbol$' denote the results with/ without MadyGraph.
• Figure 2: Principle of video SCI system. The original video frames are modulated by dynamic masks and then compressed by the camera into a snapshot measurement, then decoded by reconstruction algorithms to recover the video.
• Figure 3: Illustration of the whole framework of MadyGraph. The top row of the network takes measurements and masks as input and outputs the initial candidate video through the initial predictor, composed of three 3D convolution blocks and two residual blocks. The bottom network refers to the dynamic graph module, which starts by re-masking and mapping the initial prediction through a 4-layer 3D convolution network, then updates the video representation through graph learning, involving motion-aware dynamic graph sampling and global knowledge integration, and finally aggregates features and outputs the residual details. To this end, the initial prediction and residual information are combined together to achieve the high-quality video.
• Figure 4: Illustration of the sampling strategy and aggregation in our dynamic graph module: 1) The rows from scale 1 to scale N refer to N-cross-scale sampling pipelines. 2) In each pipeline, firstly, the initial sampling is uniformly distributed; then, the dynamic walks of B frames are generated guided by optical flow; finally, the dynamic walks are performed upon the initially sampled positions for dynamic sampling. 3) The bottom row refers to the global semantic-level graph nodes. 4) To this end, the dynamic graph nodes are aggregated and mapped to form the residual fine-grained details of videos.
• Figure 5: Selected reconstructed frames of six benchmark grayscale simulation data of different methods.