TKN: Transformer-based Keypoint Prediction Network For Real-time Video Prediction

TL;DR

TKN is the first real-time video prediction solution that achieves a prediction rate of 1,176 fps, significantly reducing computation costs while maintaining other performance, suggesting that TKN has great application potential.

Abstract

Video prediction is a complex time-series forecasting task with great potential in many use cases. However, traditional methods prioritize accuracy and overlook slow prediction speeds due to complex model structures, redundant information, and excessive GPU memory consumption. These methods often predict frames sequentially, making acceleration difficult and limiting their applicability in real-time scenarios like danger prediction and warning.Therefore, we propose a transformer-based keypoint prediction neural network (TKN). TKN extracts dynamic content from video frames in an unsupervised manner, reducing redundant feature computation. And, TKN uses an acceleration matrix to reduce the computational cost of attention and employs a parallel computing structure for prediction acceleration. To the best of our knowledge, TKN is the first real-time video prediction solution that achieves a prediction rate of 1,176 fps, significantly reducing computation costs while maintaining other performance. Qualitative and quantitative experiments on multiple datasets have demonstrated the superiority of our method, suggesting that TKN has great application potential.

Figures (12)

• Figure 1: (a) The sequential prediction scheme generally takes a long time to predict frames due to the sequential scheme. (b) The parallel prediction scheme we propose can greatly accelerate the prediction speed.
• Figure 2: Detailed structure of Keypoint Detector
• Figure 3: Comparison of three different encoder and decoder structures. (a) The structure proposed by minderer2019unsupervised requires more network layers while performing poorly at disentangling keypoints and background information. (b) A structure that can well disentangle keypoints and background information at the cost of complex network architecture and high computation cost. (c) We adopt the well-known skip connection to achieve good performance on information disentangling with simple structure.
• Figure 4: Detailed structure of TKN. Two main modules are the Keypoint Detector and the Predictor marked with the red dashed lines. The predicted frame uses the background information extracted from the last frame of the input. Both the inputting stage and prediction stage allow batch processing (e.g., input multiple frames simultaneously) and thus enable temporal parallelism. Note that the ground truth keypoints information, $P_{real}=(\bar{P}_{t+1},...,\bar{P}_{2t})$, is output by $X_{t+1},...,X_{2t}$ using keypoint detector (excluded from the figure for simplicity).
• Figure 5: Three different attention mechanism structures: (a) is the original dot-product structure, with a computational complexity of $O(l^2 d)$; (b) is the linear attention structure, with a computational complexity of $O(ld^2)$; (c) is the structure we propose, which uses an acceleration matrix A to reduce the computational complexity of the linear transformation matrix L to $O(l(d+l)^2)$.