Overcoming Semantic Dilution in Transformer-Based Next Frame Prediction

TL;DR

This work tackles semantic dilution in transformer-based video frame prediction by introducing Semantic Concentration Multi-Head Self-Attention (SCMHSA) and an embedding-space loss to align training with the model's latent outputs. The proposed SC-VFP architecture embeds frames with ViT [CLS], processes them through SCMHSA-enabled encoder blocks, and predicts the next frame as an embedding via an MLP; losses are defined in embedding space rather than on pixel reconstructions. Empirically, SC-VFP delivers strong results on UCSD Pedestrian, UCF Sports, and Penn Action, with notable gains on larger datasets, while showing limited improvement on the small KTH dataset, highlighting scalability benefits and data-size sensitivity. The approach offers a pathway to more semantically faithful predictions and has potential applications in anomaly detection and tracking where embedding-level representations are critical.

Abstract

Next-frame prediction in videos is crucial for applications such as autonomous driving, object tracking, and motion prediction. The primary challenge in next-frame prediction lies in effectively capturing and processing both spatial and temporal information from previous video sequences. The transformer architecture, known for its prowess in handling sequence data, has made remarkable progress in this domain. However, transformer-based next-frame prediction models face notable issues: (a) The multi-head self-attention (MHSA) mechanism requires the input embedding to be split into $N$ chunks, where $N$ is the number of heads. Each segment captures only a fraction of the original embeddings information, which distorts the representation of the embedding in the latent space, resulting in a semantic dilution problem; (b) These models predict the embeddings of the next frames rather than the frames themselves, but the loss function based on the errors of the reconstructed frames, not the predicted embeddings -- this creates a discrepancy between the training objective and the model output. We propose a Semantic Concentration Multi-Head Self-Attention (SCMHSA) architecture, which effectively mitigates semantic dilution in transformer-based next-frame prediction. Additionally, we introduce a loss function that optimizes SCMHSA in the latent space, aligning the training objective more closely with the model output. Our method demonstrates superior performance compared to the original transformer-based predictors.

Figures (5)

• Figure 1: Multi-head Self-attention (MHSA) block in VFP divides the input embedding $x_i$ into $N$ chunks $\{x^{h_1}_i, x^{h_2}_i, ..., x^{h_N}_i\}$ where $N$ is the number of attention heads. This can distort the learned embeddings, leading to semantic dilution of the frame representation.
• Figure 2: Overview of our proposed network model. We first extract embeddings from $M$ past video frames using the ViT model dosovitskiy2020image, utilizing only the classification token [CLS] to represent each frame. Positional encoding is then added, followed by the application of the Semantic Concentration VFP (SC-VFP) module to integrate temporal information. The SC-VFP module includes the SCMHSA block, designed to address the semantic dilution issue in Transformer-based VFP systems. Finally, the processed data is passed through an MLP prediction layer to generate the predicted embedding for the next frame.
• Figure 3: Qualitative comparison of methods on an example frame from KTH dataset. The top shows the error map of the predicted embedding and the actual embedding. The darker the error map appears, the worse the prediction is to the ground truth. The bottom shows the reconstructed frame using the predicted embedding.
• Figure 4: Consine similarity between the first five predicted embeddings and ground truth embeddings of all methods on KTH dataset. A higher cosine similarity indicates that the prediction is closer to the ground truth.
• Figure 5: Convergence speed of SC-VFP with and without SCMHSA on the training set of Penn Action dataset. SC-VFP with SCMHSA not only achieves a lower loss but also converges more quickly