MS-LSTM: Exploring Spatiotemporal Multiscale Representations in Video Prediction Domain

TL;DR

MS-LSTM tackles the inefficiency of deepening or widening video prediction models by introducing a spatiotemporal multiscale architecture. It combines SMS-LSTM (depth/downsampling with a mirrored pyramid decoder) and MK-LSTM (multi-kernel temporal memory) to expand the spatiotemporal receptive field while controlling training cost. The approach is supported by a theoretical analysis of cost and extensive experiments on Moving MNIST, TaxiBJ, KTH, and German precipitation nowcasting, showing improved performance with competitive or reduced resource usage compared to baseline ConvRNNs and competing multiscale models. The results indicate that explicit multiscale design offers robust long-term predictions and practical efficiency for high-resolution video prediction tasks.

Abstract

The drastic variation of motion in spatial and temporal dimensions makes the video prediction task extremely challenging. Existing RNN models obtain higher performance by deepening or widening the model. They obtain the multi-scale features of the video only by stacking layers, which is inefficient and brings unbearable training costs (such as memory, FLOPs, and training time). Different from them, this paper proposes a spatiotemporal multi-scale model called MS-LSTM wholly from a multi-scale perspective. On the basis of stacked layers, MS-LSTM incorporates two additional efficient multi-scale designs to fully capture spatiotemporal context information. Concretely, we employ LSTMs with mirrored pyramid structures to construct spatial multi-scale representations and LSTMs with different convolution kernels to construct temporal multi-scale representations. We theoretically analyze the training cost and performance of MS-LSTM and its components. Detailed comparison experiments with twelve baseline models on four video datasets show that MS-LSTM has better performance but lower training costs.

Figures (12)

• Figure 1: The architecture of ConvLSTM. It only uses depth to obtain the multi-scale representation.
• Figure 2: The architecture of MS-LSTM. It uses depth, downsampling, and multiple kernels to obtain multi-scale representations. The model performs one-step predictions along the spatial axis (vertical direction) while passing hidden states between layers. The model extrapolates future frames along the time axis (horizontal direction) while passing the hidden and cell states over time. Skip connections ("+") are represented by curves to combine the features of the encoder and decoder at the same scale, enabling the model to generate static or high-frequency features easily denton2018stochastic. The yellow symbols $\tilde{C}_{t}$ represent the newly added multi-scale cell memory.
• Figure 3: The architecture of MK-LSTM. Orange-filled circles denote the differences between MK-LSTM and ConvLSTM (White-filled circles).
• Figure 4: The training cost (params, FLOPs, memory, and time) and performance (MSE) comparison between ConvLSTM, SMS-LSTM, TMS-LSTM, and MS-LSTM. It is a normalized version of the data in Table \ref{['table:mnist-ablation']}. For these five indicators, the closer the model is to the center, the better.
• Figure 5: The layer outputs of ConvLSTM, SMS-LSTM, TMS-LSTM, and MS-LSTM on the Moving MNIST dataset.

Theorems & Definitions (1)

• Theorem 1