Multi-Resolution Haar Network: Enhancing human motion prediction via Haar transform

TL;DR

HaarMoDic addresses the challenge of predicting future 3D human poses by jointly leveraging spatial and temporal information through a 2D Haar transform. Central to the approach is the Multi-Resolution Haar (MR-Haar) block, which enables simultaneous processing across multiple resolutions to capture diverse cues in both axes. The method, supported by DCT-based spectrum manipulation and a velocity-aware loss, achieves state-of-the-art MPJPE on Human3.6M across all evaluation horizons and is validated by comprehensive ablations. This work advances motion prediction by providing a principled way to fuse spatial and temporal signals at multiple resolutions, with practical implications for AR, animation, and autonomous systems, and points toward future integration with attention-based techniques.

Abstract

The 3D human pose is vital for modern computer vision and computer graphics, and its prediction has drawn attention in recent years. 3D human pose prediction aims at forecasting a human's future motion from the previous sequence. Ignoring that the arbitrariness of human motion sequences has a firm origin in transition in both temporal and spatial axes limits the performance of state-of-the-art methods, leading them to struggle with making precise predictions on complex cases, e.g., arbitrarily posing or greeting. To alleviate this problem, a network called HaarMoDic is proposed in this paper, which utilizes the 2D Haar transform to project joints to higher resolution coordinates where the network can access spatial and temporal information simultaneously. An ablation study proves that the significant contributing module within the HaarModic Network is the Multi-Resolution Haar (MR-Haar) block. Instead of mining in one of two axes or extracting separately, the MR-Haar block projects whole motion sequences to a mixed-up coordinate in higher resolution with 2D Haar Transform, allowing the network to give scope to information from both axes in different resolutions. With the MR-Haar block, the HaarMoDic network can make predictions referring to a broader range of information. Experimental results demonstrate that HaarMoDic surpasses state-of-the-art methods in every testing interval on the Human3.6M dataset in the Mean Per Joint Position Error (MPJPE) metric.

Figures (5)

• Figure 1: Visualization of different methods on the virtual graph of joints. Every circle above denotes a single joint which allocates by frame T and position S. T on the axes denotes the temporal axis, representing the transition frame by frame, and S denotes the spatial axis, on behalf of the structural representation of the human body. Approaches vary in how to leverage temporal and spatial information. (a) RNN-based methods focus on the transition within the temporal axis. (b) GCN-based methods first weigh different elements at the same spatial layer and then regress them through the temporal axis. (c) Attention-based methods use attention mechanisms to represent the whole virtual graph, as shown in blue and brown arrows, representing temporal and spatial attention. (d) The proposed methods in this paper use the 2-dimensional Haar transform to transfer the whole graph into a new and high-resolution coordinate.
• Figure 2: An overview of the architecture of HaarMoDic.FC denotes for full-connected layer, and T denotes for transpose operation. DCT and IDCT denote discrete cosine transform and the inverse operation. The detailed explanation of the MR-Haar Block is in section \ref{['MR-Haar_block']}. To get better performances, $m$ MR-Haar blocks are concatenated in the network.
• Figure 3: Visualization of applying 2D Haar transform over the virtual graph. With the nature of the 2D Haar transform, the virtual graph comprises four matrices: Scale, Vertical, Horizontal, and Diagonal coefficients matrices. Scale matrix consists of the average sums, represented as circles with two colors half-half. Vertical matrix mainly contains the vertical high-frequency information, depicted as inner brown and outer blue. Similarly, the Horizontal matrix shows as inner blue and outer brown. The diagonal matrix contains a mixture of diagonal information, represented in green. Hence, we can project the motion sequence coordinate to a two-times high-resolution coordinate.
• Figure 4: Illustration of MR-Haar Block. This figure illustrates one kind of MR-Haar block containing three orders of Haar transform, and the number of the full-connected layers for each order are 4, 2, 1. FC denotes the full-connected layer, and LN denotes for layer normalization. All the spectrums in different orders are merged together at the second from the last stage. The residual pathway is not included.
• Figure 5: Visualization of the qualitative results of our methods. the time $X$ denotes the last frame input. $X+t$ denotes the $t$ milliseconds after the last frame where $t$ is discrete time. The skeleton in red is the grand truth, and the skeleton in blue is the result of prediction.