Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition

TL;DR

The paper addresses magnifying subtle facial micro-motions by a Lagrangian approach that reconstructs a time-varying optical-flow field $v(t,x_1,x_2)$ and magnifies selected components. It introduces a double sparse space-time decomposition $v(t,x_1,x_2) \,\approx\, \sum_{k=1}^K G^k(x_1,x_2) \, d^k(t)$ with $d^k(t) = d_1^k(t) d_2^k(t)$, and solves $\ ext{min}_{D,G} ||V - DG||_F^2 + \alpha ||G||_1$ subject to $||d^k||_{2,1} \le \beta$ to localize motion in space and time. It fine-tunes RAFT using ground-truth from variational dense inverse search on CASME II to produce accurate OF for facial micro-motions and constructs an ME dataset by forward-warping frames with the estimated flow. Magnification is performed by forward warping with $\tilde{v}(t) = v(t) + \mu \sum_{k \in \mathcal{I}} d^k(t) G^k$, applied to a displaced triangulated frame and rasterized with barycentric interpolation, enabling unsupervised region-selective visualization with potential for annotation.

Abstract

Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this paper, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is three-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.

Figures (8)

• Figure 1: An illustration showing the results of forward versus backward warping with an example displacement field.
• Figure 2: Illustration of the forward warping process. The image is first triangulated as shown on the left. The triangles are displaced and the warped image is then rasterized from these distorted triangles, see next figure.
• Figure 3: MSE for a sequence on a microexpression sequence after motion compensation. The values are normalized with respect to the MSE of the raw recording. While DISO performs better than the refined RAFT, training improves performance over other stat-of-the-art models trained on Sintel and Kitti.
• Figure 4: Illustration of our motion magnification procedure with the $\left\Vert \cdot \right\Vert_{2,1}$-constraint. The algorithm starts with a sequence of $80$ video frames of size $640 {\times} 480$ illustrated in the top left. The OF field computed for each time step is illustrated in the lower left. Then we use our method to decompose this OF field into $G^1,\ldots,G^9$ shown in the bottom middle as well as the $(d^1_1,d^1_2),\ldots,(d^9_1, d^9_2)$ in the bottom right. In this example, two components, shown by thicker lines in the plot, were selected by thresholding to be micro-movements. The frames are warped accordingly to yield the video sequence in the top right. The pixels from the red and blue lines are plotted over time in the corresponding red and blue boxes. This shows that the motion around the mouth is amplified while leaving the blinking motion unchanged. $\lambda_1 = 0.1, \lambda_2=0.2, \eta=4, \alpha=0.15, \mu=8$
• Figure 5: Illustration of the same magnified video when changing the thresholds to only amplify the strongest motions. This results in the blinking motion being amplified while leaving other facial movements unchanged.$\lambda_1 = 0.3, \lambda_2 = \infty, \alpha=0.15, \mu=8$

Theorems & Definitions (2)

• Remark 1
• Remark 2