What If : Understanding Motion Through Sparse Interactions

TL;DR

Flow Poke Transformer (FPT) directly models the distribution of local scene motion conditioned on sparse interactions, addressing the multimodality and uncertainty inherent in real-world dynamics. By outputting a full-covariance Gaussian Mixture Model for motion at query points and employing a query-causal transformer with sparse poke/query tokens, FPT enables efficient, interpretable multimodal predictions and downstream tasks such as dense motion prediction and moving-part segmentation. The model is pretrained on diverse web videos, demonstrates strong uncertainty calibration, and shows competitive or superior performance on face-motion and articulated-object tasks, especially after fine-tuning on out-of-domain data. This approach provides a practical, real-time, probabilistic framework for interactive motion understanding with broad applicability in robotics, animation, and perception.

Abstract

Understanding the dynamics of a physical scene involves reasoning about the diverse ways it can potentially change, especially as a result of local interactions. We present the Flow Poke Transformer (FPT), a novel framework for directly predicting the distribution of local motion, conditioned on sparse interactions termed "pokes". Unlike traditional methods that typically only enable dense sampling of a single realization of scene dynamics, FPT provides an interpretable directly accessible representation of multi-modal scene motion, its dependency on physical interactions and the inherent uncertainties of scene dynamics. We also evaluate our model on several downstream tasks to enable comparisons with prior methods and highlight the flexibility of our approach. On dense face motion generation, our generic pre-trained model surpasses specialized baselines. FPT can be fine-tuned in strongly out-of-distribution tasks such as synthetic datasets to enable significant improvements over in-domain methods in articulated object motion estimation. Additionally, predicting explicit motion distributions directly enables our method to achieve competitive performance on tasks like moving part segmentation from pokes which further demonstrates the versatility of our FPT. Code and models are publicly available at https://compvis.github.io/flow-poke-transformer.

Figures (15)

• Figure 1: What If: Our Flow Poke Transformer directly models the uncertainty of the world by predicting distributions of how objects ($\color{matplotliborange}\boldsymbol{\times}$) may move conditioned on some input movements (pokes, $\rightarrow$). We see that whether the hand (below paw) or the paw (above hand) moves downwards directly influences the other's movement. Left: the paw pushing the hand down, will force the hand downwards, resulting in a unimodal distribution. Right: the hand moving down results in two modes, the paw following along or staying put.
• Figure 2: High-level Model Architecture Overview. Given an image $\mathcal{I}$, a set of given pokes $\mathcal{P}$ (visualized as arrows $\rightarrow$), and query positions $\mathbf{q}$ ($\!\color{matplotliborange}\times\!$), our model directly predicts an explicit distribution of the movement at each query position. The flow poke transformer cross-attends to features from a jointly trained image encoder to incorporate visual information. Crucially, our architecture represents movement at individual points $\mathbf{q}$ (enabling sparse & off-grid motion processing) and directly predicts continuous, multimodal output distributions.
• Figure 3: Multimodal Motion Distribution Prediction. We condition on one or multiple pokes ($\rightarrow$) and then query the motion distribution of specific points ($\boldsymbol{\times}$). Our model's predictions capture the multi-modal nature of motion and exhibit understanding of interactions, such as only lifting the cup by its handle not necessarily causing the whole cup to move upwards, while grabbing it at stable points does. It also demonstrates prior understanding from scenes, such as a car in an intersection being more likely to move forwards than backward and cars in traffic likely moving together.
• Figure 4: Predicted Mode Analysis. Values are computed at a resolution of $64^2$. (a) Diversity of predicted modes is high, with mode variation covering a large fraction of poke magnitude. (b) One mode typically has a substantially higher confidence than others, which increases with given poke count. The mode closest to the ground truth consistently has a higher-than-average confidence. (c) More confident modes are more accurate as measured by PCK.
• Figure 5: Uncertainty Calibration. We find that the motion prediction error measured by EPE strongly correlates with the predicted uncertainty (Pearson $\rho \!=\! {0.64}$). This relationship holds for low & high numbers of given pokes.