Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation

TL;DR

The paper addresses controllable video generation for multi-object scenes from a single frame and sparse motion inputs in an unsupervised setting. It introduces YODA, which uses flow-matching-based RIVER as a backbone, enhances it with force embeddings from a sparse optical-flow encoder, and injects controls via cross-attention within a memory-enabled transformer framework. Key contributions include a sparse, tile-based optical-flow encoding, randomized conditioning to improve robustness, and empirical demonstrations on BAIR, CLEVRER, and iPER that YODA matches or surpasses state-of-the-art controllable generation while handling multiple objects and long-range dynamics. The approach enables direct manipulation of object motion without touching them, offering scalable, annotation-free controllable video synthesis with practical implications for planning, segmentation, and interactive video modeling.

Abstract

We propose a novel unsupervised method to autoregressively generate videos from a single frame and a sparse motion input. Our trained model can generate unseen realistic object-to-object interactions. Although our model has never been given the explicit segmentation and motion of each object in the scene during training, it is able to implicitly separate their dynamics and extents. Key components in our method are the randomized conditioning scheme, the encoding of the input motion control, and the randomized and sparse sampling to enable generalization to out of distribution but realistic correlations. Our model, which we call YODA, has therefore the ability to move objects without physically touching them. Through extensive qualitative and quantitative evaluations on several datasets, we show that YODA is on par with or better than state of the art video generation prior work in terms of both controllability and video quality.

Figures (14)

• Figure 1: Examples of videos generated through controlled motions by YODA on the BAIR dataset. Both videos are generated autoregressively by starting from the same single image and then by providing control inputs in the form of 2D shifts (shown as red arrows superimposed to the frames). To play the videos in the first column on the left, view the paper with Acrobat Reader.
• Figure 2: Sparse optical flow (OF) encoder. The dense OF is sampled and tiled in a $16\times 16$ grid. Each OF tile is fed independently to the MLP, and then combined with a learnable positional encoding into a code.
• Figure 3: Violin plots of the local error distributions. Top: the distribution of the errors when using a convolutional encoder of the motion controls (i.e., with a large receptive field). Bottom: the distribution with our encoder (i.e., with a limited receptive field). Our encoder not only reduces the overall average error but also tends to have more errors of smaller magnitude when compared to the convolutional encoder. In contrast, the convolutional encoder shows a distinct long tail in errors of larger magnitude.
• Figure 4: The effect of the number of control vectors when training on the BAIR dataset. The local error (in blue and on the left-hand side) is the error of the optical flow in the neighborhood of the controlled pixel, while the global error (in orange and on the right-hand side) is the average optical flow vector outside of a circle around the interacted pixel. A smaller local error indicates a better response of the model to the control, while a low global error ensures that only the object of interest moves. The two boxes at the bottom of the figure show the regions across which the error is averaged (left - local error, right - global error). Best viewed in color.
• Figure 5: Ablation of the number of control vectors $n_c$ during training. The videos in each column start from the same initial frame and are generated with the same sequence of control vectors. Notice, however, that for $n_c = 100$ one has to use more controls at inference to bridge the gap between training and test settings. With too many control vectors during training the model demonstrates decent control over background objects, but struggles modelling interactions. With too few control vectors the interactions are modelled well, while the model lacks control over background objects. With the optimal $n_c = 5$ we get the best of the two worlds. Use Acrobat Reader to play the videos.