FIction: 4D Future Interaction Prediction from Video

TL;DR

FIction addresses the problem of 4D future interaction prediction by fusing egocentric video with explicit 3D scene context to predict where ($\mathcal{F}_o$) and how ($\mathcal{F}_p$) future interactions occur over a horizon of up to $\\tau_f$ seconds. It employs a multimodal transformer to encode past observations into a latent $\\mathbf{\\bar{r}}$, then uses a voxel decoder for locations and a CVAE for pose samples. Evaluations on Ego-Exo4D show over 30% relative gains over autoregressive and 2D-video baselines across multiple tasks, demonstrating the value of 3D environment grounding for long-horizon interaction anticipation. The approach provides a dataset, benchmarks, and a framework to enable assistive robotics, path planning, and AR systems with 4D foresight.

Abstract

Anticipating how a person will interact with objects in an environment is essential for activity understanding, but existing methods are limited to the 2D space of video frames-capturing physically ungrounded predictions of "what" and ignoring the "where" and "how". We introduce FIction for 4D future interaction prediction from videos. Given an input video of a human activity, the goal is to predict which objects at what 3D locations the person will interact with in the next time period (e.g., cabinet, fridge), and how they will execute that interaction (e.g., poses for bending, reaching, pulling). Our novel model FIction fuses the past video observation of the person's actions and their environment to predict both the "where" and "how" of future interactions. Through comprehensive experiments on a variety of activities and real-world environments in EgoExo4D, we show that our proposed approach outperforms prior autoregressive and (lifted) 2D video models substantially, with more than 30% relative gains.

Figures (5)

• Figure 1: Future interaction prediction. When doing a procedure like making milk tea, a person moves around in their environment, interacting with different objects like water dispenser, stove, skillet holder, and cups. Each interaction has an associated body pose, e.g., using two hands when fetching water, extending the body to reach upper wall cabinets. Given an environment and the procedure till a time $t$, we predict all future object interactions (specifically, pooled over the next 3 mins) and the likely body poses during those object interactions. Best viewed in zoom. Only representative object bounding boxes shown for clarity.
• Figure 2: Overview of the FIction approach. The past observation information (video, pose, and environment) is encoded into a multimodal representation (left). The multimodal encoder $\mathcal{L}$ encodes the past observation, and is used to predict the interaction locations using a decoder (top right). We use the past observation encoding, along with the query location, to train a CVAE encoder decoder to generate a location-specific pose distribution conditioned on the past activity (bottom right).
• Figure 3: An interaction instance. We mark a timestamp as an interaction when the hands are within the 3D bounding box of an object referenced in the narration. An LLM is used to match the narration with the objects in the detector's vocabulary, e.g., stove and gas burner.
• Figure 4: Visualization of example results. (Top) Interaction location prediction in a cooking take. Based on the observed input ego, the model is able to correctly predict future interaction locations--- refrigerator, faucet, cabinet. (Middle, bottom): Interaction sequences in a cooking and a bike repair take. Given the past observation information, our model is able to accurately predict future interactions. If the observed sequence shows the person fixing the tire, the model predicts that the tire will be fixed onto the bike at a later stage, at a different spatial location (bottom, left most pose visualization). Best viewed in zoom. Exo view shown for visualization only.
• Figure 5: Comparison of our method with baselines and a cooking (left) and a bike-repair (right) take.