ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos

TL;DR

ObjectForesight introduces a 3D object-centric forward dynamics framework that predicts future 6-DoF trajectories of rigid objects from short egocentric video, grounding predictions in explicit SE(3) geometry. It combines a geometry-aware encoder with a diffusion transformer to model multimodal, temporally coherent futures, trained on a 2M+-clip dataset curated from EPIC-Kitchens via automated 3D reconstruction and pose estimation. The approach yields accurate and physically plausible 3D motion across unseen objects and scenes, outperforming autoregressive and video-generation baselines and enabling scalable, observation-driven physical reasoning for embodied perception. The work also provides a large-scale data-pipeline and benchmarks, highlighting the value of leveraging monocular geometry and object-centric representations for 3D motion forecasting in real-world settings.

Abstract

Humans can effortlessly anticipate how objects might move or change through interaction--imagining a cup being lifted, a knife slicing, or a lid being closed. We aim to endow computational systems with a similar ability to predict plausible future object motions directly from passive visual observation. We introduce ObjectForesight, a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short egocentric video sequences. Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level, enabling geometrically grounded and temporally coherent predictions that capture object affordances and trajectories. To train such a model at scale, we leverage recent advances in segmentation, mesh reconstruction, and 3D pose estimation to curate a dataset of 2 million plus short clips with pseudo-ground-truth 3D object trajectories. Through extensive experiments, we show that ObjectForesight achieves significant gains in accuracy, geometric consistency, and generalization to unseen objects and scenes, establishing a scalable framework for learning physically grounded, object-centric dynamics models directly from observation. objectforesight.github.io

Figures (4)

• Figure 1: Data curation pipeline from egocentric video to 3D object trajectories. Starting from EPIC-Kitchens action segments, we detect hands and objects, refine masks, and filter for clear manipulations. We then reconstruct an object mesh, recover metric depth and camera geometry, and do 6-DoF pose estimation and tracking. Sliding windows over these tracks yield short, clean, anchor-frame–canonicalized 6-DoF trajectories used to train ObjectForesight.
• Figure 2: Model architecture. Given past pose tokens and their normalized bounding boxes, we summarize motion context with anchor-query attention and use it to guide object-centric pooling in a PointTransformerV3 encoder, producing a geometry-aware scene embedding. A diffusion transformer (DiT, AdaLN-Zero) then denoises future depth-normalized pose tokens, conditioned on the scene embedding and an explicit prefix of past pose tokens. This design allows ObjectForesight to generate diverse, physically coherent, and temporally smooth 3D motion predictions.
• Figure 3: Qualitative results from ObjectForesight. Given only the past context and the anchor-frame geometry, ObjectForesight generates physically plausible and semantically meaningful 6-DoF trajectories of manipulated objects. For each sequence, we overlay 8 predicted poses on the last observed frame, illustrating both (i) the projected coordinate axes and (ii) the transformed object mesh, with increasing blur indicating further steps into the future. The images are zoomed in for clarity, and the arrows indicate the direction of motion. These results and additional results on HOT3D-clips are best viewed as videos in the website https://objectforesight.github.io/
• Figure 4: Visual comparison between generations from ObjectForesight and Luma AI Ray3. Both methods are conditioned on the same three-frame context. ObjectForesight generates future 3D object poses, while Luma AI Ray3 generates a short video. We then apply our pose extraction pipeline to the generated video. Under this procedure, ObjectForesight yields temporally consistent pose trajectories, whereas poses extracted from Ray3 generations are less consistent. Results are best viewed as videos in the website https://objectforesight.github.io/