Light Future: Multimodal Action Frame Prediction via InstructPix2Pix
Zesen Zhong, Duomin Zhang, Yijia Li
TL;DR
This work reframes robotic action frame prediction as a multimodal forecasting task by fine-tuning InstructPix2Pix, a diffusion-based image editing model, to predict future robot observations from a current image and a text instruction. Using the RoboTwin simulation data, the approach achieves high structural similarity and PSNR (SSIM up to $0.93$ and PSNR up to $39.71$ dB) while dramatically reducing computational requirements relative to video diffusion and transformer baselines, enabling inference from a single image and instruction. The method employs parameter-efficient fine-tuning, progressive resolution training, and DDIM sampling with classifier-free guidance, delivering fast, low-resource future-frame predictions across three robotic tasks (hammer beat, handover, and stacking). This lightweight, multimodal framework supports real-time robotic decision-making and has potential applicability to other motion-trajectory analytics tasks such as sports, where trajectory precision is prioritized over photorealism.
Abstract
Predicting future motion trajectories is a critical capability across domains such as robotics, autonomous systems, and human activity forecasting, enabling safer and more intelligent decision-making. This paper proposes a novel, efficient, and lightweight approach for robot action prediction, offering significantly reduced computational cost and inference latency compared to conventional video prediction models. Importantly, it pioneers the adaptation of the InstructPix2Pix model for forecasting future visual frames in robotic tasks, extending its utility beyond static image editing. We implement a deep learning-based visual prediction framework that forecasts what a robot will observe 100 frames (10 seconds) into the future, given a current image and a textual instruction. We repurpose and fine-tune the InstructPix2Pix model to accept both visual and textual inputs, enabling multimodal future frame prediction. Experiments on the RoboTWin dataset (generated based on real-world scenarios) demonstrate that our method achieves superior SSIM and PSNR compared to state-of-the-art baselines in robot action prediction tasks. Unlike conventional video prediction models that require multiple input frames, heavy computation, and slow inference latency, our approach only needs a single image and a text prompt as input. This lightweight design enables faster inference, reduced GPU demands, and flexible multimodal control, particularly valuable for applications like robotics and sports motion trajectory analytics, where motion trajectory precision is prioritized over visual fidelity.