Robot Shape and Location Retention in Video Generation Using Diffusion Models

TL;DR

This work tackles the difficulty of preserving robot shape and location in diffusion-model–based video generation. It introduces robot pose embedding and SPADE-based semantic mask regulation within a ConvNext backbone to tighten shape and spatial fidelity, enabling synthetic data for dangerous human–robot interaction detection. Across three scenes with two robot types, the approach yields higher SSIM and improved shape and location retention (Hu moments and IoU) compared to a SinFusion baseline. The findings suggest strong practical value for generating ethical, legally compliant training data for collision-detection models and related safety applications.

Abstract

Diffusion models have marked a significant milestone in the enhancement of image and video generation technologies. However, generating videos that precisely retain the shape and location of moving objects such as robots remains a challenge. This paper presents diffusion models specifically tailored to generate videos that accurately maintain the shape and location of mobile robots. This development offers substantial benefits to those working on detecting dangerous interactions between humans and robots by facilitating the creation of training data for collision detection models, circumventing the need for collecting data from the real world, which often involves legal and ethical issues. Our models incorporate techniques such as embedding accessible robot pose information and applying semantic mask regulation within the ConvNext backbone network. These techniques are designed to refine intermediate outputs, therefore improving the retention performance of shape and location. Through extensive experimentation, our models have demonstrated notable improvements in maintaining the shape and location of different robots, as well as enhancing overall video generation quality, compared to the benchmark diffusion model. Codes will be opensourced at \href{https://github.com/PengPaulWang/diffusion-robots}{Github}.

Figures (10)

• Figure 1: The original and generated frames of a robot. Left: the original frame; Middle: the frame generated by a proposed model; Right: the frame generated by the benchmark model. The robot arm is broken in the frame generated by the benchmark model.
• Figure 2: The overall architecture of the proposed diffusion model for shape and location retention. The black arrows indicate residual connections. It is worth noting we use images that depict masks of both the robot and the background. However, we also consider cases where only robot masks are used in this paper.
• Figure 3: The improved ConvNext block with a SPADE module. The symbol $\otimes$ represents element-wise products and $\oplus$ indicates the sum of two tensors.
• Figure 4: The embedding block in Fig. \ref{['fig:pipeline']}. We use $(\Delta x, \Delta y, \Delta z, \Delta \phi, \Delta \theta, \Delta \psi)^T$ to represent the robot pose difference between the condition frame and the current frame (frame to generate), the frame index difference is denoted as $\Delta k$, and the diffusion time step is denoted as $t$.
• Figure 5: Results on Scene III, from top to bottom rows: 1) Original frames; 2) Ours-Mask-Pose; 3) Ours-Mask; 4) Ours-Poses; 5) SinFusion. It is noteworthy that only robot masks are used in models where masks are required for this set of results. While robot shape retention can be observed by comparing original and generated frames, location retention can be observed by comparing the robot location with the background in the upper right corner of the frames.