An Investigation on The Position Encoding in Vision-Based Dynamics Prediction

TL;DR

This work addresses how spatial position information can be implicitly encoded in vision-based dynamics prediction when only object bounding-box abstracts are used. By systematically modifying the Hourglass backbone inputs and CNN padding while evaluating on the SimB billiard dataset (including environment-variant variants SimB-Border and SimB-Split), the authors trace how inconsistency in feature maps enables RoI Pooling to fragment object features that carry location cues. They show that such position encoding can suffice when the environment context remains constant, but struggle under context variation, indicating that environment information is necessary for robust generalization. These findings contribute to the explainability and design of more robust cross-domain dynamics predictors, highlighting the conditions under which object abstractions alone are adequate and when explicit environment modeling becomes essential. The study proposes a pathway toward more interpretable neural representations of spatial relations that can generalize beyond controlled domains, with implications for broader AI explainability.

Abstract

Despite the success of vision-based dynamics prediction models, which predict object states by utilizing RGB images and simple object descriptions, they were challenged by environment misalignments. Although the literature has demonstrated that unifying visual domains with both environment context and object abstract, such as semantic segmentation and bounding boxes, can effectively mitigate the visual domain misalignment challenge, discussions were focused on the abstract of environment context, and the insight of using bounding box as the object abstract is under-explored. Furthermore, we notice that, as empirical results shown in the literature, even when the visual appearance of objects is removed, object bounding boxes alone, instead of being directly fed into the network, can indirectly provide sufficient position information via the Region of Interest Pooling operation for dynamics prediction. However, previous literature overlooked discussions regarding how such position information is implicitly encoded in the dynamics prediction model. Thus, in this paper, we provide detailed studies to investigate the process and necessary conditions for encoding position information via using the bounding box as the object abstract into output features. Furthermore, we study the limitation of solely using object abstracts, such that the dynamics prediction performance will be jeopardized when the environment context varies.

Figures (3)

• Figure 1: Illustration of output features of Hourglass module with various inputs and padding settings, and depiction of the process of position information encoding utilizing bounding-box and RoI Pooling operation. The position information can be encoded when there is inconsistency in the distribution of output features of the Hourglass module, where different values across fragmented object state features are incorporated to encode position information. Such inconsistency can be brought up by either proper padding settings or discrepancies existing in original inputs.
• Figure 2: Illustration of our investigation details. 1) We replace the original Hourglass Module input, which is the output of the original backbone, with All-Zeros Input, All-Ones Input, and Random Input to study the effect on the global feature map of various meaningless inputs; 2) We test multiple CNN padding methods in the Hourglass Module, which are Zero-Pad, Reflect-Pad, Replicate-Pad, and Circular-Pad, to study the effect on the the global feature map of various padding setting. Further, we also studied the joint effect of padding modes with or without the bias weights (not illustrated). We include a simple illustration of the Hourglass Modulehourglass and RPCINrpcin for completion and refer readers to the original papers for detailed illustrations and discussions.
• Figure 3: Quantitative comparison between different padding modes and padding size with bias weight trained on Fixed-Random Inputs. We repeat the experiments of each padding mode with 10 trials. This quantitative results reveal that when bias weight is incorporated, models with different padding modes show comparable performance. P1 and P2 measure the prediction errors for short-term and long-term dynamics prediction, respectively.