Nearest Neighbor Future Captioning: Generating Descriptions for Possible Collisions in Object Placement Tasks

TL;DR

The paper addresses the challenge of describing future collisions in object placement by domestic service robots. It introduces NNFCM, which combines a Nearest Neighbor Language Model with a Collision Attention Module and a cross modal encoder/decoder architecture to predict and caption potential collisions before action. Empirical results on the BILA-caption 2.0 dataset show substantial improvements over baselines in CIDEr-D and other metrics, with additional ablation and human evaluation supporting the effectiveness of the NNLM and attention mechanisms. The work advances explainable robotics by enabling natural language explanations of predicted risks, with potential for real-time safety guidance and broader applicability to multimodal captioning tasks.

Abstract

Domestic service robots (DSRs) that support people in everyday environments have been widely investigated. However, their ability to predict and describe future risks resulting from their own actions remains insufficient. In this study, we focus on the linguistic explainability of DSRs. Most existing methods do not explicitly model the region of possible collisions; thus, they do not properly generate descriptions of these regions. In this paper, we propose the Nearest Neighbor Future Captioning Model that introduces the Nearest Neighbor Language Model for future captioning of possible collisions, which enhances the model output with a nearest neighbors retrieval mechanism. Furthermore, we introduce the Collision Attention Module that attends regions of possible collisions, which enables our model to generate descriptions that adequately reflect the objects associated with possible collisions. To validate our method, we constructed a new dataset containing samples of collisions that can occur when a DSR places an object in a simulation environment. The experimental results demonstrated that our method outperformed baseline methods, based on the standard metrics. In particular, on CIDEr-D, the baseline method obtained 25.09 points, whereas our method obtained 33.08 points.

Figures (6)

• Figure 1: Overview of our method: Given an image of the current destination and the target object, our method generates descriptions of possible collisions.
• Figure 2: A typical scene of the task. Left: an image of the simulation environment. Right: an image of the destination.
• Figure 3: The framework of our model. The Cross Attentional Image Encoder consists of two stacked blocks, which enable the model to understand the mutual relationship between the target object and its destination. "Cross Atten. Layer" and "Self Atten. Layer" denote Cross Attention Layer and Self Attention Layer, respectively.
• Figure 4: A successful sample. In this successful sample, the target object is a "plastic bottle" and the collided object is a "wodden toy car."
• Figure 5: A successful sample. In this successful sample, the target object is a "Rubik's Cube" and the collided object is a "teddy bear."