J-CRe3: A Japanese Conversation Dataset for Real-world Reference Resolution

TL;DR

A multimodal reference resolution task is proposed and a Japanese Conversation dataset for Real-world Reference Resolution (J-CRe3) is constructed, which contains egocentric video and dialogue audio of real-world conversations between two people acting as a master and an assistant robot at home.

Abstract

Understanding expressions that refer to the physical world is crucial for such human-assisting systems in the real world, as robots that must perform actions that are expected by users. In real-world reference resolution, a system must ground the verbal information that appears in user interactions to the visual information observed in egocentric views. To this end, we propose a multimodal reference resolution task and construct a Japanese Conversation dataset for Real-world Reference Resolution (J-CRe3). Our dataset contains egocentric video and dialogue audio of real-world conversations between two people acting as a master and an assistant robot at home. The dataset is annotated with crossmodal tags between phrases in the utterances and the object bounding boxes in the video frames. These tags include indirect reference relations, such as predicate-argument structures and bridging references as well as direct reference relations. We also constructed an experimental model and clarified the challenges in multimodal reference resolution tasks.

Figures (5)

• Figure 1: Example of J-CRe3. It has object bounding boxes (the orange rectangles), textual reference relations (the blue table), and text-to-object reference relations (the green lines). An object bounding box has a class name and an instance ID. Textual and text-to-object reference relations have 10--20 types of relations, including direct reference relations (=) and indirect reference relations corresponding to nominative (NOM), accusative (ACC), and dative (DAT) cases. For example, sports drink has a direct text-to-object reference relation (=) with the object bounding box, "bottle_1." Note that a particular case TOP shown in the example dialogue indicates an attached noun phrase is the sentence's topic.
• Figure 2: Example of regional bounding box annotation
• Figure 3: Distribution of Recall@$k$ and number of objects for each object class: Figure shows top 15 classes. Although we show the difference between Recall@5 and Recall@10 in red in the figure, there were very few of them.
• Figure 4: Example of phrase grounding for utterance "The notebook and mobile phone should be put on the table in the next room, and the remote control should be put on the sofa, right?" (translated). Predicted phrases and their confidences are shown with colored object bounding boxes. The confidence threshold is 0.5.
• Figure 5: Translated crowdsourcing interface used for collection of scenarios in living room