Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models

Javier Chiyah-Garcia, Alessandro Suglia, Arash Eshghi

TL;DR

This paper collects, analyse, and publicly release BlockWorld-Repairs: a dataset of multi-modal TPR sequences in an instruction-following manipulation task that is, by design, rife with referential ambiguity, and finds that all models significantly underperform in this task.

Abstract

In dialogue, the addressee may initially misunderstand the speaker and respond erroneously, often prompting the speaker to correct the misunderstanding in the next turn with a Third Position Repair (TPR). The ability to process and respond appropriately to such repair sequences is thus crucial in conversational AI systems. In this paper, we first collect, analyse, and publicly release BlockWorld-Repairs: a dataset of multi-modal TPR sequences in an instruction-following manipulation task that is, by design, rife with referential ambiguity. We employ this dataset to evaluate several state-of-the-art Vision and Language Models (VLM) across multiple settings, focusing on their capability to process and accurately respond to TPRs and thus recover from miscommunication. We find that, compared to humans, all models significantly underperform in this task. We then show that VLMs can benefit from specialised losses targeting relevant tokens during fine-tuning, achieving better performance and generalising better to new scenarios. Our results suggest that these models are not yet ready to be deployed in multi-modal collaborative settings where repairs are common, and highlight the need to design training regimes and objectives that facilitate learning from interaction. Our code and data are available at www.github.com/JChiyah/blockworld-repairs

Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models

TL;DR

This paper collects, analyse, and publicly release BlockWorld-Repairs: a dataset of multi-modal TPR sequences in an instruction-following manipulation task that is, by design, rife with referential ambiguity, and finds that all models significantly underperform in this task.

Abstract

In dialogue, the addressee may initially misunderstand the speaker and respond erroneously, often prompting the speaker to correct the misunderstanding in the next turn with a Third Position Repair (TPR). The ability to process and respond appropriately to such repair sequences is thus crucial in conversational AI systems. In this paper, we first collect, analyse, and publicly release BlockWorld-Repairs: a dataset of multi-modal TPR sequences in an instruction-following manipulation task that is, by design, rife with referential ambiguity. We employ this dataset to evaluate several state-of-the-art Vision and Language Models (VLM) across multiple settings, focusing on their capability to process and accurately respond to TPRs and thus recover from miscommunication. We find that, compared to humans, all models significantly underperform in this task. We then show that VLMs can benefit from specialised losses targeting relevant tokens during fine-tuning, achieving better performance and generalising better to new scenarios. Our results suggest that these models are not yet ready to be deployed in multi-modal collaborative settings where repairs are common, and highlight the need to design training regimes and objectives that facilitate learning from interaction. Our code and data are available at www.github.com/JChiyah/blockworld-repairs
Paper Structure (41 sections, 6 figures, 12 tables)

This paper contains 41 sections, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Communicative Grounding, Miscommunication and Repair
  4. Computational Models of Repair
  5. VLMs for Situated Collaborative Tasks
  6. The BlockWorld-Repair Dataset
  7. Differences from the original Block World
  8. Ecological validity
  9. Dataset Collection
  10. Quality Control and Human Baselines
  11. Experiments
  12. Experimental Setup
  13. Dataset
  14. Tasks
  15. Metrics
  16. ...and 26 more sections

Figures (6)

  • Figure 1: Example dialogue from BlockWorld-Repairs: after predicting an incorrect response, VLMs must accurately interpret the repair to produce the correct bounding box prediction, a critical skill for human-robot collaboration tasks.
  • Figure 2: Masking criteria for the cross-entropy loss.
  • Figure 3: Two medium-difficulty dialogues with the bounding boxes predicted by the VLMs and humans.
  • Figure 4: Instructions that AMT workers saw in the landing page of the HIT.
  • Figure 5: User interface for the data collection. The images of the robot and blocks would change depending on the task or the instructions given. The left has the chat and a text box to send messages whereas the right shows how the final placement should look in the end.
  • ...and 1 more figures