Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry

Yifan Zhu, Mariah Bradford, Kenneth Lai, Timothy Obiso, Videep Venkatesha, James Pustejovsky, Nikhil Krishnaswamy

TL;DR

Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs'abilities to track both task progression and belief state, and an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task is evaluated.

Abstract

Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich multimodal communication under epistemic asymmetry. We present a multimodal dataset of these interactions, annotated and temporally aligned across speech, gesture, and action modalities to support reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large language models (LLMs), prompted to infer shared beliefs from multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.

Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry

TL;DR

Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs'abilities to track both task progression and belief state, and an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task is evaluated.

Abstract

Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich multimodal communication under epistemic asymmetry. We present a multimodal dataset of these interactions, annotated and temporally aligned across speech, gesture, and action modalities to support reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large language models (LLMs), prompted to infer shared beliefs from multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.
Paper Structure (16 sections, 2 figures, 4 tables)

This paper contains 16 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Description and Dataset
  4. Distributed Partial Information Puzzle
  5. Data
  6. Annotation Pipeline
  7. Speech Annotations
  8. Action Annotations
  9. Gesture Annotations
  10. Inter-Annotator Agreement
  11. Cross-Modality Alignment
  12. Evaluation
  13. Common Ground (CG) Calculation
  14. Experiments
  15. Results and Analysis
  16. ...and 1 more sections

Figures (2)

  • Figure 1: A builder and 3 directors participating in the DPIP task with a partially-completed structure on the table. Director 1 (second from left) is indicating the position of a block using a gesture as well as a speech act.
  • Figure 2: 3 individual side views of the same complete structure, each assigned to a director.