Characterizing Language Use in a Collaborative Situated Game

TL;DR

This work introduces the Portal Dialogue Corpus, a richly annotated, multimodal dataset of 11.5 hours of two-player cooperative Portal 2 gameplay designed to study language use in situated, goal-driven collaboration. By collecting video, audio, transcripts, game-state data, and five-layer dialogue-act annotations, the authors analyze advanced linguistic practices such as ad-hoc convention formation, complex spatial referencing, and multimodal grounding, as well as mixed-initiative planning. Key contributions include detailed data post-processing and annotation pipelines, inter-annotator reliability analyses, and insights into how language evolves with task difficulty, including grounding through gaze and action. The publicly released corpus provides a valuable resource for advancing research in task-oriented dialogue, embodied communication, and AI systems that operate in complex multimodal environments.

Abstract

Cooperative video games, where multiple participants must coordinate by communicating and reasoning under uncertainty in complex environments, yield a rich source of language data. We collect the Portal Dialogue Corpus: a corpus of 11.5 hours of spoken human dialogue in the co-op mode of the popular Portal 2 virtual puzzle game, comprising 24.5K total utterances. We analyze player language and behavior, identifying a number of linguistic phenomena that rarely appear in most existing chitchat or task-oriented dialogue corpora, including complex spatial reference, clarification and repair, and ad-hoc convention formation. To support future analyses of language use in complex, situated, collaborative problem-solving scenarios, we publicly release the corpus, which comprises player videos, audio, transcripts, game state data, and both manual and automatic annotations of language data.

Figures (15)

• Figure 1: Portal 2 is a first-person 3D game where players must solve puzzles through the use of portals. (Left) Snapshot of the game environment, including a third-person view (bottom, not available to players) and popouts showing each player's perspective. (Right) Transcripts of the player dialogue in this scene, each paired with dialogue act annotations (top) and a list of subtasks for this puzzle (bottom, not available to players). In this game snapshot, the blue player uses spatial language to give instructions without an explicit reference, triggering a clarification request and a repair process.
• Figure 2: An example of convention formation.
• Figure 3: An example of metonymy.
• Figure 4: An example of on-the-fly abstraction.
• Figure 5: Here, Blue (top) and Orange (bottom) are facing one another, and Blue is stuck in a box out of which Orange is guiding them.