PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI

TL;DR

PLAICraft delivers a large-scale, time-aligned, multi-modal dataset and platform for embodied AI in socially interactive, open-ended environments by recording millisecond-precision vision, audio, and action data from thousands of players within Minecraft. It combines a cloud-based capture pipeline, extensive preprocessing with advanced encoders, and a CHC-aligned evaluation suite to probe perception, memory, language grounding, and social reasoning. The work provides a 200-hour privacy-reviewed public release and outlines continual data release plans, enabling training and benchmarking of open-ended, real-time agents that can act fluently in social contexts. By merging persistent world-state dynamics with rich social interactions and robust preprocessing, PLAICraft aims to accelerate real-world embodied AI research while addressing privacy and governance considerations.

Abstract

Advances in deep generative modelling have made it increasingly plausible to train human-level embodied agents. Yet progress has been limited by the absence of large-scale, real-time, multi-modal, and socially interactive datasets that reflect the sensory-motor complexity of natural environments. To address this, we present PLAICraft, a novel data collection platform and dataset capturing multiplayer Minecraft interactions across five time-aligned modalities: video, game output audio, microphone input audio, mouse, and keyboard actions. Each modality is logged with millisecond time precision, enabling the study of synchronous, embodied behaviour in a rich, open-ended world. The dataset comprises over 10,000 hours of gameplay from more than 10,000 global participants.\footnote{We have done a privacy review for the public release of an initial 200-hour subset of the dataset, with plans to release most of the dataset over time.} Alongside the dataset, we provide an evaluation suite for benchmarking model capabilities in object recognition, spatial awareness, language grounding, and long-term memory. PLAICraft opens a path toward training and evaluating agents that act fluently and purposefully in real time, paving the way for truly embodied artificial intelligence.

Figures (12)

• Figure 1: Illustration of plaicraft.ai dataset collection infrastructure with two, but often many, Minecraft players interacting with each other through our instrumented gameplay environment. Each player has five modalities: screen, microphone input audio, game output audio, mouse and keyboard. All interactions, including voice chat, are real-time and recorded with time alignment in millisecond precision between modalities. The players connect through a web browser to AWS EC2 Ubuntu instances running Minecraft with the instrumentation (\ref{['sec:data_collection']}).
• Figure 2: Visualization of the complex dynamics in the PLAICraft dataset. The frames here are a visualization that combines all modalities, not the actual data format, which stores each modality separately. Keyboard clicks are overlayed in the top left corner, and mouse clicks are overlayed in the top right corner. In the middle of the frame, the mouse movements are visualized by light blue arrows. To visualize speaking and hearing audio, we attached their corresponding transcript at the bottom of each frame.
• Figure 3: Players' demographic distributions. Noted that all the players' demographic information is provided voluntarily by themselves. Left: Played hours distribution over the gender and age groups. Right: Player count distribution by their played hours and experiences.
• Figure 4: Left: Labelled segments within a session using an LLM on the audio transcripts (\ref{['sec:automatic_data_annotation']}). Right: Temporal alignment of the encoded data within a 400 ms window.
• Figure 5: The birdview images showing the world state changes. Left Two: A 10-player base. Right Two: The world origin at $x=0$, $z=0$.