Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments

TL;DR

Deliberate Lab tackles the lack of real-time, scalable infrastructure for studying human–AI social dynamics by delivering a no-code, open-source platform that treats LLM agents as first-class participants. The authors detail a TypeScript-based system with real-time synchronization, cohort management, and structured prompts to enable both human-only and hybrid studies, along with a security-conscious architecture built on Firebase. A 12-month deployment (N=88 experimenters, N=9,195 participants, 597 experiments) provides deployment analytics and qualitative insights, complemented by case studies such as a large-scale election and a bespoke human–AI negotiation. The work demonstrates broad applicability across disciplines and emphasizes reproducibility and accessibility for interdisciplinary researchers. Overall, Deliberate Lab advances the methodological toolkit for digitally mediated group reasoning in an era where AI agents actively participate in deliberation and decision-making.

Abstract

Social and behavioral scientists increasingly aim to study how humans interact, collaborate, and make decisions alongside artificial intelligence. However, the experimental infrastructure for such work remains underdeveloped: (1) few platforms support real-time, multi-party studies at scale; (2) most deployments require bespoke engineering, limiting replicability and accessibility, and (3) existing tools do not treat AI agents as first-class participants. We present Deliberate Lab, an open-source platform for large-scale, real-time behavioral experiments that supports both human participants and large language model (LLM)-based agents. We report on a 12-month public deployment of the platform (N=88 experimenters, N=9195 experiment participants), analyzing usage patterns and workflows. Case studies and usage scenarios are aggregated from platform users, complemented by in-depth interviews with select experimenters. By lowering technical barriers and standardizing support for hybrid human-AI experimentation, Deliberate Lab expands the methodological repertoire for studying collective decision-making and human-centered AI.

Figures (16)

• Figure 1: Experiment creation view. (1) Settings configuration, such as metadata and LLMs. (2) Stage ordering panel; stages can be added manually (full list in Appendix \ref{['app:architecture-stages']}) or loaded from a template. (3) Stage editing panel; a survey question editor is shown.
• Figure 2: Prompt editor for chat stage. On the left, a list editor with prompt items enables the experimenter to define and reorder custom text as well as variables like stage context. On the right, the experimenter can use the dropdown to select a structured output mode and/or customize structured output fields.
• Figure 3: Facilitator view for a "lobby"-style experiment. (1) Experiment management toggle, including a search bar to find participants, a messaging feature to message participants, LLM API configuration settings, and a notification panel to address active assistance requests. (2) On the left, a cohort management panel showing all participant groups, with live participant monitoring and attention check features. (3) On the right, an experiment preview panel shows a selected participant's live view; experimenters can view and submit input on the participant's behalf if needed, and toggle debugging messages for LLMs (visible only to the experimenter).
• Figure 4: Simplified architecture of the Deliberate Lab platform stack. (1) Experiments are configured and submitted to a backend endpoint, (2) then stored in the primary Firestore database (Figure \ref{['fig:system-architecture']}) for frontend access. (3) Participant inputs are sent to the backend and logged in the primary Firestore database. (4) Presence signals are written to the secondary Realtime database to track active users.
• Figure 5: Cumulative experimenter signups over a 12-month period, color-coded by organization; there were 88 account applications across 26 organizations. The bursty pattern reflects waves of coordinated institutional uptake.