Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulation

Xuhui Zhou, Zhe Su, Sophie Feng, Jiaxu Zhou, Jen-tse Huang, Hsien-Te Kao, Spencer Lynch, Svitlana Volkova, Tongshuang Sherry Wu, Anita Woolley, Hao Zhu, Maarten Sap

TL;DR

This paper addresses the challenge of conducting large-scale, human-like social simulations with LLM agents, which are typically difficult to set up and evaluate. It introduces SOTOPIA-S$^4$, a three-part system with a high-performance simulation engine, a REST API, and a web UI, enabling natural-language configuration, parallelized execution, and customizable evaluation metrics. Key contributions include a NL-based configuration workflow, an asynchronous multi-agent interaction framework with information asymmetry, a default and customizable evaluation suite powered by LLMs, and LiteLLM-based multi-LLM integration. Use cases demonstrate dyadic hiring negotiations, multiparty planning, and large-scale stress tests, showing personality effects and scalability. The system lowers barriers for social science researchers to test hypotheses and analyze LLM agent behavior at scale through a user-friendly API and web interface.

Abstract

Social simulation through large language model (LLM) agents is a promising approach to explore and validate hypotheses related to social science questions and LLM agents behavior. We present SOTOPIA-S4, a fast, flexible, and scalable social simulation system that addresses the technical barriers of current frameworks while enabling practitioners to generate multi-turn and multi-party LLM-based interactions with customizable evaluation metrics for hypothesis testing. SOTOPIA-S4 comes as a pip package that contains a simulation engine, an API server with flexible RESTful APIs for simulation management, and a web interface that enables both technical and non-technical users to design, run, and analyze simulations without programming. We demonstrate the usefulness of SOTOPIA-S4 with two use cases involving dyadic hiring negotiation and multi-party planning scenarios.

SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulation

TL;DR

This paper addresses the challenge of conducting large-scale, human-like social simulations with LLM agents, which are typically difficult to set up and evaluate. It introduces SOTOPIA-S, a three-part system with a high-performance simulation engine, a REST API, and a web UI, enabling natural-language configuration, parallelized execution, and customizable evaluation metrics. Key contributions include a NL-based configuration workflow, an asynchronous multi-agent interaction framework with information asymmetry, a default and customizable evaluation suite powered by LLMs, and LiteLLM-based multi-LLM integration. Use cases demonstrate dyadic hiring negotiations, multiparty planning, and large-scale stress tests, showing personality effects and scalability. The system lowers barriers for social science researchers to test hypotheses and analyze LLM agent behavior at scale through a user-friendly API and web interface.

Abstract

Social simulation through large language model (LLM) agents is a promising approach to explore and validate hypotheses related to social science questions and LLM agents behavior. We present SOTOPIA-S4, a fast, flexible, and scalable social simulation system that addresses the technical barriers of current frameworks while enabling practitioners to generate multi-turn and multi-party LLM-based interactions with customizable evaluation metrics for hypothesis testing. SOTOPIA-S4 comes as a pip package that contains a simulation engine, an API server with flexible RESTful APIs for simulation management, and a web interface that enables both technical and non-technical users to design, run, and analyze simulations without programming. We demonstrate the usefulness of SOTOPIA-S4 with two use cases involving dyadic hiring negotiation and multi-party planning scenarios.

Paper Structure

This paper contains 36 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Social Simulation and its Applications
  4. Multi-agent Frameworks
  5. Simulation and Evaluation Overview
  6. Simulation setup
  7. Scenarios
  8. Characters
  9. Relationships
  10. Episodes
  11. Async interaction framework
  12. Message broker and information asymmetry
  13. Turn-taking
  14. Simulation evaluation
  15. Default evaluation suite
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Overview of SOTOPIA-S$^4$. The platform consists of three main components: (1) A high-performance simulation engine with automated data persistence to Redis. (2) A RESTful API server. (3) An intuitive web-based interface. The web UI interface shows an dyadic example of an AI hiring manager negotiating with a candidate.
  • Figure A.1: An example character profile in SOTOPIA-S$^4$.
  • Figure B.1: The asynchronous interaction framework for agents in SOTOPIA-S$^4$ for the simultaneous turn-taking strategy. Each agent maintains its own message queue and can decide when to respond based on the conversation context and its own state.
  • Figure D.1: The API documentation page of SOTOPIA-S$^4$. The interactive Swagger UI provides comprehensive documentation of available endpoints, with different colors indicating the HTTP methods (GET, POST, DELETE) for each operation.