S$^3$IT: A Benchmark for Spatially Situated Social Intelligence Test

TL;DR

The paper tackles the challenge of evaluating embodied social intelligence by introducing S$^3$IT, a spatially situated benchmark built around a seat-ordering task in a 3D environment with diverse NPCs. It combines procedural task generation with a three-phase evaluation—preference extraction, environmental cognition, and multi-constraint optimization—and an automatic scoring pipeline that accounts for both spatial and social constraints. Key contributions include a 7,000-instance, 70-difficulty-level dataset, a generate-and-reflect optimization framework, and comprehensive evaluation of state-of-the-art LLMs, revealing substantial gaps in embodied spatial reasoning despite strengths in explicit social constraints. The findings underscore the necessity of grounding abstract social cognition in physical contexts and suggest that spatial grounding is the primary bottleneck for current models, guiding future work toward more integrated embodied intelligence.

Abstract

The integration of embodied agents into human environments demands embodied social intelligence: reasoning over both social norms and physical constraints. However, existing evaluations fail to address this integration, as they are limited to either disembodied social reasoning (e.g., in text) or socially-agnostic physical tasks. Both approaches fail to assess an agent's ability to integrate and trade off both physical and social constraints within a realistic, embodied context. To address this challenge, we introduce Spatially Situated Social Intelligence Test (S$^{3}$IT), a benchmark specifically designed to evaluate embodied social intelligence. It is centered on a novel and challenging seat-ordering task, requiring an agent to arrange seating in a 3D environment for a group of large language model-driven (LLM-driven) NPCs with diverse identities, preferences, and intricate interpersonal relationships. Our procedurally extensible framework generates a vast and diverse scenario space with controllable difficulty, compelling the agent to acquire preferences through active dialogue, perceive the environment via autonomous exploration, and perform multi-objective optimization within a complex constraint network. We evaluate state-of-the-art LLMs on S$^{3}$IT and found that they still struggle with this problem, showing an obvious gap compared with the human baseline. Results imply that LLMs have deficiencies in spatial intelligence, yet simultaneously demonstrate their ability to achieve near human-level competence in resolving conflicts that possess explicit textual cues.

Figures (9)

• Figure 1: A typical seat arrangement task involves a given room layout and several NPCs. The agent under test (T-Agent) needs to interact with the NPCs and explore the room to devise a seating arrangement that satisfies everyone.
• Figure 2: The four groups of elements: 1) table splitting, 2) embodied spatial understanding (embodied pref.), 3) social relationship understanding (social pref.), and 4) conflicts, as well as the categories they contain.
• Figure 3: Five kinds of house layouts. (a) Single rectangular table with 4 chairs. (b) Single irregular table with 5 chairs. (c) Single circular table with 6 chairs. (d) A single room with a rectangular table (6 chairs) and a circular table (4 chairs). (e) Multiple rooms with a rectangular table (5 chairs), a circular table (4 chairs), and an oval table (4 chairs).
• Figure 4: Detailed preview of House A. The preferences of NPCs originate from the spatial constraints within rooms, as outlined by the black dotted lines in the figure. For instance, the air conditioners relate to their preferences for temperature, and kitchens correspond to NPCs' tendencies to be near or away from kitchens. Additionally, NPCs have specific requirements for tableware (chopsticks or cutlery), as shown in the enlarged view of the table area.
• Figure 5: Selection frequency of each preference and conflict. (a) The Full Dataset. (b) The Test Set, whose distribution of elements is consistent with that of the full dataset.