Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners

TL;DR

The paper investigates how Theory of Mind (ToM) can be embedded into everyday AI, arguing for a shift from static inference to situated, dynamic, and subjective social understanding in daily interactions. It employs 13 two-hour co-design sessions with 26 industry practitioners to surface design recommendations and tensions, revealing three interrelated principles: situating ToM in social context, ensuring responsiveness to dynamic mental states, and attuning to individual subjectivity. The work contributes with a design direction that treats ToM as a pervasive capability integrated into existing AI functionalities, enabling continuous human-AI interaction loops while highlighting practical constraints around data, privacy, and organizational incentives. The findings illuminate how current AI development practices constrain ToM deployment and call for new design methods that balance inference with interaction-driven adaptation to achieve more humane and context-aware everyday AI products and services.

Abstract

Theory of Mind (ToM) -- the ability to infer what others are thinking (e.g., intentions) from observable cues -- is traditionally considered fundamental to human social interactions. This has sparked growing efforts in building and benchmarking AI's ToM capability, yet little is known about how such capability could translate into the design and experience of everyday user-facing AI products and services. We conducted 13 co-design sessions with 26 U.S.-based AI practitioners to envision, reflect, and distill design recommendations for ToM-enabled everyday AI products and services that are both future-looking and grounded in the realities of AI design and development practices. Analysis revealed three interrelated design recommendations: ToM-enabled AI should 1) be situated in the social context that shape users' mental states, 2) be responsive to the dynamic nature of mental states, and 3) be attuned to subjective individual differences. We surface design tensions within each recommendation that reveal a broader gap between practitioners' envisioned futures of ToM-enabled AI and the realities of current AI design and development practices. These findings point toward the need to move beyond static, inference-driven approach to ToM and toward designing ToM as a pervasive capability that supports continuous human-AI interaction loops.

Figures (4)

• Figure 1: The six human-AI social misalignment scenarios practitioners worked on during their co-design sessions. Illustrations under each scenario description were drawn by the research team (created with StoryTribe.com) and served as the first frames of the four-frame storyboards practitioners worked on during the storyboarding activity.
• Figure 2: S7 storyboard on AI Cooking Robot. Created with StoryTribe.com.
• Figure 3: S4 Storyboard created for Autonomous Vehicle scenario. Created with StoryTribe.com.
• Figure 4: We surfaced three design recommendations for ToM-enabled AI to be situated, dynamic, and subjective, highlighting the need to move beyond inference-based approach to AI ToM in everyday contexts. Each recommendation carries built-in design tensions: "rich sensing vs. unobtrusive presence", "ongoing awareness vs. user autonomy", and "individual nuances vs. generalizability." We propose the design direction of designing ToM as a pervasive AI capability to enhance intended or existing AI functionalities through continuous interaction loops to (1) interpret local cues, (2) offer small situational adjustments, (3) refine future behavior based on user response, and repeat.