AI-Gadget Kit: Integrating Swarm User Interfaces with LLM-driven Agents for Rich Tabletop Game Applications

TL;DR

The paper addresses the lack of autonomous action planning in Swarm User Interfaces for tabletop games and proposes the AI-Gadget Kit, a multi-agent SUI that integrates LLM-driven Coordinator and Controller agents to extend meta-actions and enable complex motion planning via add-on prompts. It contributes a complete system architecture, a defined design space with eight meta-actions/prompt types, and four interactive-behavior plus four relationship prompts, demonstrated through four tabletop game scenarios. The work shows how agent-driven gadgets can deliver dynamic, narrative-aware interactions with humans, enabling personalized gameplay experiences and expanding SUI capabilities to more complex interaction tasks. It highlights limitations such as hardware variability and LLM context management, and outlines future directions including simulation tools, enhanced hardware, and broader applicability beyond tabletop games.

Abstract

While Swarm User Interfaces (SUIs) have succeeded in enriching tangible interaction experiences, their limitations in autonomous action planning have hindered the potential for personalized and dynamic interaction generation in tabletop games. Based on the AI-Gadget Kit we developed, this paper explores how to integrate LLM-driven agents within tabletop games to enable SUIs to execute complex interaction tasks. After defining the design space of this kit, we elucidate the method for designing agents that can extend the meta-actions of SUIs to complex motion planning. Furthermore, we introduce an add-on prompt method that simplifies the design process for four interaction behaviors and four interaction relationships in tabletop games. Lastly, we present several application scenarios that illustrate the potential of AI-Gadget Kit to construct personalized interaction in SUI tabletop games. We expect to use our work as a case study to inspire research on multi-agent-driven SUI for other scenarios with complex interaction tasks.

Figures (15)

• Figure 1: System Architecture of AI-Gadget Kit.
• Figure 2: Design space for SUI action planning in tabletop game scenarios through our kit. The kit includes a two-agent system(d) that can generate action sequences for gadgets to complete interaction tasks based on eight meta-actions (a), four types of interaction behaviors (b), and four types of interaction relationships (c).
• Figure 3: An example using our kit: (a) Users can declare the game they are playing through an introduction and rules entry. (b) Then, they input game commands based on the scenario to interact with the gadget. The gadget generates corresponding action sequences based on a two-agent system within the kit.
• Figure 4: The gadget starts at position (1,1) and needs to kick a light plastic soccer ball located at (3,3). a) Gadget: Moves northeast at speed 3 to approach the soccer ball's location b)Kicks the soccer ball, simulating the action by moving east to (3,3). c-d) Two gadgets open a door together by moving towards it.
• Figure 5: The gadgets were asked to present the word 'UIST' or 'HI'. a-d) Agents generated trajectories of four gadgets, which were arranged to form 'U', 'I', 'S'. e-f) A detailed demonstration of how one of the gadgets presents the letter "T" through its movement trajectory. h-i) Agents make each letter composed of multiple gadgets, using 7 and 3 gadgets to form the letters 'H' and 'I' respectively.