SAGE: Smart home Agent with Grounded Execution

TL;DR

SAGE proposes an autonomous LLM agent architecture for smart homes that grounds execution in a dynamically constructed sequence of tool calls, enabling flexible, user-specific reasoning beyond fixed pipelines. The system combines personalization (long-term memory and user profiling), device interaction (planner, disambiguation via visual context, API documentation), and persistent monitoring (code-writing and polling) within a unified agent-tool framework. A 50-task benchmark demonstrates SAGE achieving about 75% success, significantly outperforming two LLM-based baselines and illustrating strong gains from integrating diverse information sources and tools. The work highlights the potential for future open-source LLMs to approach GPT-4-level performance and argues for continued development of grounded, tool-augmented agents in practical smart home settings.

Abstract

The common sense reasoning abilities and vast general knowledge of Large Language Models (LLMs) make them a natural fit for interpreting user requests in a Smart Home assistant context. LLMs, however, lack specific knowledge about the user and their home limit their potential impact. SAGE (Smart Home Agent with Grounded Execution), overcomes these and other limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. SAGE controls this process through a dynamically constructed tree of LLM prompts, which help it decide which action to take next, whether an action was successful, and when to terminate the process. The SAGE action set augments an LLM's capabilities to support some of the most critical requirements for a Smart Home assistant. These include: flexible and scalable user preference management ("is my team playing tonight?"), access to any smart device's full functionality without device-specific code via API reading "turn down the screen brightness on my dryer", persistent device state monitoring ("remind me to throw out the milk when I open the fridge"), natural device references using only a photo of the room ("turn on the light on the dresser"), and more. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate, significantly outperforming existing LLM-enabled baselines (30% success rate).

Figures (8)

• Figure 1: Interactive demo where the user asks SAGE: "Put on the game by the dresser". The figure illustrates the sequence of tools used by SAGE to complete the task. Note that each control flow decision was made by an LLM, not by hand-coded logic. The control flow is described step by step in Section \ref{['sec:execution_example']}. This demo was executed using real SmartThings-enabled devices.
• Figure 2: Simplified LLM prompt construction and response example. Blue text indicates prompt, orange text indicates response, and black text is explanation of the section of the prompt below. In this example, the decision function ($d_i$) is being called for the second time, so the history ($H_i$) is populated with the results of the first call.
• Figure 3: An overview of the personalization tool. Long-term memory remembers and retrieves all past user utterances. User profile contains more general trends about the user's preferences.
• Figure 4: Device disambiguation. A method to resolve the device that best fits a user natural language device description using VLMs.
• Figure 5: A summary of the persistent command handling mechanism. 1: After receiving the user request, the SAGE agent-tool extracts the condition ("is the TV off?") and uses the condition code writing tool to write code to check this condition. The tool registers this code in memory and responds with the name of the function (is_tv_off). 2: The SAGE agent-tool registers the function is_tv_off with the condition polling tool, along with the command reflecting the action to take once the condition is detected "turn the lights on". At this point the SAGE agent-tool finishes executing. The condition polling tool periodically runs the is_tv_off function. Once the function outputs True, the condition polling tool triggers the SAGE agent-tool again with the registered action "turn the lights on." Note that the agent-tool will only be triggered when the status of the action transitions from False to True to avoid re-running the agent-tool the entire time the TV is off. 3: The agent-tool begins executing with the command "turn the lights on" and turns on the lights using the device interaction tool.