Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating HomeAssistant Automations Using an LLM-based Chatbot

Mathyas Giudici, Alessandro Sironi, Ismaele Villa, Samuele Scherini, Franca Garzotto

TL;DR

The paper investigates using an LLM-based chatbot to generate HomeAssistant automations aimed at promoting sustainable household energy practices. It develops EcoMate, a GPT-driven system that generates HomeAssistant-ready JSON routines from natural-language commands, comparing green versus no-green prompts across multiple LLMs and temperatures. GPT-4 emerges as the most reliable at producing parsable, correct JSON, while non-GPT models struggle with API formatting; green prompts yield subtle qualitative shifts toward sustainability but do not consistently improve quantitative metrics. A second evaluation with EcoMate shows higher perceived dialogue and engagement for the LLM version versus a rule-based baseline, supporting the potential of LLMs to enhance user adoption of energy-saving automations. The work highlights both the promise and current limitations (formatting, hallucinations, and command interpretation), and suggests future real-world, longitudinal studies and broader integration to advance sustainable smart homes.

Abstract

To combat climate change, individuals are encouraged to adopt sustainable habits, in particular, with their household, optimizing their electrical consumption. Conversational agents, such as Smart Home Assistants, hold promise as effective tools for promoting sustainable practices within households. Our research investigated the application of Large Language Models (LLM) in enhancing smart home automation and promoting sustainable household practices, specifically using the HomeAssistant framework. In particular, it highlights the potential of GPT models in generating accurate automation routines. While the LLMs showed proficiency in understanding complex commands and creating valid JSON outputs, challenges such as syntax errors and message malformations were noted, indicating areas for further improvement. Still, despite minimal quantitative differences between "green" and "no green" prompts, qualitative feedback highlighted a positive shift towards sustainability in the routines generated with environmentally focused prompts. Then, an empirical evaluation (N=56) demonstrated that the system was well-received and found engaging by users compared to its traditional rule-based counterpart. Our findings highlight the role of LLMs in advancing smart home technologies and suggest further research to refine these models for broader, real-world applications to support sustainable living.

Generating HomeAssistant Automations Using an LLM-based Chatbot

TL;DR

The paper investigates using an LLM-based chatbot to generate HomeAssistant automations aimed at promoting sustainable household energy practices. It develops EcoMate, a GPT-driven system that generates HomeAssistant-ready JSON routines from natural-language commands, comparing green versus no-green prompts across multiple LLMs and temperatures. GPT-4 emerges as the most reliable at producing parsable, correct JSON, while non-GPT models struggle with API formatting; green prompts yield subtle qualitative shifts toward sustainability but do not consistently improve quantitative metrics. A second evaluation with EcoMate shows higher perceived dialogue and engagement for the LLM version versus a rule-based baseline, supporting the potential of LLMs to enhance user adoption of energy-saving automations. The work highlights both the promise and current limitations (formatting, hallucinations, and command interpretation), and suggests future real-world, longitudinal studies and broader integration to advance sustainable smart homes.

Abstract

To combat climate change, individuals are encouraged to adopt sustainable habits, in particular, with their household, optimizing their electrical consumption. Conversational agents, such as Smart Home Assistants, hold promise as effective tools for promoting sustainable practices within households. Our research investigated the application of Large Language Models (LLM) in enhancing smart home automation and promoting sustainable household practices, specifically using the HomeAssistant framework. In particular, it highlights the potential of GPT models in generating accurate automation routines. While the LLMs showed proficiency in understanding complex commands and creating valid JSON outputs, challenges such as syntax errors and message malformations were noted, indicating areas for further improvement. Still, despite minimal quantitative differences between "green" and "no green" prompts, qualitative feedback highlighted a positive shift towards sustainability in the routines generated with environmentally focused prompts. Then, an empirical evaluation (N=56) demonstrated that the system was well-received and found engaging by users compared to its traditional rule-based counterpart. Our findings highlight the role of LLMs in advancing smart home technologies and suggest further research to refine these models for broader, real-world applications to support sustainable living.
Paper Structure (51 sections, 7 figures, 15 tables)

This paper contains 51 sections, 7 figures, 15 tables.

Table of Contents

  1. Introduction
  2. State of the Art
  3. Home Automation Environments
  4. Large Language Models
  5. Comparative Study Setup
  6. Prompt
  7. User Commands
  8. Smart Homes
  9. Energy Consumption
  10. Methodology
  11. Study Design
  12. Metrics
  13. Procedure
  14. Comparative Study Evaluation
  15. LLM as HomeAssistant Controller
  16. ...and 36 more sections

Figures (7)

  • Figure 1: Paper Overview, with mapping of RQs to empirical studies and sections
  • Figure 2: Original and Revisited H107 House Templates
  • Figure 3: Heatmap of the JSON validity percentage for each LLM, prompt and temperature
  • Figure 4: Heatmap of the JSON validity percentage for each LLM, and Command Category (a) or Command Type (b)
  • Figure 5: Plots of the Similarity metric taken into consideration in the evaluation of LLM capabilities to generate HomeAssistant JSONs
  • ...and 2 more figures