HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices

TL;DR

This work addresses the gap in robust evaluation of LLM-based smart-home assistants under invalid inputs and multi-device scenarios. It introduces HomeBench, a comprehensive virtual-smart-home benchmark with 100 scenarios and over 170k instructions spanning 47 devices and 10 device types, enabling systematic study of single- and multi-device instructions with valid and invalid cases. Evaluating 13 LLMs, the study reveals that even advanced models like GPT-4o struggle with invalid multi-device instructions, though in-context learning and LoRA fine-tuning substantially improve performance; nonetheless, a practical level of reliability remains out of reach. The public dataset and experimental framework establish a foundation for developing more robust, safe, and scalable LLM-based smart-home assistants, and point to future work in improving long-context reasoning and retrieval strategies.

Abstract

Large language models (LLMs) have the potential to revolutionize smart home assistants by enhancing their ability to accurately understand user needs and respond appropriately, which is extremely beneficial for building a smarter home environment. While recent studies have explored integrating LLMs into smart home systems, they primarily focus on handling straightforward, valid single-device operation instructions. However, real-world scenarios are far more complex and often involve users issuing invalid instructions or controlling multiple devices simultaneously. These have two main challenges: LLMs must accurately identify and rectify errors in user instructions and execute multiple user instructions perfectly. To address these challenges and advance the development of LLM-based smart home assistants, we introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices in this paper. We have experimental results on 13 distinct LLMs; e.g., GPT-4o achieves only a 0.0% success rate in the scenario of invalid multi-device instructions, revealing that the existing state-of-the-art LLMs still cannot perform well in this situation even with the help of in-context learning, retrieval-augmented generation, and fine-tuning. Our code and dataset are publicly available at https://github.com/BITHLP/HomeBench.

Figures (14)

• Figure 1: An example of valid single-device instruction, invalid single-device instruction, valid multi-device instruction, invalid multi-device instruction, and mixed multi-device instruction. Orange indicates the device to be operated, blue indicates the method of the device to be operated, and red indicates the wrong device to be operated. These mistake outputs are due to the operation of non-existent devices or non-existent device functions, causing the model to hallucinate and operate the wrong device.
• Figure 2: The whole process of collecting dataset HomeBench, from device construction, room setting, instructions generation, and user instructions synthesis to quality control.
• Figure 3: The performance of Qwen2.5-7B-Instruct model in adding different types of data samples (the order of adding is: VS, IS, VM, MM, IM).
• Figure 4: The performance gap between Qwen-ICL and Qwen-ICL-RAG.
• Figure 5: Performance of Qwen2.5-7B-Instruct in test dataset and OOD dataset under different training steps.