Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A qualitative field study on explainable AI for lay users subjected to AI cyberattacks

Kevin McAreavey, Weiru Liu, Kim Bauters, Dennis Ivory, George Loukas, Manos Panaousis, Hsueh-Ju Chen, Rea Gill, Rachael Payler, Asimina Vasalou

TL;DR

This paper addresses how lay users understand explainable AI under AI cyberattacks in a real‑world smart heating context. It adopts a seven‑week field study of Squid, a glass‑box setpoint‑learning system with interactive visualizations, and simulates three types of AI attacks to assess diagnostic utility. The findings show limited engagement with XAI features, but participants sometimes developed better mental models and used explanatory tools to diagnose attacks, revealing misalignments with prevailing XAI literature. The work provides empirical guidance on UX design and risk communication for lay users in everyday IoT contexts, highlighting the need for simpler, more actionable explanations to support safe, informed use.

Abstract

In this paper we present results from a qualitative field study on explainable AI (XAI) for lay users (n = 18) who were subjected to AI cyberattacks. The study was based on a custom-built smart heating application called Squid and was conducted over seven weeks in early 2023. Squid combined a smart radiator valve installed in participant homes with a web application that implemented an AI feature known as setpoint learning, which is commonly available in consumer smart thermostats. Development of Squid followed the XAI principle of interpretability-by-design where the AI feature was implemented using a simple glass-box machine learning model with the model subsequently exposed to users via the web interface (e.g. as interactive visualisations). AI attacks on users were simulated by injecting malicious training data and by manipulating data used for model predictions. Research data consisted of semi-structured interviews, researcher field notes, participant diaries, and application logs. In our analysis we reflect on the impact of XAI on user satisfaction and user comprehension as well as its use as a tool for diagnosing AI attacks. Our results show only limited engagement with XAI features and suggest that, for Squid users, common assumptions found in the XAI literature were not aligned to reality. On the positive side, users appear to have developed better mental models of the AI feature compared to previous work, and there is evidence that users did make some use of XAI as a diagnostic tool.

A qualitative field study on explainable AI for lay users subjected to AI cyberattacks

TL;DR

This paper addresses how lay users understand explainable AI under AI cyberattacks in a real‑world smart heating context. It adopts a seven‑week field study of Squid, a glass‑box setpoint‑learning system with interactive visualizations, and simulates three types of AI attacks to assess diagnostic utility. The findings show limited engagement with XAI features, but participants sometimes developed better mental models and used explanatory tools to diagnose attacks, revealing misalignments with prevailing XAI literature. The work provides empirical guidance on UX design and risk communication for lay users in everyday IoT contexts, highlighting the need for simpler, more actionable explanations to support safe, informed use.

Abstract

In this paper we present results from a qualitative field study on explainable AI (XAI) for lay users (n = 18) who were subjected to AI cyberattacks. The study was based on a custom-built smart heating application called Squid and was conducted over seven weeks in early 2023. Squid combined a smart radiator valve installed in participant homes with a web application that implemented an AI feature known as setpoint learning, which is commonly available in consumer smart thermostats. Development of Squid followed the XAI principle of interpretability-by-design where the AI feature was implemented using a simple glass-box machine learning model with the model subsequently exposed to users via the web interface (e.g. as interactive visualisations). AI attacks on users were simulated by injecting malicious training data and by manipulating data used for model predictions. Research data consisted of semi-structured interviews, researcher field notes, participant diaries, and application logs. In our analysis we reflect on the impact of XAI on user satisfaction and user comprehension as well as its use as a tool for diagnosing AI attacks. Our results show only limited engagement with XAI features and suggest that, for Squid users, common assumptions found in the XAI literature were not aligned to reality. On the positive side, users appear to have developed better mental models of the AI feature compared to previous work, and there is evidence that users did make some use of XAI as a diagnostic tool.
Paper Structure (16 sections, 3 equations, 9 figures, 3 tables)

This paper contains 16 sections, 3 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. AI Feature
  5. XAI Features
  6. Hardware
  7. Software & User Interface
  8. AI Attacks
  9. Recruitment
  10. Logistics & Data Collection
  11. Results
  12. User Comprehension
  13. User Satisfaction (Phase 1)
  14. Diagnosing AI Attacks (Phase 2)
  15. Discussion
  16. ...and 1 more sections

Figures (9)

  • Figure 1: Example visualisations of model parameters and model predictions, including (top) heatmaps and (bottom) summary statistics.
  • Figure 2: Example visualisation of model evolution, including (left-to-right) user inputs, model parameters, model predictions, and a learned setpoint schedule.
  • Figure 3: Netatmo hardware, including (left-to-right) relay with mains adapter, smart radiator valve with (below) radiator adapter, and smart thermostat.
  • Figure 4: Schedule page with overlays.
  • Figure 5: Profiles page with overlay.
  • ...and 4 more figures