Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learn, Explore and Reflect by Chatting: Understanding the Value of an LLM-Based Voting Advice Application Chatbot

Jianlong Zhu, Manon Kempermann, Vikram Kamath Cannanure, Alexander Hartland, Rosa M. Navarrete, Giuseppe Carteny, Daniela Braun, Ingmar Weber

TL;DR

The paper addresses the challenge that traditional VAAs, while beneficial for political knowledge, remain inaccessible to less-sophisticated voters due to complex language and rigid interfaces. It investigates an LLM-based VAA chatbot deployed in Germany ahead of the 2024 European Parliament election, using a mixed-method design with 331 participants and 10 follow-up interviews to assess usability, knowledge gain, and trust. Findings show the chatbot delivers concise, personalized explanations, stimulates curiosity, reflection, and deliberation, and is well-received, yet raises concerns about truthfulness, transparency, and potential bias. The authors offer design guidelines and extend the MATCH model for trustworthy AI in civic education, highlighting how to balance informative, deliberative engagement with accountability and external validation to enable responsible use in political decision-making.

Abstract

Voting advice applications (VAAs), which have become increasingly prominent in European elections, are seen as a successful tool for boosting electorates' political knowledge and engagement. However, VAAs' complex language and rigid presentation constrain their utility to less-sophisticated voters. While previous work enhanced VAAs' click-based interaction with scripted explanations, a conversational chatbot's potential for tailored discussion and deliberate political decision-making remains untapped. Our exploratory mixed-method study investigates how LLM-based chatbots can support voting preparation. We deployed a VAA chatbot to 331 users before Germany's 2024 European Parliament election, gathering insights from surveys, conversation logs, and 10 follow-up interviews. Participants found the VAA chatbot intuitive and informative, citing its simple language and flexible interaction. We further uncovered VAA chatbots' role as a catalyst for reflection and rationalization. Expanding on participants' desire for transparency, we provide design recommendations for building interactive and trustworthy VAA chatbots.

Learn, Explore and Reflect by Chatting: Understanding the Value of an LLM-Based Voting Advice Application Chatbot

TL;DR

The paper addresses the challenge that traditional VAAs, while beneficial for political knowledge, remain inaccessible to less-sophisticated voters due to complex language and rigid interfaces. It investigates an LLM-based VAA chatbot deployed in Germany ahead of the 2024 European Parliament election, using a mixed-method design with 331 participants and 10 follow-up interviews to assess usability, knowledge gain, and trust. Findings show the chatbot delivers concise, personalized explanations, stimulates curiosity, reflection, and deliberation, and is well-received, yet raises concerns about truthfulness, transparency, and potential bias. The authors offer design guidelines and extend the MATCH model for trustworthy AI in civic education, highlighting how to balance informative, deliberative engagement with accountability and external validation to enable responsible use in political decision-making.

Abstract

Voting advice applications (VAAs), which have become increasingly prominent in European elections, are seen as a successful tool for boosting electorates' political knowledge and engagement. However, VAAs' complex language and rigid presentation constrain their utility to less-sophisticated voters. While previous work enhanced VAAs' click-based interaction with scripted explanations, a conversational chatbot's potential for tailored discussion and deliberate political decision-making remains untapped. Our exploratory mixed-method study investigates how LLM-based chatbots can support voting preparation. We deployed a VAA chatbot to 331 users before Germany's 2024 European Parliament election, gathering insights from surveys, conversation logs, and 10 follow-up interviews. Participants found the VAA chatbot intuitive and informative, citing its simple language and flexible interaction. We further uncovered VAA chatbots' role as a catalyst for reflection and rationalization. Expanding on participants' desire for transparency, we provide design recommendations for building interactive and trustworthy VAA chatbots.
Paper Structure (53 sections, 3 figures, 6 tables)

This paper contains 53 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Functionality of Voting Advice Applications (VAAs)
  4. Limitations of Traditional VAAs
  5. LLM-Based Chatbot for Political Engagement: Promises and Gaps
  6. Trust in AI
  7. Methods
  8. Approach to Chatbot Design
  9. System design: LLM-Based Chatbot
  10. Technical Implementation
  11. Interaction Design
  12. Participant Recruitment and Demographics
  13. Study Procedure and Tasks
  14. Chatbot Interaction and Survey
  15. Follow-Up Interviews
  16. ...and 38 more sections

Figures (3)

  • Figure 1: Examples of guidance from the chatbot in English. The actual user study was performed in German.
  • Figure 2: Conceptual walk-through of the interaction with the chatbot (as explained in section \ref{['sec:interaction_design']}).
  • Figure 3: Overall flow of the study (as explained in Section \ref{['sec:study_procedure']}).