Towards Designing a Question-Answering Chatbot for Online News: Understanding Questions and Perspectives

TL;DR

This work interviews journalists and readers to understand how they answer questions from readers currently and how they want to use a QA chatbot for this purpose and proposes a framework for designing effective QA chatbots in newsrooms.

Abstract

Large Language Models (LLMs) have created opportunities for designing chatbots that can support complex question-answering (QA) scenarios and improve news audience engagement. However, we still lack an understanding of what roles journalists and readers deem fit for such a chatbot in newsrooms. To address this gap, we first interviewed six journalists to understand how they answer questions from readers currently and how they want to use a QA chatbot for this purpose. To understand how readers want to interact with a QA chatbot, we then conducted an online experiment (N=124) where we asked each participant to read three news articles and ask questions to either the author(s) of the articles or a chatbot. By combining results from the studies, we present alignments and discrepancies between how journalists and readers want to use QA chatbots and propose a framework for designing effective QA chatbots in newsrooms.

Figures (9)

• Figure 1: Web interface for the study. Participants used this interface to read three anonymous articles and ask questions to the authors or a chatbot. The screenshot shows the interface for an article. (a) A PDF reader for reading the article. (b) Instruction for completing the task. (c) Input boxes for writing the questions. (d) Buttons to write a new question (optional) and move forward (enabled only when a participant provides two questions).
• Figure 2: Number of questions asked to the authors and chatbot.
• Figure 3: Complexity of the questions. (a) Rating of the questions based on Bloom's Taxonomy. Participants asked questions higher on the taxonomy more frequently to the authors than the chatbot. (b) Length of the questions. On average, questions asked to the authors had higher length than those asked to the chatbot.
• Figure 4: Linguistic features of the questions. (a) Sentiment of the questions based on the VADER model DBLP:conf/icwsm/HuttoG14. (b) Second person (you, yours, etc.) reference in the questions. (c) Readability Index (Flesch Kincaid Grade).
• Figure 5: Structural Equation Model (SEM) for measuring the effect of perceived quality and news outlet preference. Here, perceived quality is a latent variable, derived from readability, credibility, and expertise ratings provided by the participants. We determined participants' preference for news consumption (left, neutral, right) from their reading preference (Figure \ref{['fig:participant']}e).