Understanding and Supporting Formal Email Exchange by Answering AI-Generated Questions

TL;DR

The paper tackles the time and cognitive burden of crafting formal email replies by replacing open-ended prompt construction with an LLM-driven QA-based workflow in a system called ResQ. By generating questions from incoming emails, collecting user answers, and drafting replies, the approach aims to streamline writing while preserving message quality. Across a controlled lab study and a five-day field study in real email use, ResQ improved efficiency and reduced cognitive load, with generally maintained or enhanced reply quality, though it introduced reductions in perceived agency and could affect interpersonal distance. The work highlights opportunities for adaptive QA-based mediation in AI-assisted communication, while also outlining challenges such as maintaining authorial sense of self and managing relationship dynamics, and it points to future cross-cultural evaluations and design refinements.

Abstract

Replying to formal emails is time-consuming and cognitively demanding, as it requires crafting polite phrasing and providing an adequate response to the sender's demands. Although systems with Large Language Models (LLMs) were designed to simplify the email replying process, users still need to provide detailed prompts to obtain the expected output. Therefore, we proposed and evaluated an LLM-powered question-and-answer (QA)-based approach for users to reply to emails by answering a set of simple and short questions generated from the incoming email. We developed a prototype system, ResQ, and conducted controlled and field experiments with 12 and 8 participants. Our results demonstrated that the QA-based approach improves the efficiency of replying to emails and reduces workload while maintaining email quality, compared to a conventional prompt-based approach that requires users to craft appropriate prompts to obtain email drafts. We discuss how the QA-based approach influences the email reply process and interpersonal relationship dynamics, as well as the opportunities and challenges associated with using a QA-based approach in AI-mediated communication.

Figures (7)

• Figure 1: The overview of the process of creating a reply message using ResQ. A) The LLM first generates multiple-choice questions in JSON format. B) Users select their desired responses to their counterparts. C) The LLM then generates a reply draft in JSON format based on the users' selections. D) Finally, users review and edit the LLM-generated draft before sending the reply.
• Figure 2: Interface of ResQ. On the left, the content of the email is displayed, with an editor and a "Reply" button below for sending a reply. In the center, questions and options for users are shown, allowing the creation of custom options if needed. Additionally, the section of the email corresponding to the selected question is highlighted. On the right, fields are provided to customize the reply generated by the LLM, including options to specify the relationship with the counterpart and buttons to choose the formality, tone, and length of the email. A free-text input field and a "Generate Reply" button are also below.
• Figure 3: Inclusion of Other in the Self (IOS). The diagram above the x-axis is an example of what participants were shown when responding to the questionnaire. The degree of overlap between the two circles represents the psychological distance between oneself and others.
• Figure 4: Results of participants' efficiency and cognitive load of replying to emails. Left: Efficiency for replying to emails. Middle: Prompt character count. Right: Cognitive load for replying to emails. The significant differences between conditions were from post-hoc analysis after doing one-way repeated measure ANOVA.
• Figure 5: Summary of Likert scale responses. Measurements H2 and H3-a were assessed by third-party evaluators rather than the participants themselves. The significant differences between conditions were from post-hoc analysis after one-way repeated measure ANOVA or the Friedman test (* and ** indicate the significance found at levels of 0.05 and 0.01, respectively).