Design Opportunities for Explainable AI Paraphrasing Tools: A User Study with Non-native English Speakers

TL;DR

This paper investigates how non-native English speakers (NNESs) interact with explainable AI paraphrasing by introducing ParaScope, a paraphrasing assistant that integrates five information aids (AI Score, AI Translation, AI Explanation, Example Sentence, Frequency) and logs user interactions. In a lab study with 22 Korean NNESs, the authors show that users’ preferences vary by English proficiency, with global aids favored for efficiency and back-translation frequently used but not decisive for acceptance; users often combine multiple aids to make informed decisions. The results demonstrate improvements in confidence, autonomy, and writing efficiency, while also highlighting risks of information overload and the need for staged, personalized design. The paper offers design implications for adaptive, explainable paraphrasing tools and discusses broader implications for NNES writing tasks and potential language-learning benefits, while acknowledging limitations of the in-lab, homogenous sample.

Abstract

We investigate how non-native English speakers (NNESs) interact with diverse information aids to assess and select AI-generated paraphrases. We develop ParaScope, an AI paraphrasing assistant that integrates diverse information aids, such as back-translation, explanations, and usage examples, and logs user interaction data. Our in-lab study with 22 NNESs reveals that user preferences for information aids vary by language proficiency, with workflows progressing from global to more detailed information. While back-translation was the most frequently used aid, it was not a decisive factor in suggestion acceptance; users combined multiple information aids to make informed decisions. Our findings demonstrate the potential of explainable AI paraphrasing tools to enhance NNESs' confidence, autonomy, and writing efficiency, while also emphasizing the importance of thoughtful design to prevent information overload. Based on these findings, we offer design implications for explainable AI paraphrasing tools that support NNESs in making informed decisions when using AI writing systems.

Figures (8)

• Figure 1: Overview of the ParaScope interface. The interface features two panes: (A) a pane displaying four paraphrased suggestions for the user’s input text and (B) a pane providing information aids. The right pane (B) remains empty until the user interacts with an information aids button.
• Figure 2: Illustrations of the information aids: AI Score ($F_1$), AI Translation ($F_2$), AI Explanation ($F_3$), Example Sentence ($F_4$), and Frequency ($F_5$) for the four paraphrased suggestions of the original text ("She's always nice and welcome for new member."). $F_1$ (AI Score): The scores representing paraphrased suggestions' quality are displayed next to each suggestion as both bar graphs and numerical values. $F_2$ (AI Translation): Translated versions of suggestions in users' first language (in this figure, Korean) are displayed next to each suggestion. $F_3$ (AI Explanation): Textual explanation about tones and appropriate contexts for each suggestion is displayed. $F_4$ (Example Sentence): After users search a term using the search box, example sentences containing the term are displayed, allowing users to browse the sentences. $F_5$ (Frequency): After users search a term(s) using the search box, a bar graph visualizing the frequency of the term(s) is displayed. When users scroll down the prototype, they see a line graph depicting the number of occurrences of the term(s) over the years.
• Figure 3: Overview of the user study procedure. After familiarizing themselves with each information aid in the tutorial session, participants wrote an open-ended academic email with ParaScope. After the writing, participants engaged in a post-survey and semi-structured interview about their experiences.
• Figure 4: Feature usage frequencies for each information aid. The orange bar indicates the median value. While AI Score (M=0.46, SD=0.33), AI Translation (M=0.56, SD=0.27), and AI Explanation (M=0.37, SD=0.23) were frequently used in a paraphrasing event, usage frequencies of Example Sentence (M=0.08, SD=0.1) and Frequency (M=0.11, SD=0.17) were comparatively low.
• Figure 5: Number of paraphrasing events per user. Out of the total paraphrasing events (gray bars), we show the number of events with at least one support feature usage (colored bars).