Writer-Defined AI Personas for On-Demand Feedback Generation

TL;DR

A concept that generates on-demand feedback, based on writer-defined AI personas of any target audience, is proposed, which was seen as helpful and inspired revisions of text and personas, although it was often verbose and unspecific.

Abstract

Compelling writing is tailored to its audience. This is challenging, as writers may struggle to empathize with readers, get feedback in time, or gain access to the target group. We propose a concept that generates on-demand feedback, based on writer-defined AI personas of any target audience. We explore this concept with a prototype (using GPT-3.5) in two user studies (N=5 and N=11): Writers appreciated the concept and strategically used personas for getting different perspectives. The feedback was seen as helpful and inspired revisions of text and personas, although it was often verbose and unspecific. We discuss the impact of on-demand feedback, the limited representativity of contemporary AI systems, and further ideas for defining AI personas. This work contributes to the vision of supporting writers with AI by expanding the socio-technical perspective in AI tool design: To empower creators, we also need to keep in mind their relationship to an audience.

Figures (7)

• Figure 1: A part of the UI for defining a persona: Hovering over the info button next to each category (e.g. "Style Preferences") provides a corresponding description, alongside examples of attribute-description pairs to facilitate the persona creation process.
• Figure 2: The overall workflow: (1) The writer defines a persona by adding rows of attributes and descriptions in a persona tab. (2) In the text editor, the writer selects a part of the text and then clicks on a persona button in the sidebar to generate feedback from that persona for that piece of text.
• Figure 3: The study procedure, as described in \ref{['sec:procedure']}.
• Figure 4: Overview of the words that participants entered in the attribute-description pairs (see UI in \ref{['fig:guidance']}), lightly scaled by how often these contributed to generated feedback (also annotated as numbers if >1). For example, if a participant defined "writing style: formal, scientific" for a persona and requested feedback four times from this persona, this would contribute counts of four to "writing" and "style" (in \ref{['fig:wordcloud_attributes']}), as well as to "formal" and "scientific" (in \ref{['fig:wordcloud_values']}). We removed stopwords (e.g. "the"). While shown as single words here for a better overview, participants also entered descriptions as phrases, not only single words (e.g. "Provide real-world analogies to ideas in the paper"). Log files including verbatim descriptions are available in the project repository (see link in \ref{['sec:conclusion']}).
• Figure 5: Overview of participants' workflows (in study 2): Each row (y-axis) is one participant. The x-axis shows time as study session progress. Color shows participants' focus on the editor (dark blued) or sidebar (light blue), as derived from logged UI events. Yellow lines indicate when a new persona was added. Overall, some participants switched between writing and engaging with personas more frequently than others. See \ref{['sec:results_strategies']} for details.