Personas with Attitudes: Controlling LLMs for Diverse Data Annotation

TL;DR

The results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making this approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.

Abstract

We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making our approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.

Figures (13)

• Figure 1: Boxplots of macro-average F1 scores achieved in 1,000 different persona-based LLM annotation (pers.) and 1,000 baseline LLM annotation runs (no pers.) for a) Mistral and c) Qwen. Boxplots of macro-average F1 scores achieved in 30 additional annotation runs for the 30 personas with min, median and max alignment to the human majority vote label for b) Mistral and d) Qwen.
• Figure 2: Macro-average F1 scores of majority vote performance for ten different persona-prompted LLM crowds (blue) and baseline LLM crowds (red) of sizes increasing from 1 to 100.
• Figure 3: Intra- and inter-cluster cosine distances of persona-space clusters measured in label embedding space resulting from Qwen annotations. Values are normalized per row and lighter-colored cells represent lower average distances between the respective clusters. The inset zooms in on clusters with IDs from 1,000 to 1,100.
• Figure 4: Boxplots of shifts in average toxicity labels assigned to instances in the AAE and anti-Black datasets. The shifts are on a persona-level and are calculated as the difference in average toxicity label of the manually changed black and conservative personas relative to the original, neutral persona.
• Figure A.1: Macro-average F1 score of majority vote performance for 1,000 permutations of the same persona-prompted LLM crowd (blue) and baseline LLM crowd (red) of sizes increasing from 1 to 100.