PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

TL;DR

PromptAid addresses the challenge of prompting large language models by providing a visual analytics interface that supports exploration, perturbation, testing, and iteration of prompts. It combines keyword perturbations, paraphrasing perturbations, and in-context few-shot example recommendations within six coordinated views and semi-automatic back-end guidance, enabling non-experts to craft higher-quality prompts with reduced cognitive load. Two case studies and a controlled user study demonstrate that PromptAid improves performance and user experience over baseline prompting interfaces, highlighting its potential to democratize effective prompt engineering. The work also outlines future directions for interpretability, provenance, and broader task applicability to further enhance practical impact.

Abstract

Large Language Models (LLMs) have gained widespread popularity due to their ability to perform ad-hoc Natural Language Processing (NLP) tasks with a simple natural language prompt. Part of the appeal for LLMs is their approachability to the general public, including individuals with no prior technical experience in NLP techniques. However, natural language prompts can vary significantly in terms of their linguistic structure, context, and other semantics. Modifying one or more of these aspects can result in significant differences in task performance. Non-expert users may find it challenging to identify the changes needed to improve a prompt, especially when they lack domain-specific knowledge and lack appropriate feedback. To address this challenge, we present PromptAid, a visual analytics system designed to interactively create, refine, and test prompts through exploration, perturbation, testing, and iteration. PromptAid uses multiple, coordinated visualizations which allow users to improve prompts by using the three strategies: keyword perturbations, paraphrasing perturbations, and obtaining the best set of in-context few-shot examples. PromptAid was designed through an iterative prototyping process involving NLP experts and was evaluated through quantitative and qualitative assessments for LLMs. Our findings indicate that PromptAid helps users to iterate over prompt template alterations with less cognitive overhead, generate diverse prompts with help of recommendations, and analyze the performance of the generated prompts while surpassing existing state-of-the-art prompting interfaces in performance.

Figures (5)

• Figure 1: PromptAid employs a multi-phase approach: templates are embedded in a latent space and clustered based on similarity in the exploration phase. In the perturbation phase, contextual keywords, paraphrases, and in-context examples are recommended using KD-Tree, the Parrot library, and KNN, respectively. Users can then test alterations on data points of interest in the testing phase. The frontend interface employs visual analytics to streamline these processes iteratively, leading to the generation of desired prompt templates.
• Figure 2: Case study #1 using linguistic perturbations (keywords and paraphrasing) on the RoBERTa-base model for zero-shot settings. In a two step perturbation the accuracy of the prompt template increases from a 60% accuracy to a 80% accuracy on the test data set.
• Figure 3: Case study #2 using Contextual perturbations GPT-2 model for K-shot settings with optimal K returned as $k=2$. The accuracy of the prompt template increases from 30% to 80% by adding few-shot examples recommended by the system.
• Figure 4: The baseline interface for the user study inline to most commonly available interfaces available for prompting LLMs.
• Figure 5: Participant ratings from the user study; median ratings are indicated in gray.