Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection

TL;DR

This work tackles the instability and trial-and-error nature of prompting large language models for social bias detection by proposing an adaptive prompting framework that predicts input-specific prompt compositions. It constructs a rich pool of discrete prompting techniques, collects per-composition labels, and trains an encoder to select the best composition for each input, supplemented by a Shapley-value analysis to understand technique interactions. Empirical results across three instruction-tuned LLMs and three bias datasets show that adaptive prompting can outperform fixed compositions and, in many settings, surpass baselines including fine-tuned models, with insights into which techniques most consistently contribute. The study demonstrates generalizability to other tasks and highlights the importance of considering second-order interactions among prompting techniques for reliable bias detection and more efficient LLM use.

Abstract

Recent advances on instruction fine-tuning have led to the development of various prompting techniques for large language models, such as explicit reasoning steps. However, the success of techniques depends on various parameters, such as the task, language model, and context provided. Finding an effective prompt is, therefore, often a trial-and-error process. Most existing approaches to automatic prompting aim to optimize individual techniques instead of compositions of techniques and their dependence on the input. To fill this gap, we propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding. We evaluate it with three large language models on three datasets, comparing compositions to individual techniques and other baselines. The results underline the importance of finding an effective prompt composition. Our approach robustly ensures high detection performance, and is best in several settings. Moreover, first experiments on other tasks support its generalizability.

Figures (10)

• Figure 1: Exemplary excerpt of a prompt composition for social bias detection. While certain techniques might benefit detection performance (say, those with green squares), others might not (red squares). Some parts always need to be present (blue squares). A full prompt example is shown in Figure \ref{['fig:example-prompt']}.
• Figure 2: The three steps of our adaptive prompting approach: (1) Bias labels are collected for all considered prompt compositions. (2) A model is trained on the collected labels to predict the optimal composition for any given text. (3) Given an unknown text, the model is applied to predict and use the optimal prompt composition for that text.
• Figure 3: Social bias detection results on StereoSet (others in Appendix \ref{['sec:appendix-extended-results']}: Figures \ref{['performance-sbic']}--\ref{['performance-cobra']}): Macro F$_1$-score of all prompt compositions with each LLM (baselines shown as vertical lines). Our adaptive prompting approach (Ours StereoSet) outperforms all fixed compositions. Ours SBIC and Ours Cobra are trained on other datasets. The variance over all compositions (shown as box plots) indicates the LLMs' sensitivity to the prompt.
• Figure 4: Network plots of the shapley interactions for the three evaluated LLMs on StereoSet (others in Appendix \ref{['sec:appendix-sv-composition-analysis']}: Figures \ref{['network-sbic']}-\ref{['network-cobra']}), revealing unique interaction structures among the models. Node size represents strengths of first-order interactions. Line width and translucency denote strengths of second-order interactions. Red color denotes positive interaction (increasing the performance), and blue color denotes negative interaction (decreasing the performance).
• Figure 5: Force plots of Shapley values for three variants (top: category in-context demonstrations, middle: similar in-context demonstrations, bottom: random in-context demonstrations) of the composition game for the Mistral model on Stereoset. In all three settings the in-context demonstrations are most influential. Red color denotes positive attribution (increasing the performance), and blue color denotes negative attribution (decreasing the performance).