On The Role of Reasoning in the Identification of Subtle Stereotypes in Natural Language

TL;DR

The paper investigates the role of reasoning, especially multi-step chain-of-thought prompts, in zero-shot stereotype identification within open-source LLMs. Using a modified StereoSet, it demonstrates that prompting strategies that elicit deeper reasoning significantly improve accuracy, coverage, and interpretability of stereotype detection across several models, though benefits vary by model size. The findings suggest that reasoning-based prompting and self-consistency decoding are key for reliable bias detection and have implications for mitigation pipelines and alignment approaches like Constitutional AI. Overall, the work highlights the importance of reasoning as a central design ingredient for detecting and mitigating stereotypes in natural language generation systems.

Abstract

Large language models (LLMs) are trained on vast, uncurated datasets that contain various forms of biases and language reinforcing harmful stereotypes that may be subsequently inherited by the models themselves. Therefore, it is essential to examine and address biases in language models, integrating fairness into their development to ensure that these models do not perpetuate social biases. In this work, we demonstrate the importance of reasoning in zero-shot stereotype identification across several open-source LLMs. Accurate identification of stereotypical language is a complex task requiring a nuanced understanding of social structures, biases, and existing unfair generalizations about particular groups. While improved accuracy is observed through model scaling, the use of reasoning, especially multi-step reasoning, is crucial to consistent performance. Additionally, through a qualitative analysis of select reasoning traces, we highlight how reasoning improves not just accuracy, but also the interpretability of model decisions. This work firmly establishes reasoning as a critical component in automatic stereotype detection and is a first step towards stronger stereotype mitigation pipelines for LLMs.

Figures (11)

• Figure 1: Examples of prompt templates used in the experiments. In Jump-to-Conclusion, the model is expected to provide an answer up-front without any reasoning. In Analyze-Only, the model is prompted to analyze the problem before providing the answer. Finally, in Analyze and Summarize, the model is prompted to summarize its analysis before providing a final answer.
• Figure 2: Confusion matrices across all experiments. Results in the top row (red) correspond to Vicuna, the middle row is for Llama-2-Chat (blue), and those of the third row (purple) are for Mistral models. A and B correspond to model responses of stereotypical and not stereotypical, respectively. Predicted labels correspond to rows and true labels correspond to columns.
• Figure 3: Example reasoning traces generated by Llama-2-Chat-70B for a continuation that reinforces stereotypes across the different prompt approaches. Text highlighted in red potentially relates to the model producing an incorrect response. In blue is text potentially related to the model producing a correct response.
• Figure 4: Three analysis generations excerpts for Llama-2-Chat-70B. The first two are examples of traces that ultimately led to a correct response. The third pane provides an example of reasoning that did not lead to a correct answer.
• Figure 5: Accuracy comparison across the three prompt variations for Vicuna, Llama-2-Chat, and Mistral models. Circular and triangular markers correspond to smaller and larger model variants, respectively.