LLM-Generated Black-box Explanations Can Be Adversarially Helpful

TL;DR

Light is shed on the limitations of the black-box explanation setting and advice is provided on the safe usage of LLMs to provide advice on the safe usage of LLMs.

Abstract

Large Language Models (LLMs) are becoming vital tools that help us solve and understand complex problems by acting as digital assistants. LLMs can generate convincing explanations, even when only given the inputs and outputs of these problems, i.e., in a ``black-box'' approach. However, our research uncovers a hidden risk tied to this approach, which we call *adversarial helpfulness*. This happens when an LLM's explanations make a wrong answer look right, potentially leading people to trust incorrect solutions. In this paper, we show that this issue affects not just humans, but also LLM evaluators. Digging deeper, we identify and examine key persuasive strategies employed by LLMs. Our findings reveal that these models employ strategies such as reframing the questions, expressing an elevated level of confidence, and cherry-picking evidence to paint misleading answers in a credible light. To examine if LLMs are able to navigate complex-structured knowledge when generating adversarially helpful explanations, we create a special task based on navigating through graphs. Most LLMs are not able to find alternative paths along simple graphs, indicating that their misleading explanations aren't produced by only logical deductions using complex knowledge. These findings shed light on the limitations of the black-box explanation setting and allow us to provide advice on the safe usage of LLMs.

Figures (8)

• Figure 1: Examples of LLM-generated explanations towards incorrect labels in a commonsense QA question (above) and an NLI question (below). We mark the persuasion strategies with color-coded angle brackets. To save space, the parts that do not contain persuasion strategies are omitted with [...].
• Figure 2: The prompt templates for explaining incorrect ECQA and NLI answers. The texts between double braces are to be replaced by the problem-specific contents.
• Figure 3: Two explanations towards two answer choices for an ECQA problem, where each graph node is analogous to a reasoning unit, and each graph edge serves as a reasoning step.
• Figure 4: Left: Example of a symbolic reasoning graph with non-randomized node names. Right: Example of a symbolic reasoning graph with randomized node names. The graph in string format, the graph plotted. If the path "root $\rightarrow$ 0_1 $\rightarrow$ 0_2 $\rightarrow$ A" is the reasoning path supporting answer A, supporting answer C would need a reasoning path "root $\rightarrow$ 2_1 $\rightarrow$ 2_2 $\rightarrow$ C".
• Figure 5: Success rate vs graph complexity. Left: using the default graph node names. Right: replacing node names like "0_1" with random non-overlapping characters.