GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs

TL;DR

This work introduces Goal-Reversed Prompting (GRP), a method for zero-shot pairwise evaluation of large language models that shifts the task from choosing the better answer to identifying the worse one. By using SOP-based prompts and reverse thinking, GRP aims to reduce biases and improve discrimination between competing responses. Empirical results on JudgeBench with several closed-source models show consistent evaluation gains, with GPT-4o achieving about a 4.5 percentage-point improvement and Claude-3.5-Sonnet around 6% when using GRP. The approach offers a simple, effective tool to strengthen LLM evaluation pipelines, though future work should test open-source models and extended prompt variants to broaden applicability.

Abstract

Using Large Language Models (LLMs) to evaluate and compare two answers from different models typically involves having LLM-based judges select the better answer. However, humans often approach problem-solving from a reverse perspective, for instance, by choosing the worse option instead of the better one in a pairwise comparison. Generally, this kind of reverse thinking plays a crucial role in human reasoning and decision-making and can further test the difference between original and reverse thought processes simultaneously. To address the above issue, in this paper, we propose a Goal-Reversed Prompting (GRP) approach for pairwise evaluation that shifts the original task from selecting the better answer to choosing the worse one. We encourage LLMs to think in reverse by prompting LLMs to identify the worse response. Experiments on closed-source models demonstrate that GRP significantly enhances evaluation capabilities, outperforming the prompt template with the original goal.

Figures (1)

• Figure 1: The pipeline of pairwise comparison with different goals (the second row illustrates the reverse).