Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners

Chi Hu, Yuan Ge, Xiangnan Ma, Hang Cao, Qiang Li, Yonghua Yang, Tong Xiao, Jingbo Zhu

TL;DR

RankPrompt breaks down the ranking problem into a series of comparisons among diverse responses, leveraging the inherent capabilities of LLMs to generate chains of comparison as contextual exemplars and validate RankPrompt as an effective method for eliciting high-quality feedback from language models.

Abstract

Large Language Models (LLMs) have achieved impressive performance across various reasoning tasks. However, even state-of-the-art LLMs such as ChatGPT are prone to logical errors during their reasoning processes. Existing solutions, such as deploying task-specific verifiers or voting over multiple reasoning paths, either require extensive human annotations or fail in scenarios with inconsistent responses. To address these challenges, we introduce RankPrompt, a new prompting method that enables LLMs to self-rank their responses without additional resources. RankPrompt breaks down the ranking problem into a series of comparisons among diverse responses, leveraging the inherent capabilities of LLMs to generate chains of comparison as contextual exemplars. Our experiments across 11 arithmetic and commonsense reasoning tasks show that RankPrompt significantly enhances the reasoning performance of ChatGPT and GPT-4, with improvements of up to 13%. Moreover, RankPrompt excels in LLM-based automatic evaluations for open-ended tasks, aligning with human judgments 74% of the time in the AlpacaEval dataset. It also exhibits robustness to variations in response order and consistency. Collectively, our results validate RankPrompt as an effective method for eliciting high-quality feedback from language models.

RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners

TL;DR

RankPrompt breaks down the ranking problem into a series of comparisons among diverse responses, leveraging the inherent capabilities of LLMs to generate chains of comparison as contextual exemplars and validate RankPrompt as an effective method for eliciting high-quality feedback from language models.

Abstract

Large Language Models (LLMs) have achieved impressive performance across various reasoning tasks. However, even state-of-the-art LLMs such as ChatGPT are prone to logical errors during their reasoning processes. Existing solutions, such as deploying task-specific verifiers or voting over multiple reasoning paths, either require extensive human annotations or fail in scenarios with inconsistent responses. To address these challenges, we introduce RankPrompt, a new prompting method that enables LLMs to self-rank their responses without additional resources. RankPrompt breaks down the ranking problem into a series of comparisons among diverse responses, leveraging the inherent capabilities of LLMs to generate chains of comparison as contextual exemplars. Our experiments across 11 arithmetic and commonsense reasoning tasks show that RankPrompt significantly enhances the reasoning performance of ChatGPT and GPT-4, with improvements of up to 13%. Moreover, RankPrompt excels in LLM-based automatic evaluations for open-ended tasks, aligning with human judgments 74% of the time in the AlpacaEval dataset. It also exhibits robustness to variations in response order and consistency. Collectively, our results validate RankPrompt as an effective method for eliciting high-quality feedback from language models.
Paper Structure (30 sections, 6 figures, 6 tables, 1 algorithm)

This paper contains 30 sections, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs as Reasoners.
  4. LLMs as Evaluators.
  5. Method
  6. Candidate Generation
  7. Candidate Ranking
  8. Comparative Evaluation of Reasoning Steps
  9. Construction of Comparison Exemplars
  10. Experiment
  11. Experimental Setups
  12. Models.
  13. Tasks and Datasets.
  14. Candidate Generation Setups.
  15. Ranking Setups.
  16. ...and 15 more sections

Figures (6)

  • Figure 1: An overview of Direct Scoring Zheng2023JudgingLW (left) and RankPrompt (right). Direct Scoring independently assigns scores to each candidate, whereas RankPrompt ranks candidates through a systematic, step-by-step comparative evaluation. We present the detailed instructions for comparison in Table \ref{['tab:ranking_template']} and describe the construction of comparison exemplars in Section \ref{['sec:exemplars']}.
  • Figure 2: RankPrompt performs much better than majority voting when the candidate answers are inconsistent. The results are obtained on AQuA-RAT over 5 candidates using gpt-3.5-turbo.
  • Figure 3: Performance of RankPrompt with a correct example vs. an incorrect example when ranking over 5 candidates. The results are obtained with gpt-3.5-turbo.
  • Figure 4: Test accuracy with varying complexity and numbers of comparison exemplars. The results are obtained on GSM8K (left) and CSQA (right) using gpt-3.5-turbo-16k.
  • Figure 5: Consistency rates of Zero Ranking and RankPrompt when ranking 5 candidates shuffled 3 times. The results are obtained with gpt-4.
  • ...and 1 more figures