Hint of Pseudo Code (HoPC): Zero-Shot Step by Step Pseudo Code Reasoning Prompting

TL;DR

HoPC addresses zero-shot multi-step reasoning gaps by introducing a hint-chain prompting framework that produces problem decomposition, semantic pseudo code reasoning, and answer extraction without external interpreters. It demonstrates competitive or superior performance to zero-shot CoT and PoT on arithmetic and commonsense benchmarks across GPT-3.5-Turbo, Qwen2.5, and Llama3-8b, while offering an interpretable mind map of the model's reasoning. By leveraging general pseudo code as a semantic interpreter, HoPC improves logical reasoning and transparency in zero-shot settings. The work suggests practical, interpreter-free prompting strategies that can generalize to diverse reasoning tasks.

Abstract

Prompting a language model (LM) is an increasingly important research topic for better utilization of large language models (LLMs). While simple prompting is effective for single-step questions, it fails to activate the correct knowledge path for multi-step reasoning tasks consistently. The few-shot Chain of Thought (CoT), serves as an advanced prompting strategy that explains and demonstrates the reasoning process to the LLM, outperforming simple prompting in challenging reasoning tasks such as arithmetic and common-sense reasoning. The Program of Thought (PoT) aims to generate text and programming language solutions for multi-step reasoning problems. In zero-shot CoT, the prompt is simply ``Let's think step by step'', which is overly simplistic and does not adequately demonstrate a robust reasoning process for complex reasoning challenges. Additionally, PoT requires an extra interpreter to execute the answer and struggles with semantic reasoning problems like StrategyQA. This paper introduces a novel Hint of Pseudo Code (HoPC) prompting technique that does not require extra interpreter as in PoT and incorporates a more powerful zero-shot problem decomposition and semantic code reasoning capabilities than zero-shot CoT. It consists of three components: problem decomposition, semantic code reasoning, and answer extraction. We prompt these components as hints in a sequential, step by step manner, making it easy to tailor and explain for various tasks.

Figures (10)

• Figure 1: Different structures of zero-shot prompting. (a) Direct prompt to the LLM. (b) Zero-shot CoT prompting shows the ability of "Let's think step by step" NEURIPS2022_8bb0d291. (c) Zero-shot PoT shows a ability of prompting to generating code chen2023program. (d) Our HoPC follows the structure of zero-shot CoT but performs a better reasoning process.
• Figure 2: Different reasoning process of zero-shot prompting. (a) Zero-shot CoT does step by step problem decomposition NEURIPS2022_8bb0d291. (b) Zero-shot PoT does code reasoning chen2023program. (c) Our HoPC does step by step Pseudo code reasoning.
• Figure 3: An Example output of HoPC, the orange part is our HoPC prompt and it shows the problem decomposition, the pseudo code reasoning, and the answer extraction. For every questions, we firstly decompose them into sub-questions and do pseudo code reasoning on each sub-questions. Lastly, we combine every pseudo code reasoning and do an answer extraction to achieve the final answer.
• Figure 4: An arithmetic example of zero-shot CoT, zero-shot PoT and HoPC. These three examples show three different approaches to reach the same arithmetic question. In zero-shot CoT, it goes through the questions briefly and do little calculations to achieve the output which is wrong. In zero-shot PoT, it provides a good written program and a correct answer. However, it is with a long prompt and requires an extra interpreter. In our HoPC, we use a relatively short prompt and generates a long reasoning process with a good structure. Our HoPC with a more precise and structured reasoning process, it gets a correct answer without extra helps. Our HoPC, without an extra program interpreter, but with a more precise and structured reasoning process, gets a same correct answer as PoT does. Note: the orange parts are the prompts.
• Figure 5: A logical reasoning example of zero-shot CoT, zero-shot PoT and HoPC. These three examples show three different approaches to reach the same question. In zero-shot CoT, the lacking of structured reasoning process is resulted in a short and wrong output. In zero-shot PoT, it gives a well written python code, however, this code does not provide a final correct answer. In Our HoPC, it generates reasoning processes by decompose the questions and provides pseudo code reasoning on each of sub-questions. Our HoPC with a more precise and structured reasoning process, gets a correct answer. Note: the orange parts are the prompts.