Give me a hint: Can LLMs take a hint to solve math problems?

TL;DR

This work proposes giving "hints" to improve the language model's performance on advanced mathematical problems, taking inspiration from how humans approach math pedagogically and demonstrates the effectiveness of this approach by evaluating various diverse LLMs.

Abstract

While state-of-the-art LLMs have shown poor logical and basic mathematical reasoning, recent works try to improve their problem-solving abilities using prompting techniques. We propose giving "hints" to improve the language model's performance on advanced mathematical problems, taking inspiration from how humans approach math pedagogically. We also test robustness to adversarial hints and demonstrate their sensitivity to them. We demonstrate the effectiveness of our approach by evaluating various diverse LLMs, presenting them with a broad set of problems of different difficulties and topics from the MATH dataset and comparing against techniques such as one-shot, few-shot, and chain of thought prompting.

Figures (8)

• Figure 1: A comparison of various prompting techniques
• Figure 2: Left: Comparing different prompting techniques, hinting boosts performance and CoT performs worst, as explained in section \ref{['Evaluating Hint based prompting']}. Right: Effect of adversarial hints: Adversarial hinting greatly reduces performances, as explained in section \ref{['Evaluating Adversarial Hinting']}
• Figure 3: Adversarial and random hinting strategies
• Figure 4: Examples of Base Prompting techniques
• Figure 5: Examples of Hint-based Prompting techniques: Hinting, One-shot Hinting, and Few-Shot Hinting