DaVinci at SemEval-2024 Task 9: Few-shot prompting GPT-3.5 for Unconventional Reasoning

TL;DR

This study investigates few-shot prompting of GPT-3.5 for SemEval-2024 BrainTeaser tasks, comprising Sentence Puzzles and Word Puzzles that require lateral reasoning. It introduces task-specific prompting strategies and evaluates zero-shot, two-shot, and five-shot configurations, revealing divergent effects: Word Puzzles benefit from more examples while Sentence Puzzles often suffer from non-generalizable prompts. The findings highlight how the nature of the puzzle influences generalization under few-shot regimes and point to future directions such as chain-of-thought prompting and model fine-tuning to better capture unconventional reasoning. Overall, the work provides actionable insights into leveraging LLMs for cognitive-style reasoning tasks beyond standard commonsense or logical inference.

Abstract

While significant work has been done in the field of NLP on vertical thinking, which involves primarily logical thinking, little work has been done towards lateral thinking, which involves looking at problems from an unconventional perspective and defying existing conceptions and notions. Towards this direction, SemEval 2024 introduces the task of BRAINTEASER, which involves two types of questions -- Sentence Puzzles and Word Puzzles that defy conventional common-sense reasoning and constraints. In this paper, we tackle both types of questions using few-shot prompting on GPT-3.5 and gain insights regarding the difference in the nature of the two types. Our prompting strategy placed us 26th on the leaderboard for the Sentence Puzzle and 15th on the Word Puzzle task.