Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization

TL;DR

This paper tackles the challenge of solving lateral-thinking BrainTeaser tasks with large language models by introducing an iterative, human-in-the-loop Chain-of-Thought prompting framework. It starts with naive CoT prompts, then uses detailed human evaluation to identify reasoning gaps and informs successive refinements, culminating in the Iterated CoT prompting with a New CoT-Mix set that emphasizes disconfirming incorrect options. The approach yields significant gains on adversarial data and provides insights into data collection and synthesis, including limitations observed for context-reconstruction (CR) questions. The work demonstrates practical improvements in prompt robustness and offers a template for combining model reasoning with human feedback to mitigate memorization and improve dataset quality. Overall, the method advances prompt engineering for lateral-thinking tasks and has implications for designing more reliable reasoning systems with open-ended datasets.

Abstract

Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system's ability to significantly improve model performance by optimizing prompts and evaluate the input dataset.