Are Language Models Up to Sequential Optimization Problems? From Evaluation to a Hegelian-Inspired Enhancement
Soheil Abbasloo
TL;DR
This work investigates how Large Language Models handle Sequential Optimization Problems (SOPs) and introduces WorldGen, a dynamic benchmark that generates unseen $n$-dimensional worlds with controllable complexity to test SOP-solving abilities. It reveals that LLMs perform well on simple SOPs but struggle as problem complexity increases, motivating a Hegelian-inspired enhancement called ACE, which integrates a three-party dialectical loop—Actor, Critic, and Synthesizer—without retraining. ACE significantly boosts SOP performance, achieving up to 88% success in 3-D L1 problems with GPT-4-32K and outperforming prompts-based baselines and other multi-agent schemes, albeit with some limits when base models lack foundational capabilities. The work highlights the potential of combining philosophical reasoning principles with prompting techniques to create more robust, resource-conscious AI systems for complex optimization tasks.
Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities across numerous fields, presenting an opportunity to revolutionize optimization problem-solving, a crucial, ubiquitous, and complex domain. This paper explores the proficiency of LLMs in handling Sequential Optimization Problems (SOPs). We introduce WorldGen, a dynamic framework for generating unseen SOPs with controllable complexities, to evaluate LLM performance. Our initial observations reveal that while LLMs perform well on simple SOPs, their performance significantly degrades with increased complexity. Motivated by this, we revisit philosophical hypotheses on reasoning to enhance LLM performance. Inspired by the influential framework of Hegelian Dialectics, we propose ACE, demonstrating how the performance of LLMs in SOP contexts can be significantly improved without any retraining or further fine-tuning.