Verbalized Algorithms

TL;DR

Verbalized Algorithms address the unreliability of end-to-end LLM reasoning for algorithmic tasks by embedding classical discrete algorithms within an LLM-guided framework. The core idea is to replace a single atomic operation with LLM-based yes/no checks while preserving the algorithm's control flow, enabling correctness guarantees when the LLM's reward model is accurate. The paper demonstrates this paradigm with Verbalized Sorting and Verbalized Clustering, showing that VA variants can outperform naive baselines and, in some cases, match the performance of much larger models; notably, Powersort uses $O(n \log n)$ comparisons, while Bitonic sorting achieves $O(n(\log n)^2)$ parallel-time complexity. This approach leverages parallel sorting theory and heavy-tailed similarity kernels to yield practical improvements for RAG and NLP tasks, particularly in low-resource settings.

Abstract

Instead of querying LLMs in a one-shot manner and hoping to get the right answer for a reasoning task, we propose a paradigm we call \emph{verbalized algorithms} (VAs), which leverage classical algorithms with established theoretical understanding. VAs decompose a task into simple elementary operations on natural language strings that they should be able to answer reliably, and limit the scope of LLMs to only those simple tasks. For example, for sorting a series of natural language strings, \emph{verbalized sorting} uses an LLM as a binary comparison oracle in a known and well-analyzed sorting algorithm (e.g., bitonic sorting network). We demonstrate the effectiveness of this approach on sorting and clustering tasks.

Theorems & Definitions (1)

• Definition 1: Verbalized Algorithms