Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies

TL;DR

The paper systematically benchmarks GPT-4 on three algorithmic reasoning tasks—ListOps, Arithmetic, and Algebra—whose difficulty is controlled by nesting depth ($Nesting$) and operand count ($Operands$), and compares GPT-4 to GPT-3.5 and a Neural Data Router (NDR). It evaluates a broad set of prompting techniques (Zero-shot, Role, Few-shot, Chain-of-Thought variants, and Self-consistency) and analyzes cross-distribution generalization, finding that Self-consistency with GPT-4 yields the best results, while none of the approaches achieves true systematic generalization. The Neural Data Router is competitive with GPT-3.5 on several splits, and on some harder ListOps cases even surpasses GPT-3.5, but overall GPT-4 remains the strongest baseline. The study highlights that while explicit reasoning prompts improve performance—especially in Arithmetic—current methods fall short of enabling robust extrapolation, pointing to future work in prompting innovations and architectural/training-distribution changes to promote systematic generalization across complex, nested symbolic tasks.

Abstract

Large Language Models (LLMs) have revolutionized the field of Natural Language Processing thanks to their ability to reuse knowledge acquired on massive text corpora on a wide variety of downstream tasks, with minimal (if any) tuning steps. At the same time, it has been repeatedly shown that LLMs lack systematic generalization, which allows to extrapolate the learned statistical regularities outside the training distribution. In this work, we offer a systematic benchmarking of GPT-4, one of the most advanced LLMs available, on three algorithmic tasks characterized by the possibility to control the problem difficulty with two parameters. We compare the performance of GPT-4 with that of its predecessor (GPT-3.5) and with a variant of the Transformer-Encoder architecture recently introduced to solve similar tasks, the Neural Data Router. We find that the deployment of advanced prompting techniques allows GPT-4 to reach superior accuracy on all tasks, demonstrating that state-of-the-art LLMs constitute a very strong baseline also in challenging tasks that require systematic generalization.

Figures (4)

• Figure 1: Performance of the Neural Data Router, GPT-3.5 and GPT-4 using Self-consistency prompting on the test splits. Values represent output accuracy in percentage: the performance of all models and prompting methods clearly decreases on data splits of higher complexity.
• Figure 2: Accuracy and loss of the Neural Data Router during training on the three algorithmic tasks. Val. IID and Val. OOD refer to in-distribution and out-of-distribution validation sets, respectively. The model overfits the in-distribution split on all tasks, failing to generalize to more difficult samples.
• Figure 3: Performance gain measured as percentage accuracy resulting from each prompting method on GPT-4 compared to Zero-shot baseline. The accuracy gains from the best prompting methods are concentrated in simpler data splits, especially on Arithmetic.
• Figure :