Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models
John Hawkins
TL;DR
This paper tackles evaluating reasoning in transformer-decoder language models by highlighting architectural constraints and latent-variable structure that shape their reasoning capabilities. It introduces enigme, an open-source, template-based generator that creates text-only puzzles across three classes—numeric, sequence, and physics—to stress abstract reasoning and world-model formation while avoiding reliance on memorised data. The generation pipeline is parameterized by a dimension/complexity setting, enabling large numbers of diverse variants and controlled difficulty for benchmarking current LLMs and future AI architectures. The work aims to provide a practical, extensible tool for rigorous reasoning evaluation with potential applications in education and cognitive science, addressing concerns about evaluation bias and the memorisation problem in large language models.
Abstract
Transformer-decoder language models are a core innovation in text based generative artificial intelligence. These models are being deployed as general-purpose intelligence systems in many applications. Central to their utility is the capacity to understand natural language commands and exploit the reasoning embedded in human text corpora to apply some form of reasoning process to a wide variety of novel tasks. To understand the limitations of this approach to generating reasoning we argue that we need to consider the architectural constraints of these systems. Consideration of the latent variable structure of transformer-decoder models allows us to design reasoning tasks that should probe the boundary of their capacity to reason. We present enigme, an open-source library for generating text-based puzzles to be used in training and evaluating reasoning skills within transformer-decoder models and future AI architectures.