The Mystery of the Pathological Path-star Task for Language Models
Arvid Frydenlund
TL;DR
The paper investigates why the path-star task—a minimal graph-based reasoning task—poses a challenge for autoregressive language models trained with teacher-forcing. It shows that the observed failure (the Clever Hans cheat) is not solely due to data size or exposure bias but arises from representation, permutation, and causal constraints, and it demonstrates that alternative training paradigms (encoder-decoder, non-autoregressive, IAR) and structured data augmentations can overcome these obstacles in certain settings. The authors provide a theoretical basis via RASP that the task is solvable by transformers and outline several algorithmic approaches, including a logarithmic-depth method, while empirically achieving consistent success primarily with encoder-only models under structured sampling. These findings suggest that next-token prediction limitations can be mitigated under appropriate representations and supervision, offering insights into planning-oriented tasks and the design of training regimes for graph-centric reasoning.
Abstract
The recently introduced path-star task is a minimal task designed to exemplify limitations to the abilities of language models (Bachmann and Nagarajan, 2024). It involves a path-star graph where multiple arms radiate from a single starting node and each node is unique. Given the start node and a specified target node that ends an arm, the task is to generate the arm containing that target node. This is straightforward for a human but surprisingly difficult for language models, which did not outperform the random baseline. The authors hypothesized this is due to a deficiency in teacher-forcing and the next-token prediction paradigm. We demonstrate the task is learnable using teacher-forcing in alternative settings and that the issue is partially due to representation. We introduce a regularization method using structured samples of the same graph but with differing target nodes, improving results across a variety of model types. We provide RASP proofs showing the task is theoretically solvable. Finally, we find settings where an encoder-only model can consistently solve the task.