HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games

TL;DR

HardcoreLogic addresses the challenge of evaluating whether large reasoning systems can flexibly apply rules in long-tail logic puzzles, addressing overfitting to canonical formats. It introduces a benchmark with over $5{,}000$ puzzles across $10$ games, systematically transforming canonical problems along three axes: $IC$, $UE$, and $UP$. Evaluations across diverse LRMs show substantial performance drops on the transformed data, with $IC$ being the dominant source of difficulty and non-canonical rule variations also causing errors, suggesting heavy memorization. The benchmark provides a rigorous platform for studying high-level logical reasoning and guiding future improvements in rule-based deduction.

Abstract

Large Reasoning Models (LRMs) have demonstrated impressive performance on complex tasks, including logical puzzle games that require deriving solutions satisfying all constraints. However, whether they can flexibly apply appropriate rules to varying conditions, particularly when faced with non-canonical game variants, remains an open question. Existing corpora focus on popular puzzles like 9x9 Sudoku, risking overfitting to canonical formats and memorization of solution patterns, which can mask deficiencies in understanding novel rules or adapting strategies to new variants. To address this, we introduce HardcoreLogic, a challenging benchmark of over 5,000 puzzles across 10 games, designed to test the robustness of LRMs on the "long-tail" of logical games. HardcoreLogic systematically transforms canonical puzzles through three dimensions: Increased Complexity (IC), Uncommon Elements (UE), and Unsolvable Puzzles (UP), reducing reliance on shortcut memorization. Evaluations on a diverse set of LRMs reveal significant performance drops, even for models achieving top scores on existing benchmarks, indicating heavy reliance on memorized stereotypes. While increased complexity is the dominant source of difficulty, models also struggle with subtle rule variations that do not necessarily increase puzzle difficulty. Our systematic error analysis on solvable and unsolvable puzzles further highlights gaps in genuine reasoning. Overall, HardcoreLogic exposes the limitations of current LRMs and establishes a benchmark for advancing high-level logical reasoning.