Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models

TL;DR

Knowledge Crosswords introduces a geometric knowledge reasoning benchmark for LLMs, where incomplete knowledge networks bounded by geometric constraints must be completed. It formalizes the task with $G_Q$, constructs the dataset from filtered $YAGO$, and defines distractor sampling across easy/medium/hard settings, plus w/ and w/o knowledge, evaluated via PC and FC. The study reveals that LLMs have preliminary geometric reasoning abilities, that noisy knowledge aids performance, and that standard prompting yields limited gains; it also proposes two instruction-based approaches, Staged Prompting and Verify-All, with Verify-All delivering robust gains and Staged Prompting offering strong gains with more capable LLMs. Overall, the work highlights new challenges in robust geometric reasoning for LLMs and provides a testbed and methods that push toward non-linear, constraint-aware reasoning over parametric knowledge.

Abstract

We propose Knowledge Crosswords, a geometric knowledge reasoning benchmark consisting of incomplete knowledge networks bounded by structured factual constraints, where LLMs are tasked with inferring the missing facts to meet all constraints. The novel setting of geometric knowledge reasoning necessitates new LM abilities beyond existing atomic/linear multi-hop QA, such as backtracking, verifying facts and constraints, reasoning with uncertainty, and more. Knowledge Crosswords contains 2,101 individual problems, covering diverse knowledge domains, and is further divided into three difficulty levels. We conduct extensive experiments to evaluate existing LLMs and approaches on Knowledge Crosswords. Results demonstrate that baseline approaches struggle with larger knowledge networks and semantically-equivalent entity distractors. In light of their limitations, we propose two new approaches, Staged Prompting and Verify-All, to augment LLMs' abilities for error-aware backtracking and constraint verification. Our Verify-All significantly outperforms prior methods and is more robust towards problems in the hard subset. Further analysis shows that geometric knowledge reasoning poses new challenges to LLMs' knowledge abilities, particularly in robustness towards varying option orders, complex structural constraints in knowledge networks, "none of the above" scenarios, and more.

Figures (8)

• Figure 1: Illustration of the differences of atomic, linear (multi-hop), and geometric knowledge reasoning. Each step of atomic or linear QA leads to a unique and definite (intermediate) answer, while multiple candidates in each step should be jointly considered to satisfy structural constraints in geometric knowledge reasoning.
• Figure 2: Overview of Knowledge Crosswords and two proposed approaches, Staged Prompting and Verify-All. Each knowledge crossword includes task instruction, factual constraints, and multiple-choice QA options. In each stage of Staged Prompting, LLMs ① solve one blank based on one remaining constraint; ② update the status by filling in the proposed answer; then ③ verify filled constraints to proceed or backtrack. In Verify-All, LLMs propose a combination of ① candidates and ② verify all constraints with those candidates, and repeat this process until all constraints are met.
• Figure 3: Problem distribution based on the correctness under the w/ knowledge and w/o knowledge settings using CoT and Verify-All. The results indicate that while easier problems are mainly hindered by a lack of knowledge, the bottleneck for hard problems lies in geometric knowledge reasoning abilities.
• Figure 4: FC and PC (%) under the w/o knowledge setting using CoT and Staged Prompting with different orders of options evaluated on the hard subset.
• Figure 5: FC and PC (%) for problems with specific structural patterns using CoT. "A-B" denotes two connected blanks, "A-B-C" denotes a chain of three blanks, "cycle" denotes a cycle of three or more blanks, and "overall" denotes the performance on all question graphs.