Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking

TL;DR

This work critically assesses the generalization capabilities of structural knowledge prompting (SKP) for large language models by introducing the SUBARU benchmark, a 9-task suite spanning entity, triple, and subgraph granularity across three difficulty levels. The authors frame four evaluation axes—Granularity, Transferability, Scalability, and Universality—and provide a rigorous training/evaluation protocol to analyze SKP adapters and KG encoders. Key findings show SKP delivers coarse-grained, structure-informed guidance across tasks, but struggles with fine-grained accuracy and inductive generalization for unseen entities, while simple adapters like MLP often outperform more complex designs. The SUBARU benchmark and analyses aim to guide future SKP work toward enhancing factual accuracy in LLMs, with implications for broader knowledge-intensive applications and cross-model transfer.

Abstract

Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty.

Figures (18)

• Figure 1: An intuition of the structural knowledge prompting paradigm in current LLM research.
• Figure 2: An overview of the construction pipeline of SUBARU benchmark and the evaluation process. In SUBARU, we construct 9 different tasks with multiple granularity (entity/triple/subgraph) and multiple difficulty levels for comprehensive evaluation of the generalization capability of the structural knowledge prompting paradigm.
• Figure 3: A general prompt template for all tasks.
• Figure 4: The transferability experiments among different granularities and levels with MLP adapter.
• Figure 5: The inductive transfer experiments.