SR-LLM: Rethinking the Structured Representation in Large Language Model

TL;DR

SR-LLM investigates integrating structured representations (AMR, PST, FOL) with LLMs via training-free SR-NLD and training-dependent Gen-SR. It demonstrates that converting SR into natural language descriptions and fine-tuning on SR data can outperform code-based SR prompts, achieving notable gains on PAWS (+3.17% training-free, +12.38% training-dependent). The work provides a thorough empirical evaluation across 10 NLP tasks, including robustness checks with high-quality SR parsers and larger models, and shows SR-NLD improves weaker models more than strong ones. The findings suggest a path to improve LLM reasoning and interoperability through structured representations.

Abstract

Structured representations, exemplified by Abstract Meaning Representation (AMR), have long been pivotal in computational linguistics. However, their role remains ambiguous in the Large Language Models (LLMs) era. Initial attempts to integrate structured representation into LLMs via a zero-shot setting yielded inferior performance. We hypothesize that such a decline stems from the structure information being passed into LLMs in a code format unfamiliar to LLMs' training corpora. Consequently, we propose SR-LLM, an innovative framework with two settings to explore a superior way of integrating structured representation with LLMs from training-free and training-dependent perspectives. The former integrates structural information through natural language descriptions in LLM prompts, whereas its counterpart augments the model's inference capability through fine-tuning on linguistically described structured representations. Performance improvements were observed in widely downstream datasets, with particularly notable gains of 3.17% and 12.38% in PAWS. To the best of our knowledge, this work represents the pioneering demonstration that leveraging structural representations can substantially enhance LLMs' inference capability. We hope that our work sheds light and encourages future research to enhance the reasoning and interoperability of LLMs by structure data.

Figures (26)

• Figure 1: We propose two novel AMR integration approaches: a training-free method using natural language descriptions and a training-dependent fine-tuning paradigm. Evaluation on PAWS shows +3.17% and +12.38% improvements respectively, contrasting with the -5.18% decline in conventional code-format methods.
• Figure 2: The AMR, PST, and FOL of the sentence "John saw a dog".
• Figure 3: The whole process of SR-LLM in training-free setting. Initially, a task-specific prompt consists of an instruction, input sentence, and input SR structure (AMR is used here). Subsequently, the original AMR undergoes transformation via the AMR-to-NLD module, which employs predefined rules to map the AMR into an easily interpretable natural language description. This description is then subjected to refinement by a language model, ensuring fluency and coherence, resulting in AMR-NLD. Finally, the AMR-NLD is seamlessly integrated into the input, which is then fed into the LLM to generate the ultimate response.
• Figure 4: Base prompt and AMRCOT prompt.(Top) This is the original task prompt, with only the raw text as input, serving as the standards for performance. (Bottom) This is the AMRCOT prompt method proposed by jin2024analyzing, serving as a baseline.
• Figure 5: The whole process of SR-LLM in training-dependent setting. Taking AMR as an example, a dataset called Gen-AMR, created by combining inputs consisting of sentences and their corresponding AMR structures, is utilized for the SFT of LLM to enhance the reasoning capability.