Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints

Kaikai An, Shuzheng Si, Helan Hu, Haozhe Zhao, Yuchi Wang, Qingyan Guo, Baobao Chang

TL;DR

Rethinking Semantic Parsing for Large Language Models shows that directly injecting semantic parsing results into LLMs often hurts performance, unlike smaller models where parsing can help. It introduces SENSE, a prompting strategy that embeds semantic hints to coax LLMs to leverage internal semantic understanding without exposing explicit parsing outputs. Across GLUE understanding tasks and generation tasks (paraphrasing, simplification, machine translation), SENSE delivers consistent gains and produces outputs that more closely align with linguistic metrics, supported by analyses comparing paradigms, CoT, and attention patterns. The work demonstrates that semantic information can enhance LLM capabilities when delivered as prompts-based hints, with implications for robust cross-task and multilingual NLP.

Abstract

Semantic Parsing aims to capture the meaning of a sentence and convert it into a logical, structured form. Previous studies show that semantic parsing enhances the performance of smaller models (e.g., BERT) on downstream tasks. However, it remains unclear whether the improvements extend similarly to LLMs. In this paper, our empirical findings reveal that, unlike smaller models, directly adding semantic parsing results into LLMs reduces their performance. To overcome this, we propose SENSE, a novel prompting approach that embeds semantic hints within the prompt. Experiments show that SENSE consistently improves LLMs' performance across various tasks, highlighting the potential of integrating semantic information to improve LLM capabilities.

Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints

TL;DR

Rethinking Semantic Parsing for Large Language Models shows that directly injecting semantic parsing results into LLMs often hurts performance, unlike smaller models where parsing can help. It introduces SENSE, a prompting strategy that embeds semantic hints to coax LLMs to leverage internal semantic understanding without exposing explicit parsing outputs. Across GLUE understanding tasks and generation tasks (paraphrasing, simplification, machine translation), SENSE delivers consistent gains and produces outputs that more closely align with linguistic metrics, supported by analyses comparing paradigms, CoT, and attention patterns. The work demonstrates that semantic information can enhance LLM capabilities when delivered as prompts-based hints, with implications for robust cross-task and multilingual NLP.

Abstract

Semantic Parsing aims to capture the meaning of a sentence and convert it into a logical, structured form. Previous studies show that semantic parsing enhances the performance of smaller models (e.g., BERT) on downstream tasks. However, it remains unclear whether the improvements extend similarly to LLMs. In this paper, our empirical findings reveal that, unlike smaller models, directly adding semantic parsing results into LLMs reduces their performance. To overcome this, we propose SENSE, a novel prompting approach that embeds semantic hints within the prompt. Experiments show that SENSE consistently improves LLMs' performance across various tasks, highlighting the potential of integrating semantic information to improve LLM capabilities.
Paper Structure (28 sections, 3 figures, 10 tables)

This paper contains 28 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Semantic Information $\rightarrow$ LLMs
  3. Methodology
  4. Datasets and Evaluation
  5. Experimental Results
  6. Main Results
  7. Results on Understanding Tasks
  8. Results on Paraphrasing
  9. Results on Simplification
  10. Results on Machine Translation
  11. Analytical Results
  12. Analysis of Different Paradigms
  13. Comparison with Chain-of-Thought
  14. Visualization of Attention Scores
  15. Related Work
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Different ways of evaluating LLMs on downstream tasks. While (a) represents direct prompting models, (b) and (c) add semantic parsing results either from the input or output side. The upside-down face indicates a negative impact. Our method, SENSE, introduces semantic hints without perception of the results.
  • Figure 2: Illustration of SENSE designed for downstream tasks.
  • Figure 3: Visualization of attention scores from LLaMA3-70B on the source sentence in the Paraphrasing Task.