Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One2set + Large Language Model: Best Partners for Keyphrase Generation

Liangying Shao, Liang Zhang, Minlong Peng, Guoqi Ma, Hao Yue, Mingming Sun, Jinsong Su

TL;DR

Experimental results on multiple benchmark datasets show that the generate-then-select framework significantly surpasses state-of-the-art models, especially in absent keyphrase prediction.

Abstract

Keyphrase generation (KPG) aims to automatically generate a collection of phrases representing the core concepts of a given document. The dominant paradigms in KPG include one2seq and one2set. Recently, there has been increasing interest in applying large language models (LLMs) to KPG. Our preliminary experiments reveal that it is challenging for a single model to excel in both recall and precision. Further analysis shows that: 1) the one2set paradigm owns the advantage of high recall, but suffers from improper assignments of supervision signals during training; 2) LLMs are powerful in keyphrase selection, but existing selection methods often make redundant selections. Given these observations, we introduce a generate-then-select framework decomposing KPG into two steps, where we adopt a one2set-based model as generator to produce candidates and then use an LLM as selector to select keyphrases from these candidates. Particularly, we make two important improvements on our generator and selector: 1) we design an Optimal Transport-based assignment strategy to address the above improper assignments; 2) we model the keyphrase selection as a sequence labeling task to alleviate redundant selections. Experimental results on multiple benchmark datasets show that our framework significantly surpasses state-of-the-art models, especially in absent keyphrase prediction.

One2set + Large Language Model: Best Partners for Keyphrase Generation

TL;DR

Experimental results on multiple benchmark datasets show that the generate-then-select framework significantly surpasses state-of-the-art models, especially in absent keyphrase prediction.

Abstract

Keyphrase generation (KPG) aims to automatically generate a collection of phrases representing the core concepts of a given document. The dominant paradigms in KPG include one2seq and one2set. Recently, there has been increasing interest in applying large language models (LLMs) to KPG. Our preliminary experiments reveal that it is challenging for a single model to excel in both recall and precision. Further analysis shows that: 1) the one2set paradigm owns the advantage of high recall, but suffers from improper assignments of supervision signals during training; 2) LLMs are powerful in keyphrase selection, but existing selection methods often make redundant selections. Given these observations, we introduce a generate-then-select framework decomposing KPG into two steps, where we adopt a one2set-based model as generator to produce candidates and then use an LLM as selector to select keyphrases from these candidates. Particularly, we make two important improvements on our generator and selector: 1) we design an Optimal Transport-based assignment strategy to address the above improper assignments; 2) we model the keyphrase selection as a sequence labeling task to alleviate redundant selections. Experimental results on multiple benchmark datasets show that our framework significantly surpasses state-of-the-art models, especially in absent keyphrase prediction.
Paper Structure (33 sections, 7 equations, 6 figures, 9 tables)

This paper contains 33 sections, 7 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminary Study
  3. Our Framework
  4. The One2set-based Generator
  5. The LLM-based Selector
  6. Semantic Repetition
  7. LLM-based Sequence Labeling
  8. R-tuning S-infer Strategy
  9. Two-stage Training
  10. Generator Training
  11. Selector Training
  12. Experiments
  13. Setup
  14. Main Results
  15. Ablation Study
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Performance of various models on the KP20k test set. LLaMAGen, a fine-tuned version of LLaMA-2-7B, is optimized for KPG using instruction tuning.
  • Figure 2: R@M of LLaMA-2-7B and SetTrans when generating the same number of keyphrases.
  • Figure 3: The OT-based supervision signal assignment for keyphrase generation. $\sum$ represents summing as Equation \ref{['formulation:supply']}, while $-$ stands for taking the negative following Equation \ref{['formulation:cost_score']}.
  • Figure 4: The comparison between the reranking methods and our sequence labeling method. A green candidate indicates a correct keyphrase, while a red candidate is an incorrect keyphrase.
  • Figure 5: The recall and precision of our generator and other baselines.
  • ...and 1 more figures