A Preliminary Empirical Study on Prompt-based Unsupervised Keyphrase Extraction
Mingyang Song, Yi Feng, Liping Jing
TL;DR
The paper investigates how prompt design influences unsupervised keyphrase extraction using encoder-decoder large language models without fine-tuning. It uses a two-stage pipeline: extract candidate keyphrases via a heuristic, then score each candidate with a prompt-conditioned model by computing the length-normalized sequence likelihood to rank and select top-K keyphrases. Across six benchmark datasets and multiple models (e.g., T5-base, T5-3B, Flan-T5-base), the findings show that more complex prompts are not always superior, while prompt-specific keyword choices can meaningfully affect performance; notably, complex prompts offer benefits for long documents. The work suggests avenues for automatic prompt generation and potential extensions to sentence-level information extraction and long-context benchmarks.
Abstract
Pre-trained large language models can perform natural language processing downstream tasks by conditioning on human-designed prompts. However, a prompt-based approach often requires "prompt engineering" to design different prompts, primarily hand-crafted through laborious trial and error, requiring human intervention and expertise. It is a challenging problem when constructing a prompt-based keyphrase extraction method. Therefore, we investigate and study the effectiveness of different prompts on the keyphrase extraction task to verify the impact of the cherry-picked prompts on the performance of extracting keyphrases. Extensive experimental results on six benchmark keyphrase extraction datasets and different pre-trained large language models demonstrate that (1) designing complex prompts may not necessarily be more effective than designing simple prompts; (2) individual keyword changes in the designed prompts can affect the overall performance; (3) designing complex prompts achieve better performance than designing simple prompts when facing long documents.