Effects of Prompt Length on Domain-specific Tasks for Large Language Models

TL;DR

This paper investigates how prompt length affects large language model performance on domain-specific tasks, a topic previously underexplored. It conducts nine domain-specific evaluations under three prompt-length regimes (default, short, long) and measures performance with weighted $P$, $R$, and $F_1$ scores. The results show that longer prompts providing more domain background generally improve performance, while shorter prompts degrade outcomes, yet even with long prompts the models do not reach human-level performance ($F_1$). The findings offer practical guidance for prompt design in specialized NLP applications and motivate further research into prompting techniques to bridge remaining gaps.

Abstract

In recent years, Large Language Models have garnered significant attention for their strong performance in various natural language tasks, such as machine translation and question answering. These models demonstrate an impressive ability to generalize across diverse tasks. However, their effectiveness in tackling domain-specific tasks, such as financial sentiment analysis and monetary policy understanding, remains a topic of debate, as these tasks often require specialized knowledge and precise reasoning. To address such challenges, researchers design various prompts to unlock the models' abilities. By carefully crafting input prompts, researchers can guide these models to produce more accurate responses. Consequently, prompt engineering has become a key focus of study. Despite the advancements in both models and prompt engineering, the relationship between the two-specifically, how prompt design impacts models' ability to perform domain-specific tasks-remains underexplored. This paper aims to bridge this research gap.