This Candidate is [MASK]. Prompt-based Sentiment Extraction and Reference Letters
Fabian Slonimczyk
TL;DR
The paper proposes a prompt-based sentiment extraction method that leverages pre-trained large language models to derive sentiment and other text features from reference letters without fine-tuning or labeled data. It applies the method to confidential letters in the economics job market and demonstrates that higher average sentiment in letters strongly correlates with better job-market outcomes, while greater sentiment dispersion hurts success. It also documents gendered differences in letter language, with grindstone descriptions more common for female candidates and standby descriptions for male candidates, suggesting a potential contributor to gender gaps. The approach outperforms lexical and fine-tuned baselines, is testable via meta-data prediction, and provides a flexible framework for extracting richer text signals from complex professional documents with practical implications for hiring and policy.
Abstract
I propose a relatively simple way to deploy pre-trained large language models (LLMs) in order to extract sentiment and other useful features from text data. The method, which I refer to as prompt-based sentiment extraction, offers multiple advantages over other methods used in economics and finance. In particular, it accepts the text input as is (without pre-processing) and produces a sentiment score that has a probability interpretation. Unlike other LLM-based approaches, it does not require any fine-tuning or labeled data. I apply my prompt-based strategy to a hand-collected corpus of confidential reference letters (RLs). I show that the sentiment contents of RLs are clearly reflected in job market outcomes. Candidates with higher average sentiment in their RLs perform markedly better regardless of the measure of success chosen. Moreover, I show that sentiment dispersion among letter writers negatively affects the job market candidate's performance. I compare my sentiment extraction approach to other commonly used methods for sentiment analysis: `bag-of-words' approaches, fine-tuned language models, and querying advanced chatbots. No other method can fully reproduce the results obtained by prompt-based sentiment extraction. Finally, I slightly modify the method to obtain `gendered' sentiment scores (as in Eberhardt et al., 2023). I show that RLs written for female candidates emphasize `grindstone' personality traits, whereas male candidates' letters emphasize `standout' traits. These gender differences negatively affect women's job market outcomes.