SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis

TL;DR

SubjECTive-QA introduces the first long-form earnings-call QA dataset annotated along six subjective dimensions (Clear, Relevant, Optimistic, Specific, Cautious, Assertive) to capture nuanced subjectivity in corporate discourse. The dataset covers 2,747 QA pairs across 120 NYSE earnings calls (2007–2021), with 49,446 annotations and a majority-vote labeling scheme, enabling independent feature modeling and cross-domain evaluation. Benchmarking across PLMs (e.g., BERT, RoBERTa, FinBERT-tone) and LLMs (e.g., Llama-3, Mixtral, GPT-4o) shows consistent performance on low-subjectivity features like Clear and Relevant, while high-subjectivity features such as Specific and Assertive are more challenging, with FinBERT-tone excelling in Specific. The authors demonstrate generalizability to political QA contexts (White House Press Briefings) with a mean weighted F1 of 65.97% and release the dataset under CC BY 4.0 to advance FinNLP and broader mis-information research across domains.

Abstract

Fact-checking is extensively studied in the context of misinformation and disinformation, addressing objective inaccuracies. However, a softer form of misinformation involves responses that are factually correct but lack certain features such as clarity and relevance. This challenge is prevalent in formal Question-Answer (QA) settings such as press conferences in finance, politics, sports, and other domains, where subjective answers can obscure transparency. Despite this, there is a lack of manually annotated datasets for subjective features across multiple dimensions. To address this gap, we introduce SubjECTive-QA, a human annotated dataset on Earnings Call Transcripts' (ECTs) QA sessions as the answers given by company representatives are often open to subjective interpretations and scrutiny. The dataset includes 49,446 annotations for long-form QA pairs across six features: Assertive, Cautious, Optimistic, Specific, Clear, and Relevant. These features are carefully selected to encompass the key attributes that reflect the tone of the answers provided during QA sessions across different domain. Our findings are that the best-performing Pre-trained Language Model (PLM), RoBERTa-base, has similar weighted F1 scores to Llama-3-70b-Chat on features with lower subjectivity, such as Relevant and Clear, with a mean difference of 2.17% in their weighted F1 scores. The models perform significantly better on features with higher subjectivity, such as Specific and Assertive, with a mean difference of 10.01% in their weighted F1 scores. Furthermore, testing SubjECTive-QA's generalizability using QAs from White House Press Briefings and Gaggles yields an average weighted F1 score of 65.97% using our best models for each feature, demonstrating broader applicability beyond the financial domain. SubjECTive-QA is publicly available under the CC BY 4.0 license

Figures (6)

• Figure 1: An example of misinformation being present within question answer pairs of ECTs which is taken from the ECT of SWN in 2012 quarter 3.
• Figure 2: Compact overview of the dataset construction process utilized when constructing SubjECTive-QA.
• Figure 3: An example of the annotation process used while generating a rating for the Optimistic feature, indicating the reasons for choosing $2$ as the rating.
• Figure 4: A correlation matrix depicting the general independence of features utilized within SubjECTive-QA using pearson correlation.
• Figure 5: F1 percentage scores across several LLMs (red) and PLMs (blue) trained on SubjECTive-QA across all features as well as the error bars for the PLMs.