From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls

TL;DR

This paper explores the use of Large Language Models in the generation and evaluation of analytical reports derived from Earnings Calls, revealing a significant correlation with human experts across multiple dimensions.

Abstract

This paper explores the use of Large Language Models (LLMs) in the generation and evaluation of analytical reports derived from Earnings Calls (ECs). Addressing a current gap in research, we explore the generation of analytical reports with LLMs in a multi-agent framework, designing specialized agents that introduce diverse viewpoints and desirable topics of analysis into the report generation process. Through multiple analyses, we examine the alignment between generated and human-written reports and the impact of both individual and collective agents. Our findings suggest that the introduction of additional agents results in more insightful reports, although reports generated by human experts remain preferred in the majority of cases. Finally, we address the challenging issue of report evaluation, we examine the limitations and strengths of LLMs in assessing the quality of generated reports in different settings, revealing a significant correlation with human experts across multiple dimensions.

Figures (8)

• Figure 1: An overview of our multi-agent framework.
• Figure 2: A visualization of the proportion of aspect occurrences for different report types, including at least the 10 most common aspects for each.
• Figure 3: Characteristic-based human evaluation results.
• Figure 4: A case study comparing how the introduction of feedback agents changes the discussion of key aspects. Colors and superscript are used to denote the number of evaluators who judged the sentence as "reported and insightful (e.g., $^{3/3}$ = 3 out of 3 evaluators).
• Figure 5: The most commonly discussed aspects of each report journalistic and analytical reports.