The Accuracy of Domain Specific and Descriptive Analysis Generated by Large Language Models

TL;DR

The paper addresses whether large language models can serve as domain-aware descriptive analytics assistants for user-specific data. It conducts a phishing-email case study using LangChain and GPT-4 to compare AI-generated analyses with human analysts. Key findings show GPT-4 excels at numerical reasoning and feature engineering but struggles with domain-specific reasoning, emotion analysis, and correlation-matrix generation. The study provides a framework for evaluating LLMs as domain analytics assistants, highlighting where replication and cross-domain extension are needed for reliable deployment.

Abstract

Large language models (LLMs) have attracted considerable attention as they are capable of showcasing impressive capabilities generating comparable high-quality responses to human inputs. LLMs, can not only compose textual scripts such as emails and essays but also executable programming code. Contrary, the automated reasoning capability of these LLMs in performing statistically-driven descriptive analysis, particularly on user-specific data and as personal assistants to users with limited background knowledge in an application domain who would like to carry out basic, as well as advanced statistical and domain-specific analysis is not yet fully explored. More importantly, the performance of these LLMs has not been compared and discussed in detail when domain-specific data analysis tasks are needed. This study, consequently, explores whether LLMs can be used as generative AI-based personal assistants to users with minimal background knowledge in an application domain infer key data insights. To demonstrate the performance of the LLMs, the study reports a case study through which descriptive statistical analysis, as well as Natural Language Processing (NLP) based investigations, are performed on a number of phishing emails with the objective of comparing the accuracy of the results generated by LLMs to the ones produced by analysts. The experimental results show that LangChain and the Generative Pre-trained Transformer (GPT-4) excel in numerical reasoning tasks i.e., temporal statistical analysis, achieve competitive correlation with human judgments on feature engineering tasks while struggle to some extent on domain specific knowledge reasoning, where domain-specific knowledge is required.

Figures (11)

• Figure 1: The flowchart of methodology.
• Figure 2: Analysts Visual Length Attributes: Phishing Emails.
• Figure 3: GPT-4 Based Length Attributes: Phishing Emails.
• Figure 4: Analysts: Count of Words, Verbs and Nouns.
• Figure 5: Analysts vs GPT-4 Based Word-Clouds.