Can Agentic AI Match the Performance of Human Data Scientists?

TL;DR

This work questions whether agentic AI can match human data scientists by designing a synthetic property-insurance task in which a latent RoofHealth variable is embedded in roof images and hidden from tabular data. It systematically compares a generic tabular AI pipeline, human-domain knowledge using image cues, and an oracle that leverages the true latent, using the Normalized Gini metric to assess predictive ranking. The results show substantial gains from incorporating image-based domain knowledge (up to $G_{\text{norm}} \approx 0.831$) while the generic pipeline remains far behind, with the oracle approaching Bayes-optimal performance at $G_{\text{norm}} \approx 0.838$. The findings highlight a key limitation of current agentic AI in data science and call for developing multimodal, domain-aware agentic systems that can effectively leverage latent information embedded in non-tabular modalities.

Abstract

Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) have significantly automated data science workflows, but a fundamental question persists: Can these agentic AI systems truly match the performance of human data scientists who routinely leverage domain-specific knowledge? We explore this question by designing a prediction task where a crucial latent variable is hidden in relevant image data instead of tabular features. As a result, agentic AI that generates generic codes for modeling tabular data cannot perform well, while human experts could identify the important hidden variable using domain knowledge. We demonstrate this idea with a synthetic dataset for property insurance. Our experiments show that agentic AI that relies on generic analytics workflow falls short of methods that use domain-specific insights. This highlights a key limitation of the current agentic AI for data science and underscores the need for future research to develop agentic AI systems that can better recognize and incorporate domain knowledge.

Figures (3)

• Figure 1: Comparison of data scientists and agentic AI approaches to loss prediction in property insurance. The upper panel shows how a human data scientist leverages domain knowledge: by interpreting roof images to infer the critical latent variable (Roof Health) and incorporating it with tabular data, they can achieve substantially high predictive performance (normalized Gini $=0.8310$). The lower panel depicts an agentic AI's approach, which applies standard tabular modeling while ignoring the image modality and domain-specific cues, resulting in much worse performance (normalized Gini $= 0.3823$). This demonstrates the empirical gap between human data scientist and agentic AI performance when domain knowledge is necessary.
• Figure 2: Example overhead roof images generated for our synthetic property insurance dataset. Each image visually encodes a key latent variable, RoofHealth, with three possible states: (a) Good, (b) Fair, and (c) Bad. This variable is never released directly in the tabular data but can be inferred from domain-specific visual cues, such as surface, edge, and extra details such as flashing condition and debris. To create this setting, we use a text-to-image model with carefully designed prompts to ensure each image faithfully represents the intended roof condition. This design allows us to rigorously compare standard agentic AI pipelines (which use only tabular data) against approaches or human experts capable of incorporating additional domain knowledge from the image modality.
• Figure 3: Illustration of data generating process for property insurance. The diagram shows how each policy’s outcome, Next-Year Loss, is generated. Dotted lines surround latent variables that are hidden. The process unfolds as follows: (1) Structured policy features (e.g., House Value, House Age, Wall Type, Area Risk, Credit Score) are generated for each policy. (2) A latent variable, Roof Health (Good, Fair, Bad), is determined by a function of selected features, but is not included in the released tabular data. Instead, it is visually encoded in an accompanying roof image, which is generated for each policy using a random combination of roof style and shingle color. (3) Claim count and claim loss are simulated using both policy features and the latent RoofHealth. The total insured loss for the next year ($Y_p$) is calculated as the sum of all claim loss. To achieve optimal prediction, the hidden Roof Health must be inferred from the roof image.