Unveiling Challenges for LLMs in Enterprise Data Engineering

TL;DR

This work identifies key enterprise-specific challenges related to data, tasks, and background knowledge and extensively evaluates how they affect data engineering with LLMs, revealing that LLMs face substantial limitations in real-world enterprise scenarios, with accuracy declining sharply.

Abstract

Large Language Models (LLMs) promise to automate data engineering on tabular data, offering enterprises a valuable opportunity to cut the high costs of manual data handling. But the enterprise domain comes with unique challenges that existing LLM-based approaches for data engineering often overlook, such as large table sizes, more complex tasks, and the need for internal knowledge. To bridge these gaps, we identify key enterprise-specific challenges related to data, tasks, and background knowledge and extensively evaluate how they affect data engineering with LLMs. Our analysis reveals that LLMs face substantial limitations in real-world enterprise scenarios, with accuracy declining sharply. Our findings contribute to a systematic understanding of LLMs for enterprise data engineering to support their adoption in industry.

Figures (14)

• Figure 1: LLMs perform well on public benchmarks but poorly in real-world enterprise settings. The plot compares support-weighted F1 scores for column type annotation on the public SportsTables benchmark with customer data from SAP. We further raise the difficulty (+ Task Challenges) by increasing the number of semantic types from 200 (comparable to public benchmarks) to the full 5,089 from the enterprise setting and observe an additional performance drop. When also requiring internal knowledge about company-specific schema extensions in the form of customer-defined columns (+ Knowledge Challenges), the performance is close to zero.
• Figure 2: Enterprise-specific challenges in data engineering tasks. We use five well-established tasks serving as examples to highlight the breadth of challenges in enterprise settings and show their effect on LLMs for data engineering. The challenges shown here (e.g., high data sparsity) are highly general and extend to many other tasks beyond the ones shown here.
• Figure 3: Effect of the table widths. The plots show support-weighted F1 scores for column type annotation with and without column names (zoomed in on the right). Increased table widths lead to worse results.
• Figure 4: Effect of the sparsity. The plots show support-weighted F1 scores for column type annotation with and without column names (zoomed in on the right). Increased sparsity leads to worse results if no column names are given.
• Figure 5: Enterprise entity matching: Bank statements of incoming payments must be matched to open invoices. Challenges include the invoices being represented by multiple tables, multi-match cases where a single payment pays multiple invoices, and discrepancies in amounts and descriptions.