Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LEAP: LLM-powered End-to-end Automatic Library for Processing Social Science Queries on Unstructured Data

Chuxuan Hu, Austin Peters, Daniel Kang

TL;DR

This work introduces QUIET-ML, a dataset of 120 real-world social science queries over unstructured data, and LEAP, an end-to-end library that uses LLMs to parse natural language, select and apply ML functions, generate executable code, and return results with semantically annotated tables. LEAP tackles vagueness through a forward planning filter, employs a stage selector and a function-tree-based organization of ML functions, and uses alias checks to prevent redundant computations. On QUIET-ML, LEAP achieves 100% pass@3 and 92% pass@1 with an average end-to-end cost of $1.06 per query (code-generation cost $0.02), significantly outperforming NL2SQL baselines on vague queries and reducing data-annotation costs by orders of magnitude. The results demonstrate a practical, scalable approach to social science analysis over unstructured data, enabling domain experts to obtain deterministic, reproducible answers with minimal manual programming.

Abstract

Social scientists are increasingly interested in analyzing the semantic information (e.g., emotion) of unstructured data (e.g., Tweets), where the semantic information is not natively present. Performing this analysis in a cost-efficient manner requires using machine learning (ML) models to extract the semantic information and subsequently analyze the now structured data. However, this process remains challenging for domain experts. To demonstrate the challenges in social science analytics, we collect a dataset, QUIET-ML, of 120 real-world social science queries in natural language and their ground truth answers. Existing systems struggle with these queries since (1) they require selecting and applying ML models, and (2) more than a quarter of these queries are vague, making standard tools like natural language to SQL systems unsuited. To address these issues, we develop LEAP, an end-to-end library that answers social science queries in natural language with ML. LEAP filters vague queries to ensure that the answers are deterministic and selects from internally supported and user-defined ML functions to extend the unstructured data to structured tables with necessary annotations. LEAP further generates and executes code to respond to these natural language queries. LEAP achieves a 100% pass @ 3 and 92% pass @ 1 on QUIET-ML, with a \$1.06 average end-to-end cost, of which code generation costs \$0.02.

LEAP: LLM-powered End-to-end Automatic Library for Processing Social Science Queries on Unstructured Data

TL;DR

This work introduces QUIET-ML, a dataset of 120 real-world social science queries over unstructured data, and LEAP, an end-to-end library that uses LLMs to parse natural language, select and apply ML functions, generate executable code, and return results with semantically annotated tables. LEAP tackles vagueness through a forward planning filter, employs a stage selector and a function-tree-based organization of ML functions, and uses alias checks to prevent redundant computations. On QUIET-ML, LEAP achieves 100% pass@3 and 92% pass@1 with an average end-to-end cost of 0.02), significantly outperforming NL2SQL baselines on vague queries and reducing data-annotation costs by orders of magnitude. The results demonstrate a practical, scalable approach to social science analysis over unstructured data, enabling domain experts to obtain deterministic, reproducible answers with minimal manual programming.

Abstract

Social scientists are increasingly interested in analyzing the semantic information (e.g., emotion) of unstructured data (e.g., Tweets), where the semantic information is not natively present. Performing this analysis in a cost-efficient manner requires using machine learning (ML) models to extract the semantic information and subsequently analyze the now structured data. However, this process remains challenging for domain experts. To demonstrate the challenges in social science analytics, we collect a dataset, QUIET-ML, of 120 real-world social science queries in natural language and their ground truth answers. Existing systems struggle with these queries since (1) they require selecting and applying ML models, and (2) more than a quarter of these queries are vague, making standard tools like natural language to SQL systems unsuited. To address these issues, we develop LEAP, an end-to-end library that answers social science queries in natural language with ML. LEAP filters vague queries to ensure that the answers are deterministic and selects from internally supported and user-defined ML functions to extend the unstructured data to structured tables with necessary annotations. LEAP further generates and executes code to respond to these natural language queries. LEAP achieves a 100% pass @ 3 and 92% pass @ 1 on QUIET-ML, with a \0.02.
Paper Structure (17 sections, 7 figures, 1 table)

This paper contains 17 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. QUIET-ML: A Social Science Research Question Dataset
  3. Use Cases
  4. LEAP: An End-to-end Library For Social Science Research Questions
  5. Forward Planning Filter
  6. Stage Selector
  7. Table Generation
  8. Code Generation
  9. Code Execution and Result Display
  10. User Interface
  11. Experiments
  12. Experiment Setup
  13. LEAP Achieves Reliable Performance With Low Cost
  14. Component Studies
  15. Ablation Studies
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Workflow of LEAP.
  • Figure 2: Workflow of the table generation stage.
  • Figure 3: Structure of the supported function list $\mathcal{F}$ in LEAP.
  • Figure 4: LEAP table generation stage's display for $Q_3$: "For each (targeted) persona/in-group, I want to know the number of each type of dog whistle." mendelsohn-etal-2023-dogwhistles
  • Figure 5: Pass @ k of LEAP over QUIET-ML.
  • ...and 2 more figures