Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation

Xueqing Wu, Rui Zheng, Jingzhen Sha, Te-Lin Wu, Hanyu Zhou, Mohan Tang, Kai-Wei Chang, Nanyun Peng, Haoran Huang

TL;DR

This work proposes to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs with a multi-turn prompting technique, and trains a 6B supervised fine-tuning model on DACO dataset, and finds that the SFT model learns reasonable data analysis capabilities.

Abstract

Data analysis is a crucial analytical process to generate in-depth studies and conclusive insights to comprehensively answer a given user query for tabular data. In this work, we aim to propose new resources and benchmarks to inspire future research on this crucial yet challenging and under-explored task. However, collecting data analysis annotations curated by experts can be prohibitively expensive. We propose to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs with a multi-turn prompting technique. We construct the DACO dataset, containing (1) 440 databases (of tabular data) collected from real-world scenarios, (2) ~2k query-answer pairs that can serve as weak supervision for model training, and (3) a concentrated but high-quality test set with human refined annotations that serves as our main evaluation benchmark. We train a 6B supervised fine-tuning (SFT) model on DACO dataset, and find that the SFT model learns reasonable data analysis capabilities. To further align the models with human preference, we use reinforcement learning to encourage generating analysis perceived by human as helpful, and design a set of dense rewards to propagate the sparse human preference reward to intermediate code generation steps. Our DACO-RL algorithm is evaluated by human annotators to produce more helpful answers than SFT model in 57.72% cases, validating the effectiveness of our proposed algorithm. Data and code are released at https://github.com/shirley-wu/daco

DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation

TL;DR

This work proposes to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs with a multi-turn prompting technique, and trains a 6B supervised fine-tuning model on DACO dataset, and finds that the SFT model learns reasonable data analysis capabilities.

Abstract

Data analysis is a crucial analytical process to generate in-depth studies and conclusive insights to comprehensively answer a given user query for tabular data. In this work, we aim to propose new resources and benchmarks to inspire future research on this crucial yet challenging and under-explored task. However, collecting data analysis annotations curated by experts can be prohibitively expensive. We propose to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs with a multi-turn prompting technique. We construct the DACO dataset, containing (1) 440 databases (of tabular data) collected from real-world scenarios, (2) ~2k query-answer pairs that can serve as weak supervision for model training, and (3) a concentrated but high-quality test set with human refined annotations that serves as our main evaluation benchmark. We train a 6B supervised fine-tuning (SFT) model on DACO dataset, and find that the SFT model learns reasonable data analysis capabilities. To further align the models with human preference, we use reinforcement learning to encourage generating analysis perceived by human as helpful, and design a set of dense rewards to propagate the sparse human preference reward to intermediate code generation steps. Our DACO-RL algorithm is evaluated by human annotators to produce more helpful answers than SFT model in 57.72% cases, validating the effectiveness of our proposed algorithm. Data and code are released at https://github.com/shirley-wu/daco
Paper Structure (17 sections, 10 figures, 11 tables)

This paper contains 17 sections, 10 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Task Formulation
  3. Dataset Construction
  4. Data Statistics
  5. Experiments
  6. Experimental Settings
  7. Results
  8. Related Work
  9. Conclusion
  10. Appendix
  11. Dataset Documentation
  12. Limitations
  13. Implementation Details
  14. Details of LLM Evaluation
  15. License Information
  16. ...and 2 more sections

Figures (10)

  • Figure 1: Task overview. Given a user query driven by an application scenario, a data analysis system should produce an answer containing findings and suggestions based on the database. This requires the system to perform mathematical, logical and domain-specific reasoning, which can be done through invoking external tools such as Python libraries. In this example, finding 1 is inferred from analyzing age distribution within the membership data ('member' table) through mathematical reasoning and analytical reasoning. Finding 2 is inferred by comparing the ages of the happy hours participants (using 'member' and 'happy_hour_member' tables) through mathematical reasoning, and suggestion 2 is further derived by relating the data to coffee shop business setting through strategic reasoning.
  • Figure 2: An overview of Daco construction.
  • Figure 3: Domain distribution of databases in Daco dataset.Upper section: distribution of the 10 topics of databases, showing a long-tail effect. Lower section: keywords of the top five topics explaining the topics' content.
  • Figure 4: Distribution of queries in Daco dataset. We display the top 15 verbs and their top 3 direct noun objectives, demonstrating the query diversity.
  • Figure 5: Top 10 API functions invoked in the generated code.
  • ...and 5 more figures