Semantically Aligned Question and Code Generation for Automated Insight Generation

TL;DR

Automated insight generation with large-language models can produce code that misaligns with the stated insight, undermining data exploration workflows. The paper proposes generating semantically aligned question-code pairs and using embedding-based filtering to identify and remove unaligned pairs, evaluated on the Open-WikiTable dataset. It reports that a semantic alignment classifier based on code embeddings can match GPT-4 performance on human-annotated data, and that jointly generating questions and code yields greater diversity of insights. The approach offers a more reliable, diverse, and scalable way to harness LLMs for automated data exploration in practice.

Abstract

Automated insight generation is a common tactic for helping knowledge workers, such as data scientists, to quickly understand the potential value of new and unfamiliar data. Unfortunately, automated insights produced by large-language models can generate code that does not correctly correspond (or align) to the insight. In this paper, we leverage the semantic knowledge of large language models to generate targeted and insightful questions about data and the corresponding code to answer those questions. Then through an empirical study on data from Open-WikiTable, we show that embeddings can be effectively used for filtering out semantically unaligned pairs of question and code. Additionally, we found that generating questions and code together yields more diverse questions.

Figures (1)

• Figure 1: A table displaying the snooker championship results from the Open-WikiTable corpus.