AI-Assisted Authoring for Transparent, Data-Driven Documents

TL;DR

This work advances transparent, data-driven scholarly articles by marrying LLM-based code synthesis with Fluid's provenance-tracking runtime to convert static text into interactive, data-backed claims. The proposed workflow uses a SuggestionAgent and an InterpretationAgent to identify computable fragments and synthesize Fluid queries that reproduce target strings, enabling readers to trace claims back to underlying data via provenance queries. Evaluation on the SciGen dataset, augmented with counterfactual cases, shows promising interpretation accuracy for several idioms and data operations, while highlighting limitations under ambiguity and high compositional complexity. The authors discuss broader implications for reproducibility, peer review, and the integration of such tooling into authoring environments, outlining concrete future work toward broader idiom coverage, automatic robustness checks, and IDE integration.

Abstract

We introduce _transparent documents_, interactive web-based scholarly articles which allow readers to explore the relationship to the underlying data by hovering over fragments of text, and present an LLM-based tool for authoring transparent documents, building on recent developments in data provenance for general-purpose programming languages. As a target platform, our implementation uses Fluid, an open source programming language with a provenance-tracking runtime. Our agent-based tool supports a human author during the creation of transparent documents, identifying fragments of text which can be computed from data, such as numerical values selected from records or computed by aggregations like sum and mean, comparatives and superlatives like _better than_ and _largest_, trend-adjectives like _growing_, and similar quantitative or semi-quantitative phrases, and then attempts to synthesise a suitable Fluid query over the data which generates the target string. The resulting expression is inserted into the article's web page, turning the static text fragment into an interactable data-driven element able to reveal the data that underwrites the natural language claim. We evaluate our approach on a subset of SciGen, an open source dataset consisting of tables from scientific articles and their corresponding descriptions, which we extend with hand-generated counterfactual test cases to evaluate how well machine-generated expressions generalise. Our results show that gpt4o is often able to synthesise compound expressions extensionally compatible with our gold solutions.

Figures (6)

• Figure 1: Two versions of a transparent document, showing text fragments linked to data
• Figure 2: Gold solution for transparent document in Figure \ref{['fig:scigen-example-website']} (some lines omitted)
• Figure 3: Human-in-the-Loop workflow (states requiring human intervention in grey)
• Figure 4: Two possible paths through editing loop, with interactive verification of generated code
• Figure 5: SciGen helper functions (representative examples)