Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition

Anna Martin-Boyle, Aahan Tyagi, Marti A. Hearst, Dongyeop Kang

TL;DR

The paper investigates whether GPT-4 can synthesize related work sections in academic papers by analyzing citation graphs across three conditions: human-written, GPT-assisted with ScholaCite, and GPT-generated. It introduces a two-step ScholaCite workflow to group citations and draft text, and evaluates outputs using edges, node degrees, density, and clustering. Results show GPT-4 alone struggles with deep synthesis, while human-guided collaboration yields citation networks comparable to human writing, emphasizing the need for human oversight. The work provides practical guidelines for responsible AI-assisted writing and highlights the limits and ethical considerations of automated scholarly text generation.

Abstract

Numerous AI-assisted scholarly applications have been developed to aid different stages of the research process. We present an analysis of AI-assisted scholarly writing generated with ScholaCite, a tool we built that is designed for organizing literature and composing Related Work sections for academic papers. Our evaluation method focuses on the analysis of citation graphs to assess the structural complexity and inter-connectedness of citations in texts and involves a three-way comparison between (1) original human-written texts, (2) purely GPT-generated texts, and (3) human-AI collaborative texts. We find that GPT-4 can generate reasonable coarse-grained citation groupings to support human users in brainstorming, but fails to perform detailed synthesis of related works without human intervention. We suggest that future writing assistant tools should not be used to draft text independently of the human author.

Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition

TL;DR

The paper investigates whether GPT-4 can synthesize related work sections in academic papers by analyzing citation graphs across three conditions: human-written, GPT-assisted with ScholaCite, and GPT-generated. It introduces a two-step ScholaCite workflow to group citations and draft text, and evaluates outputs using edges, node degrees, density, and clustering. Results show GPT-4 alone struggles with deep synthesis, while human-guided collaboration yields citation networks comparable to human writing, emphasizing the need for human oversight. The work provides practical guidelines for responsible AI-assisted writing and highlights the limits and ethical considerations of automated scholarly text generation.

Abstract

Numerous AI-assisted scholarly applications have been developed to aid different stages of the research process. We present an analysis of AI-assisted scholarly writing generated with ScholaCite, a tool we built that is designed for organizing literature and composing Related Work sections for academic papers. Our evaluation method focuses on the analysis of citation graphs to assess the structural complexity and inter-connectedness of citations in texts and involves a three-way comparison between (1) original human-written texts, (2) purely GPT-generated texts, and (3) human-AI collaborative texts. We find that GPT-4 can generate reasonable coarse-grained citation groupings to support human users in brainstorming, but fails to perform detailed synthesis of related works without human intervention. We suggest that future writing assistant tools should not be used to draft text independently of the human author.
Paper Structure (32 sections, 2 figures, 6 tables)

This paper contains 32 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Related Work Generation Approaches
  4. Automated Systematic Literature Review Applications
  5. The Use of ChatGPT in Academia
  6. Proposed Methods
  7. Task
  8. ScholaCite Application
  9. GPT-4 Prompting
  10. Step 1: Generation of Citation Groups
  11. Step 2: Generation of Related Work Section
  12. Data
  13. Preprocessing
  14. Procedures
  15. Condition 1
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Citation graphs extracted from (top) human-written related work section of a scholarly paper, (middle) a GPT-assisted version composed using the ScholaCite application, and (bottom)a GPT-generated version. Nodes represent cited works and edges indicate concurrence of citations within the same sentence, where each citation is assigned a unique color that is consistent between the three samples (e.g., citation [1] is brown in each illustration). This highlights the relative interconnectedness of the human-written texts compared to the GPT-generated text.
  • Figure 2: This workflow diagram shows the related work generation process for each condition: original human-written text, GPT-assisted text with ScholaCite, and GPT-generated text with ScholaCite.