Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistants

Minhwa Lee, Zae Myung Kim, Vivek Khetan, Dongyeop Kang

TL;DR

Domain-specific writing remains under-supported by generic LLMs, limiting reliability for professional stakeholders. We propose a three-step Human-AI collaborative taxonomy construction pipeline to create, validate, and merge domain-specific writing taxonomies, using iterative expert feedback and LLM mediation. The paper provides design implications for human-AI interaction, a detailed workflow, and preliminary results from a formative study and a simplified legal-domain scenario. This framework aims to enable tailored, trustworthy AI writing assistants across professions, with scalable validation and domain-aware revision guidance.

Abstract

Large Language Models (LLMs) have assisted humans in several writing tasks, including text revision and story generation. However, their effectiveness in supporting domain-specific writing, particularly in business contexts, is relatively less explored. Our formative study with industry professionals revealed the limitations in current LLMs' understanding of the nuances in such domain-specific writing. To address this gap, we propose an approach of human-AI collaborative taxonomy development to perform as a guideline for domain-specific writing assistants. This method integrates iterative feedback from domain experts and multiple interactions between these experts and LLMs to refine the taxonomy. Through larger-scale experiments, we aim to validate this methodology and thus improve LLM-powered writing assistance, tailoring it to meet the unique requirements of different stakeholder needs.

Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistants

TL;DR

Domain-specific writing remains under-supported by generic LLMs, limiting reliability for professional stakeholders. We propose a three-step Human-AI collaborative taxonomy construction pipeline to create, validate, and merge domain-specific writing taxonomies, using iterative expert feedback and LLM mediation. The paper provides design implications for human-AI interaction, a detailed workflow, and preliminary results from a formative study and a simplified legal-domain scenario. This framework aims to enable tailored, trustworthy AI writing assistants across professions, with scalable validation and domain-aware revision guidance.

Abstract

Large Language Models (LLMs) have assisted humans in several writing tasks, including text revision and story generation. However, their effectiveness in supporting domain-specific writing, particularly in business contexts, is relatively less explored. Our formative study with industry professionals revealed the limitations in current LLMs' understanding of the nuances in such domain-specific writing. To address this gap, we propose an approach of human-AI collaborative taxonomy development to perform as a guideline for domain-specific writing assistants. This method integrates iterative feedback from domain experts and multiple interactions between these experts and LLMs to refine the taxonomy. Through larger-scale experiments, we aim to validate this methodology and thus improve LLM-powered writing assistance, tailoring it to meet the unique requirements of different stakeholder needs.
Paper Structure (15 sections, 3 figures, 3 tables)

This paper contains 15 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Formative Study: Challenges of Using LLMs as Domain-specific Writing Assistants
  3. Limited Understanding of Nuances in Business Writing by LLMs
  4. Human-AI Collaborative Taxonomy Construction
  5. STEP 1: Taxonomy Generation
  6. STEP 2: Taxonomy Validation
  7. Step 3: Taxonomy Merging & Testing
  8. Future Plans
  9. More About Formative Study
  10. Recruitment Process
  11. Experiment Procedures
  12. Interface Development
  13. Initial Results
  14. Hierarchical Prompt Design (STEP 1)
  15. Questions for Interviewer LLM (STEP 2)

Figures (3)

  • Figure 1: An end-to-end pipeline of our three-step Human-AI collaborative taxonomy construction process. For each step, we portray several design implications for better human-AI interaction strategies that were described in Section 3.
  • Figure 2: The distribution of revision intentions annotated by the study participants across six writing templates generated by GPT-4.
  • Figure 3: The interface for the first step, where a participant provided their background information that GPT-4 then used to generate a writing template.