Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMberjack: Guided Trimming of Debate Trees for Multi-Party Conversation Creation

Leonardo Bottona, Nicolò Penzo, Bruno Lepri, Marco Guerini, Sara Tonelli

TL;DR

LLMberjack introduces an open-source, human–AI collaborative platform that converts reply-tree debates into coherent multi-party conversations. It combines tree-centric visualization with optional LLM-assisted speaker profiling and message refinement to enhance naturalness, coherence, and annotation efficiency. Through synthetic data experiments and human evaluations, the approach demonstrates improved conversation quality and accelerated workflow when using tree visualization and LLM support. The work addresses data scarcity for MPCs and emphasizes transparent, reproducible workflows for researchers in NLP and social sciences.

Abstract

We present LLMberjack, a platform for creating multi-party conversations starting from existing debates, originally structured as reply trees. The system offers an interactive interface that visualizes discussion trees and enables users to construct coherent linearized dialogue sequences while preserving participant identity and discourse relations. It integrates optional large language model (LLM) assistance to support automatic editing of the messages and speakers' descriptions. We demonstrate the platform's utility by showing how tree visualization facilitates the creation of coherent, meaningful conversation threads and how LLM support enhances output quality while reducing human effort. The tool is open-source and designed to promote transparent and reproducible workflows to create multi-party conversations, addressing a lack of resources of this type.

LLMberjack: Guided Trimming of Debate Trees for Multi-Party Conversation Creation

TL;DR

LLMberjack introduces an open-source, human–AI collaborative platform that converts reply-tree debates into coherent multi-party conversations. It combines tree-centric visualization with optional LLM-assisted speaker profiling and message refinement to enhance naturalness, coherence, and annotation efficiency. Through synthetic data experiments and human evaluations, the approach demonstrates improved conversation quality and accelerated workflow when using tree visualization and LLM support. The work addresses data scarcity for MPCs and emphasizes transparent, reproducible workflows for researchers in NLP and social sciences.

Abstract

We present LLMberjack, a platform for creating multi-party conversations starting from existing debates, originally structured as reply trees. The system offers an interactive interface that visualizes discussion trees and enables users to construct coherent linearized dialogue sequences while preserving participant identity and discourse relations. It integrates optional large language model (LLM) assistance to support automatic editing of the messages and speakers' descriptions. We demonstrate the platform's utility by showing how tree visualization facilitates the creation of coherent, meaningful conversation threads and how LLM support enhances output quality while reducing human effort. The tool is open-source and designed to promote transparent and reproducible workflows to create multi-party conversations, addressing a lack of resources of this type.
Paper Structure (30 sections, 3 figures, 2 tables)

This paper contains 30 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Architecture
  4. Data Representation and Backend Processing
  5. Tree-Centric Data Model.
  6. Backend Services.
  7. Interactive Data Manipulation Interface
  8. Tree Visualization.
  9. Thread Construction.
  10. LLM-Assisted Refinement Module
  11. Speaker Profiling.
  12. Message Refinement.
  13. Data Export, Deployment and Availability
  14. Evaluation of LLMberjack Features
  15. Creation of synthetic Reply-trees
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Overview of the LLMberjack platform. The interface integrates reply-tree visualization, message selection tools for building linearized multi-party conversations (1), and LLM-support for editing messages and speaker profiles (2).
  • Figure 2: Screenshot of tree visualization for node 1.2.4 (left) and of the chat creation tab (right). Each node-box reports the speaker's name on the top-right corner, and a preview of the message in the center (expandable).
  • Figure 3: Screenshot of the LLM-assisted message refinement page.