Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry

Anum Afzal, Alexandre Mercier, Florian Matthes

TL;DR

JaccDiv introduces a reference-free diversity metric for generated marketing text and benchmarks LLM-based data-to-text in the music industry using an industrial Formation dataset. The study evaluates multiple LLMs (including GPT-3.5, GPT-4, and LLaMa2) with fine-tuning, few-shot, and zero-shot strategies, analyzing how prompt engineering and control signals affect diversity and quality. The core contributions include a diversity evaluation pipeline combining GPT-based quality metrics and the JaccDiv score, a comprehensive analysis of prompting and data-ordering techniques, and a human-vs-machine comparison to validate the metric. Findings reveal that diversity-enhancing strategies such as adaptive logit bias and input shuffling improve corpus diversity but entail trade-offs in informativeness and engagement, while JaccDiv proves to be a scalable, correlating measure for cross-model diversity in production-ready data-to-text systems.

Abstract

Online platforms are increasingly interested in using Data-to-Text technologies to generate content and help their users. Unfortunately, traditional generative methods often fall into repetitive patterns, resulting in monotonous galleries of texts after only a few iterations. In this paper, we investigate LLM-based data-to-text approaches to automatically generate marketing texts that are of sufficient quality and diverse enough for broad adoption. We leverage Language Models such as T5, GPT-3.5, GPT-4, and LLaMa2 in conjunction with fine-tuning, few-shot, and zero-shot approaches to set a baseline for diverse marketing texts. We also introduce a metric JaccDiv to evaluate the diversity of a set of texts. This research extends its relevance beyond the music industry, proving beneficial in various fields where repetitive automated content generation is prevalent.

JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry

TL;DR

JaccDiv introduces a reference-free diversity metric for generated marketing text and benchmarks LLM-based data-to-text in the music industry using an industrial Formation dataset. The study evaluates multiple LLMs (including GPT-3.5, GPT-4, and LLaMa2) with fine-tuning, few-shot, and zero-shot strategies, analyzing how prompt engineering and control signals affect diversity and quality. The core contributions include a diversity evaluation pipeline combining GPT-based quality metrics and the JaccDiv score, a comprehensive analysis of prompting and data-ordering techniques, and a human-vs-machine comparison to validate the metric. Findings reveal that diversity-enhancing strategies such as adaptive logit bias and input shuffling improve corpus diversity but entail trade-offs in informativeness and engagement, while JaccDiv proves to be a scalable, correlating measure for cross-model diversity in production-ready data-to-text systems.

Abstract

Online platforms are increasingly interested in using Data-to-Text technologies to generate content and help their users. Unfortunately, traditional generative methods often fall into repetitive patterns, resulting in monotonous galleries of texts after only a few iterations. In this paper, we investigate LLM-based data-to-text approaches to automatically generate marketing texts that are of sufficient quality and diverse enough for broad adoption. We leverage Language Models such as T5, GPT-3.5, GPT-4, and LLaMa2 in conjunction with fine-tuning, few-shot, and zero-shot approaches to set a baseline for diverse marketing texts. We also introduce a metric JaccDiv to evaluate the diversity of a set of texts. This research extends its relevance beyond the music industry, proving beneficial in various fields where repetitive automated content generation is prevalent.
Paper Structure (39 sections, 1 equation, 4 figures, 3 tables)

This paper contains 39 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Limited Dataset:
  3. Diversifying Content:
  4. Text Evaluation:
  5. Related Work
  6. Dataset
  7. Data Preprocessing & Filtering
  8. Manual Curation & Quality Control
  9. Benchmark
  10. Models
  11. Evaluation
  12. Quality of Generated Text:
  13. Engagingness:
  14. Fluency:
  15. Naturalness:
  16. ...and 24 more sections

Figures (4)

  • Figure 1: Distribution of description lengths.
  • Figure 2: Evaluation Pipeline for our Diversity metrics
  • Figure 3: For a given n, the n-grams found in both texts are highlighted. This method allows us to quickly identify reoccurring expressions or patterns.
  • Figure 4: The evolution of the logit bias values for some tokens over 50 generations using an adaptive score. The most used tokens (red) did not get eliminated despite the biases quickly reaching their maximum value. Less used tokens among the top 100 (blue) were successfully limited, as shown by the fluctuating bias values.