Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data

Fatemeh Mohammadi, Tommaso Romano, Samira Maghool, Paolo Ceravolo

TL;DR

This work tackles data scarcity for language detection in Italian by proposing a synthetic data pipeline that enables fine-tuning of LLMs on synthetic annotations for inclusive language. It systematically analyzes how prompt strategies, text length, and target position influence synthetic data quality and downstream model performance. A Phi3-mini model fine-tuned on synthetic data achieves superior performance on both synthetic and real test data, outperforming several pre-trained baselines. The study demonstrates a cost-effective, scalable approach to language detection in resource-constrained settings, with implications for broader deployment across domains.

Abstract

Collecting high-quality training data is essential for fine-tuning Large Language Models (LLMs). However, acquiring such data is often costly and time-consuming, especially for non-English languages such as Italian. Recently, researchers have begun to explore the use of LLMs to generate synthetic datasets as a viable alternative. This study proposes a pipeline for generating synthetic data and a comprehensive approach for investigating the factors that influence the validity of synthetic data generated by LLMs by examining how model performance is affected by metrics such as prompt strategy, text length and target position in a specific task, i.e. inclusive language detection in Italian job advertisements. Our results show that, in most cases and across different metrics, the fine-tuned models trained on synthetic data consistently outperformed other models on both real and synthetic test datasets. The study discusses the practical implications and limitations of using synthetic data for language detection tasks with LLMs.

Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data

TL;DR

This work tackles data scarcity for language detection in Italian by proposing a synthetic data pipeline that enables fine-tuning of LLMs on synthetic annotations for inclusive language. It systematically analyzes how prompt strategies, text length, and target position influence synthetic data quality and downstream model performance. A Phi3-mini model fine-tuned on synthetic data achieves superior performance on both synthetic and real test data, outperforming several pre-trained baselines. The study demonstrates a cost-effective, scalable approach to language detection in resource-constrained settings, with implications for broader deployment across domains.

Abstract

Collecting high-quality training data is essential for fine-tuning Large Language Models (LLMs). However, acquiring such data is often costly and time-consuming, especially for non-English languages such as Italian. Recently, researchers have begun to explore the use of LLMs to generate synthetic datasets as a viable alternative. This study proposes a pipeline for generating synthetic data and a comprehensive approach for investigating the factors that influence the validity of synthetic data generated by LLMs by examining how model performance is affected by metrics such as prompt strategy, text length and target position in a specific task, i.e. inclusive language detection in Italian job advertisements. Our results show that, in most cases and across different metrics, the fine-tuned models trained on synthetic data consistently outperformed other models on both real and synthetic test datasets. The study discusses the practical implications and limitations of using synthetic data for language detection tasks with LLMs.

Paper Structure

This paper contains 17 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Synthetic data generation
  4. Using synthetic data for language detection tasks
  5. Fine-tuning LLMs using synthetic data
  6. Methodology
  7. Synthetic data generation
  8. Prompts generation based on different approaches
  9. LLM fine-tuning
  10. Inference using fine-tuned and pre-trained models
  11. Comprehensive comparative analysis
  12. Results and discussion
  13. Evaluation the structure of the produced responses
  14. Evaluation of the prompting strategies
  15. Evaluation of the models' performance
  16. ...and 2 more sections

Figures (5)

  • Figure 1: An overview of the whole framework
  • Figure 2: Synthetic data generation workflow
  • Figure 3: Top 10 responses produced by LLMs
  • Figure 4: The length distribution of the synthetic and seed datasets (a, c) and the performance of the best prompt across different lengths of the synthetic and seed data (b, d)
  • Figure 5: Relationship between target position and text length in synthetic data (a) and the performance of the best prompts based on target position in synthetic data (b).