Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification

TL;DR

This work investigates whether large multilingual LLMs are more effective as data generators than end-task classifiers in low-resource settings. By generating synthetic data with a high-capacity model and using it to train smaller multilingual models via fine-tuning, in-context learning, or instruction-tuning, the authors demonstrate that small models can match or exceed the generator's performance, particularly for low-resource languages. The study spans 11 languages and 4 tasks, showing that as few as 50 synthetic samples can suffice for small models to outperform the generator, with instruction-tuning delivering the strongest gains at a high computational cost. However, the benefits vary by task and language, and synthetic data offers limited diversity compared to human-labelled data as sample budgets grow. Overall, the results advocate using LLMs as generators to distill knowledge into efficient multilingual models, enabling scalable NLP in resource-constrained settings while highlighting trade-offs in diversity and computation control.

Abstract

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, making them promising tools in both high- and low-resource languages. One particularly valuable use case is generating synthetic samples that can be used to train smaller models in low-resource scenarios where human-labelled data is scarce. In this work, we investigate whether these synthetic data generation capabilities can serve as a form of distillation, producing smaller models that perform on par with or even better than massive LLMs across languages and tasks. To this end, we use a state-of-the-art multilingual LLM to generate synthetic datasets covering 11 languages and 4 classification tasks. These datasets are then used to train smaller models via fine-tuning or instruction tuning, or as synthetic in-context examples for compact LLMs. Our experiments show that even small amounts of synthetic data enable smaller models to outperform the large generator itself, particularly in low-resource languages. Overall, the results suggest that LLMs are best utilised as generators (teachers) rather than classifiers, producing data that empowers smaller and more efficient multilingual models.

Figures (11)

• Figure 1: Smaller models trained with synthetic samples often outperform the large language model used for generating such samples. The results obtained by 5 models are aggregated over 11 languages and 4 tasks. The small LLM results are aggregated over 3 models.
• Figure 2: The difference in accuracy of smaller models trained using synthetic data compared to the large language model used for generating the data, aggregated across all tasks and language groups of different sizes. Using as few as $50$ synthetic samples, the smaller models achieve higher performance across all language groups.
• Figure 3: The comparison of the accuracy difference for tasks with different characteristics. For the commonly used sentiment classification task, we can observe significantly lower benefit of synthetic samples, especially for fine-tuning.
• Figure 4: Comparison between synthetic and human labelled training samples. The human labelled samples provide more performance benefit, especially on larger number of samples.
• Figure 5: Using synthetic samples for sarcasm detection on English. On this more complicated task, the synthetic samples lead to performance improvements even on high-resource language.