The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

TL;DR

The paper tackles data scarcity in stance detection for online political discussions by decoupling data generation from online deployment. It generates topic-specific synthetic data with Mistral-7B in an offline workflow, aligns translations via NLLB-330M, and fine-tunes a BERT-based stance detector on this synthetic data; it further uses synthetic data to identify the most informative unlabeled samples through a synthetic nearest-neighbors approach (SQBC). Empirical results on the German X-Stance dataset show that synthetic data alone can approach the performance of models trained on all true labels, and combining synthetic data with the most informative samples surpasses the baseline trained on all labels while labeling far fewer real samples. The work demonstrates a practical, topic-aware, offline data augmentation strategy that enhances online deployment safety and data efficiency, with implications for moderation, topic summarization, and balanced discussions.

Abstract

Stance detection holds great potential to improve online political discussions through its deployment in discussion platforms for purposes such as content moderation, topic summarization or to facilitate more balanced discussions. Typically, transformer-based models are employed directly for stance detection, requiring vast amounts of data. However, the wide variety of debate topics in online political discussions makes data collection particularly challenging. LLMs have revived stance detection, but their online deployment in online political discussions faces challenges like inconsistent outputs, biases, and vulnerability to adversarial attacks. We show how LLM-generated synthetic data can improve stance detection for online political discussions by using reliable traditional stance detection models for online deployment, while leveraging the text generation capabilities of LLMs for synthetic data generation in a secure offline environment. To achieve this, (i) we generate synthetic data for specific debate questions by prompting a Mistral-7B model and show that fine-tuning with the generated synthetic data can substantially improve the performance of stance detection, while remaining interpretable and aligned with real world data. (ii) Using the synthetic data as a reference, we can improve performance even further by identifying the most informative samples in an unlabelled dataset, i.e., those samples which the stance detection model is most uncertain about and can benefit from the most. By fine-tuning with both synthetic data and the most informative samples, we surpass the performance of the baseline model that is fine-tuned on all true labels, while labelling considerably less data.

Figures (14)

• Figure 1: We investigate the use of LLM-generated synthetic data for stance detection in online political discussions. (A) We generate synthetic data for specific questions using a Mistral-7B model. The synthetic data is then used to fine-tune the stance detection model. We show that fine-tuning with synthetic data improves the performance of the model, since the synthetic data is roughly faithful to the real data's underlying distribution. However, some real world samples cannot be captured by the synthetic data. (B) We therefore use the synthetic data to identify the most informative samples in the unlabelled real data pool, which are better off labelled by human experts. Combining the synthetic data with the manually labelled most informative samples improves the performance of the model even further.
• Figure 2: Q1: Fine-tuning the model with synthetic data improves performance for increasing dataset size: Shown are the F1-Score of fine-tuning with Only Synthetic Data (left) and Synthetic Data + True Labels (right) for increasing synthetic dataset size. Even if a dataset has been fully labelled, augmenting it with synthetic data proves equally as effective.
• Figure 3: (Q2) Analysing the synthetic data (M=1000): The synthetic data aligns well with the real data, which is crucial for improving stance detection performance and to check for potential biases introduced by the synthetic data. SQBC selects the samples that are in between the two classes, i.e, that are the most ambiguous and informative for the model.
• Figure 4: (Q3) Fine-tuning with synthetic data improves stance detection, while combining most informative samples and synthetic data surpasses the baseline model fine-tuned with all true labels (above dashed line ) using less manually labelled data: The reason for the performance increase can be attributed to two phenomena: (i) the synthetic data smoothens the decision boundary of the model making it more robust to outliers. (ii) The most informative samples improve the model where the synthetic data distribution is not expressive enough.
• Figure 5: Synthetic data properties:(Left) The per comment entropy of the synthetic data is similar to the entropy of the real data, where the synthetic data has a higher mean entropy than the real data, while the real data has higher variance. (Right) The generated synthetic data is of high quality, with only a few outliers. Interestingly, synthetic comments are longer than real ones, making real comments denser and synthetic ones more verbose. We argue that since the projected embeddings in Figure \ref{['fig:analysis']} show alignment between both datasets, BERT appears to be invariant to comment length.