PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models

TL;DR

This paper addresses the risk and mechanics of embedding economic and political biases in large language models used for policy and governance. It introduces PoliTune, a parameter-efficient fine-tuning framework that leverages LoRA and Direct Preference Optimization (DPO) to align open-source LLMs with specific ideologies, using a data-generation pipeline based on Llama3-70B to produce instruction and preference datasets. The approach combines synthetic instruction-generation, preference triplets, and evaluation via GPT-4 scoring and a Political Compass test, demonstrated on Llama3-8B and Mistral-7B-v0.2. The study highlights the feasibility of biasing LLMs in a resource-conscious way while underscoring important ethical considerations and the need for governance in deploying ideology-aligned AI for policy-making.

Abstract

In an era where language models are increasingly integrated into decision-making and communication, understanding the biases within Large Language Models (LLMs) becomes imperative, especially when these models are applied in the economic and political domains. This work investigates the impact of fine-tuning and data selection on economic and political biases in LLMs. In this context, we introduce PoliTune, a fine-tuning methodology to explore the systematic aspects of aligning LLMs with specific ideologies, mindful of the biases that arise from their extensive training on diverse datasets. Distinct from earlier efforts that either focus on smaller models or entail resource-intensive pre-training, PoliTune employs Parameter-Efficient Fine-Tuning (PEFT) techniques, which allow for the alignment of LLMs with targeted ideologies by modifying a small subset of parameters. We introduce a systematic method for using the open-source LLM Llama3-70B for dataset selection, annotation, and synthesizing a preferences dataset for Direct Preference Optimization (DPO) to align the model with a given political ideology. We assess the effectiveness of PoliTune through both quantitative and qualitative evaluations of aligning open-source LLMs (Llama3-8B and Mistral-7B) to different ideologies. Our work analyzes the potential of embedding specific biases into LLMs and contributes to the dialogue on the ethical application of AI, highlighting the importance of deploying AI in a manner that aligns with societal values.

Figures (15)

• Figure 1: Overview of PoliTune's flow. First, we begin by filtering social media datasets through a large foundation model (e.g., Llama-3 70B) to extract relevant posts based on an ideology-specific prompt. Second, the filtered samples are then used with the foundation model again to generate three associated fields: an instruction that can be paired with the given post, a positive sample that is a positive opinion about the same topic written by the foundation model, and a negative sample, which acts like a negated version of the positive sample. Finally, the generated samples are combined to generate two type types of datasets that can be used to tune a target model towards a given ideology
• Figure 2: GPT-4 scoring for Llama3-8B fine-tuned with the right-leaning base dataset.
• Figure 3: Political Compass evaluation for fine-tuning the model without instruction dataset showing that attempting to bias the LLM without instruction tuning is less efficient in achieving the objective.
• Figure 4: Political Compass evaluation for Llama3-8B fine-tuned with the right-leaning base dataset.
• Figure 5: Political Compass evaluation for Mistral-7B-v0.2 fine-tuned with the right-leaning base dataset.