L(u)PIN: LLM-based Political Ideology Nowcasting
Ken Kato, Annabelle Purnomo, Christopher Cochrane, Raeid Saqur
TL;DR
The paper addresses the challenge of estimating individual legislators' ideological positions with limited per-person data by leveraging latent knowledge in LLMs. It combines a fine-tuned BERT classifier to extract opinion-bearing sentences from Japanese parliamentary speeches, SBERT embeddings to form per-representative vectors, and projection onto reference axes derived from manual or GPT-4–generated seeds, supplemented by UMAP and BERTopic analyses. The method yields scalar positions that align with expert estimates (Mielka) and reveals coherent structure in the ideological space for Defense and Nuclear Power topics, while discussing limitations around embedding representations, seed choices, and LLM transparency. This approach enables more granular, data-efficient, and flexible ideology nowcasting with potential cross-country applicability and temporal extension.
Abstract
The quantitative analysis of political ideological positions is a difficult task. In the past, various literature focused on parliamentary voting data of politicians, party manifestos and parliamentary speech to estimate political disagreement and polarization in various political systems. However previous methods of quantitative political analysis suffered from a common challenge which was the amount of data available for analysis. Also previous methods frequently focused on a more general analysis of politics such as overall polarization of the parliament or party-wide political ideological positions. In this paper, we present a method to analyze ideological positions of individual parliamentary representatives by leveraging the latent knowledge of LLMs. The method allows us to evaluate the stance of politicians on an axis of our choice allowing us to flexibly measure the stance of politicians in regards to a topic/controversy of our choice. We achieve this by using a fine-tuned BERT classifier to extract the opinion-based sentences from the speeches of representatives and projecting the average BERT embeddings for each representative on a pair of reference seeds. These reference seeds are either manually chosen representatives known to have opposing views on a particular topic or they are generated sentences which where created using the GPT-4 model of OpenAI. We created the sentences by prompting the GPT-4 model to generate a speech that would come from a politician defending a particular position.