Selecting Language Models for Social Science: Start Small, Start Open, and Validate

TL;DR

The paper addresses how social scientists should select among thousands of LLMs by emphasizing validity, reliability, reproducibility, and replicability. It advocates starting with small, open models and building delimited, task-specific benchmarks for ex-post validation, rather than relying solely on ex-ante benchmarks. It surveys benchmarks, openness levels, footprint considerations, training data ethics, and architecture/fine-tuning choices, offering concrete guidance to maximize reproducibility and transparency. The proposed approach aims to make GenAI in social science more accountable, energy-efficient, and grounded in verifiable pipelines that enable iteration and robust replication across research teams.

Abstract

Currently, there are thousands of large pretrained language models (LLMs) available to social scientists. How do we select among them? Using validity, reliability, reproducibility, and replicability as guides, we explore the significance of: (1) model openness, (2) model footprint, (3) training data, and (4) model architectures and fine-tuning. While ex-ante tests of validity (i.e., benchmarks) are often privileged in these discussions, we argue that social scientists cannot altogether avoid validating computational measures (ex-post). Replicability, in particular, is a more pressing guide for selecting language models. Being able to reliably replicate a particular finding that entails the use of a language model necessitates reliably reproducing a task. To this end, we propose starting with smaller, open models, and constructing delimited benchmarks to demonstrate the validity of the entire computational pipeline.

Figures (1)

• Figure 1: Example tokenization using OpenAI's tiktoken algorithm.