Tinkering Against Scaling

TL;DR

The paper addresses the risks of scaling-centric AI, which constrains academic access and reproducibility in computational social science and critical algorithm studies. It proposes a tinkering methodology that engages with smaller models and components, guided by Heideggerian reflection and STS care practices, to democratize understanding and critique of AI systems. The approach is articulated through CSS-specific practices (small-model tinkering, probing large models, and new modeling methodologies) and critical-algorithm perspectives (thick descriptions, operational analyses, and interventionist making). Collectively, tinkering offers a practical, care-centered alternative to scaling, aiming to democratize tools, improve interpretability, and foster local world-making rather than pursuing ever-larger models.

Abstract

The ascent of scaling in artificial intelligence research has revolutionized the field over the past decade, yet it presents significant challenges for academic researchers, particularly in computational social science and critical algorithm studies. The dominance of large language models, characterized by their extensive parameters and costly training processes, creates a disparity where only industry-affiliated researchers can access these resources. This imbalance restricts academic researchers from fully understanding their tools, leading to issues like reproducibility in computational social science and a reliance on black-box metaphors in critical studies. To address these challenges, we propose a "tinkering" approach that is inspired by existing works. This method involves engaging with smaller models or components that are manageable for ordinary researchers, fostering hands-on interaction with algorithms. We argue that tinkering is both a way of making and knowing for computational social science and a way of knowing for critical studies, and fundamentally, it is a way of caring that has broader implications for both fields.

Figures (4)

• Figure 1: Applying STIP on a topic from Fligstein et al. 2016. Y axis presents the estimates and X axis presents the AJI, a similarity measurement. We found that even among topics highly resemble the reference model, there are opposite results.
• Figure 2: Impact of survey-inspired interventions on LLM annotation reliability. Adapted from Figure 3 in li_old_2025.
• Figure 3: Category-specific impacts of interventions on LLM annotation reliability. Adapted from Figure 4 in li_old_2025.
• Figure 4: Multimodaility in communication