We Can't Understand AI Using our Existing Vocabulary

TL;DR

The paper argues that existing human vocabulary cannot fully capture AI concepts, reframing interpretability as a human–machine communication problem. It proposes neologism embedding learning to create concise, compositional words for human and machine concepts, enabling control and understanding without altering model weights. The authors demonstrate proof-of-concept neologisms for length ($H \rightarrow M$), diversity ($H \rightarrow M$), and model-preference ($M \rightarrow H$), showing improvements in controlled generation and diversity. The work outlines connections to prior interpretability methods and discusses potential practical impact and dual-use considerations.

Abstract

This position paper argues that, in order to understand AI, we cannot rely on our existing vocabulary of human words. Instead, we should strive to develop neologisms: new words that represent precise human concepts that we want to teach machines, or machine concepts that we need to learn. We start from the premise that humans and machines have differing concepts. This means interpretability can be framed as a communication problem: humans must be able to reference and control machine concepts, and communicate human concepts to machines. Creating a shared human-machine language through developing neologisms, we believe, could solve this communication problem. Successful neologisms achieve a useful amount of abstraction: not too detailed, so they're reusable in many contexts, and not too high-level, so they convey precise information. As a proof of concept, we demonstrate how a "length neologism" enables controlling LLM response length, while a "diversity neologism" allows sampling more variable responses. Taken together, we argue that we cannot understand AI using our existing vocabulary, and expanding it through neologisms creates opportunities for both controlling and understanding machines better.

Figures (7)

• Figure 1: Humans and machines conceptualize the world differently from each other. Mismatches in communication occur, which lead to misunderstandings. To understand and control AI, we must bridge this gap by developing new words corresponding to human and machine concepts, and use these words to control machines.
• Figure 2: Machine and humans may fundamentally understand the world differently, enabling different concepts, knowledge and capabilities. Figure reproduced from kim2022beyondschut2023bridging with permission.
• Figure 3: Concept-based neologisms sit in-between mechanistic interpretability (which is closer to mechanistic details) and behavioral experiments/capability benchmarking (which is only concerned with the model's output, not how it arrived there).
• Figure 4: Our neologism embedding learning only updates new word embedding, preserving the original model's responses when the new word is not used.
• Figure 5: Base models prompted for length control fail to generate specified long generations (blue), but with a neologism (orange), they consistently generate longer responses.