Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neologism Learning for Controllability and Self-Verbalization

John Hewitt, Oyvind Tafjord, Robert Geirhos, Been Kim

TL;DR

This work introduces neologism learning as a vocabulary-based method to control and interpret language models without changing their parameters. By expanding the embedding matrix with new tokens and training these embeddings to encode specific concepts through data generation and a specialized APO-up loss, the approach achieves targeted steering on simple and AxBench concepts and reveals model-described self-verbalizations. It further demonstrates that multiple neologisms can be learned jointly to compose complex behaviors, and that self-verbalizations and plug-in evaluation uncover machine-only synonyms that transfer across models. Overall, the paper provides a framework for measuring and leveraging learned neologisms to improve controllability and interpretability in contemporary LLMs.

Abstract

Humans invent new words when there is a rising demand for a new useful concept (e.g., doomscrolling). We explore and validate a similar idea in our communication with LLMs: introducing new words to better understand and control the models, expanding on the recently introduced neologism learning. This method introduces a new word by adding a new word embedding and training with examples that exhibit the concept with no other changes in model parameters. We show that adding a new word allows for control of concepts such as flattery, incorrect answers, text length, as well as more complex concepts in AxBench. We discover that neologisms can also further our understanding of the model via self-verbalization: models can describe what each new word means to them in natural language, like explaining that a word that represents a concept of incorrect answers means ``a lack of complete, coherent, or meaningful answers...'' To validate self-verbalizations, we introduce plug-in evaluation: we insert the verbalization into the context of a model and measure whether it controls the target concept. In some self-verbalizations, we find machine-only synonyms: words that seem unrelated to humans but cause similar behavior in machines. Finally, we show how neologism learning can jointly learn multiple concepts in multiple words.

Neologism Learning for Controllability and Self-Verbalization

TL;DR

This work introduces neologism learning as a vocabulary-based method to control and interpret language models without changing their parameters. By expanding the embedding matrix with new tokens and training these embeddings to encode specific concepts through data generation and a specialized APO-up loss, the approach achieves targeted steering on simple and AxBench concepts and reveals model-described self-verbalizations. It further demonstrates that multiple neologisms can be learned jointly to compose complex behaviors, and that self-verbalizations and plug-in evaluation uncover machine-only synonyms that transfer across models. Overall, the paper provides a framework for measuring and leveraging learned neologisms to improve controllability and interpretability in contemporary LLMs.

Abstract

Humans invent new words when there is a rising demand for a new useful concept (e.g., doomscrolling). We explore and validate a similar idea in our communication with LLMs: introducing new words to better understand and control the models, expanding on the recently introduced neologism learning. This method introduces a new word by adding a new word embedding and training with examples that exhibit the concept with no other changes in model parameters. We show that adding a new word allows for control of concepts such as flattery, incorrect answers, text length, as well as more complex concepts in AxBench. We discover that neologisms can also further our understanding of the model via self-verbalization: models can describe what each new word means to them in natural language, like explaining that a word that represents a concept of incorrect answers means ``a lack of complete, coherent, or meaningful answers...'' To validate self-verbalizations, we introduce plug-in evaluation: we insert the verbalization into the context of a model and measure whether it controls the target concept. In some self-verbalizations, we find machine-only synonyms: words that seem unrelated to humans but cause similar behavior in machines. Finally, we show how neologism learning can jointly learn multiple concepts in multiple words.

Paper Structure

This paper contains 34 sections, 3 equations, 2 figures, 24 tables.

Table of Contents

  1. Introduction
  2. An Aperitif: Discovering a Machine-Only Synonym
  3. The Neologism Learning Method
  4. Vocabulary Expansion.
  5. Concept definition through data generation.
  6. Training objective.
  7. Neologisms for simple and complex concepts in AxBench
  8. Simple concept steering
  9. AxBench concept steering
  10. Self-verbalization and machine-only synonyms
  11. Synonym self-verbalizations
  12. Synonym Evaluation.
  13. Questionnaire-based self-verbalizations
  14. Learning combinations of neologisms
  15. Data generation and setup
  16. ...and 19 more sections

Figures (2)

  • Figure 1: At left: neologism learning places neologisms---new tokens---in natural language concepts, and trains them to predict concept-bearing outputs (e.g., "single-sentence answer"), while keeping the rest of the model fixed. Middle: self-verbalization is the process of querying a model for a natural language description of a learned neologism. Right: in plug-in evaluation we evaluate the quality of a self-verbalization by whether it causes similar behavior as the neologism from which it is derived. All text in this figure is demonstrative, not real model outputs.
  • Figure 2: Prompt used for in-context neologism learning