Language processing in humans and computers
Dusko Pavlovic
TL;DR
This work surveys language processing in humans and computers, contrasting how meaning is constructed and communicated in biological systems with how language engines operate. It frames language production as predictive inference and situates it within a broad information-theoretic and channel-based perspective, arguing that both human and machine learning rely on language-like structures to build and share meaning. The text traverses syntax (from formal grammars and pregroup grammars to dependency parsing), semantics (static vector-based representations and dynamic, context-sensitive meaning), and learning theory (from pigeons to perceptrons and Kolmogorov-Arnold decompositions), presenting language as a universal learning mechanism. It also highlights practical and philosophical challenges—hallucinations, copyright, grounding, and the evolving role of the Web—while proposing a framework in which future AI systems could safely extend human cognition through grounded, context-aware learning pipelines.
Abstract
Machine-learned language models have transformed everyday life: they steer us when we study, drive, manage money. They have the potential to transform our civilization. But they hallucinate. Their realities are virtual. This note provides a high-level overview of language models and outlines a low-level model of learning machines. It turns out that, after they become capable of recognizing hallucinations and dreaming safely, as humans tend to be, the language-learning machines proceed to generate broader systems of false beliefs and self-confirming theories, as humans tend to do.