Verbalized Machine Learning: Revisiting Machine Learning with Language Models

TL;DR

Verbalized Machine Learning (VML) reframes learning as prompt-driven optimization where natural language tokens serve as the model’s parameters. By treating LLM prompts as learnable function descriptions and employing a second LLM as optimizer, VML enables explicit encoding of inductive biases, automatic model-class selection, and interpretable updates, extending the scope of in-context learning to iterative, explainable training. The approach is validated through multiple simple and complex tasks, including linear, polynomial, sinusoidal, and image-based problems, and is contrasted with traditional prompt-engineering methods (e.g., APE). While promising for interpretability and adaptability, the authors acknowledge limitations related to variance, data dimensionality, and the need for robust prompting strategies, outlining paths toward more scalable, multi-modal, and theoretically grounded future work.

Abstract

Motivated by the progress made by large language models (LLMs), we introduce the framework of verbalized machine learning (VML). In contrast to conventional machine learning (ML) models that are typically optimized over a continuous parameter space, VML constrains the parameter space to be human-interpretable natural language. Such a constraint leads to a new perspective of function approximation, where an LLM with a text prompt can be viewed as a function parameterized by the text prompt. Guided by this perspective, we revisit classical ML problems, such as regression and classification, and find that these problems can be solved by an LLM-parameterized learner and optimizer. The major advantages of VML include (1) easy encoding of inductive bias: prior knowledge about the problem and hypothesis class can be encoded in natural language and fed into the LLM-parameterized learner; (2) automatic model class selection: the optimizer can automatically select a model class based on data and verbalized prior knowledge, and it can update the model class during training; and (3) interpretable learner updates: the LLM-parameterized optimizer can provide explanations for why an update is performed. We empirically verify the effectiveness of VML, and hope that VML can serve as a stepping stone to stronger interpretability.

Figures (34)

• Figure 1: A comparison between numerical machine learning and VML.
• Figure 2: An overview of iterative optimization and text prompt templates of the learner and the optimizer in the regression example.
• Figure 3: Training dynamics for VML based linear regression. The model is trained for 2 epochs, each with 10 steps.
• Figure 4: Training dynamic for VML based polynomial regression. The model is trained for 2 epochs, each with 10 steps.
• Figure 5: Demonstration of prior injection, and comparison of Llama-3, GPT-4o and a neural net in the sinusoidal regression setting.