Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Programmatic Representation Learning with Language Models

Gabriel Poesia, Georgia Gabriela Sampaio

TL;DR

LeaPR addresses the limitations of hand-engineered features in classical predictors and the interpretability gaps of deep nets by learning programmatic features generated by LLMs and combining them with decision trees. It introduces two algorithms, F2 and D-ID3, enabling end-to-end learning of thousands of domain-specific features on top of simple predictors. Across chess, image, and text domains, LeaPR achieves competitive accuracy with neural networks while remaining neural-network-free and interpretable; it also enables data exploration and failure debugging via SHAP. The work suggests a flexible paradigm where features and predictions are inspectable and can be adapted across domains.

Abstract

Classical models for supervised machine learning, such as decision trees, are efficient and interpretable predictors, but their quality is highly dependent on the particular choice of input features. Although neural networks can learn useful representations directly from raw data (e.g., images or text), this comes at the expense of interpretability and the need for specialized hardware to run them efficiently. In this paper, we explore a hypothesis class we call Learned Programmatic Representations (LeaPR) models, which stack arbitrary features represented as code (functions from data points to scalars) and decision tree predictors. We synthesize feature functions using Large Language Models (LLMs), which have rich prior knowledge in a wide range of domains and a remarkable ability to write code using existing domain-specific libraries. We propose two algorithms to learn LeaPR models from supervised data. First, we design an adaptation of FunSearch to learn features rather than directly generate predictors. Then, we develop a novel variant of the classical ID3 algorithm for decision tree learning, where new features are generated on demand when splitting leaf nodes. In experiments from chess position evaluation to image and text classification, our methods learn high-quality, neural network-free predictors often competitive with neural networks. Our work suggests a flexible paradigm for learning interpretable representations end-to-end where features and predictions can be readily inspected and understood.

Programmatic Representation Learning with Language Models

TL;DR

LeaPR addresses the limitations of hand-engineered features in classical predictors and the interpretability gaps of deep nets by learning programmatic features generated by LLMs and combining them with decision trees. It introduces two algorithms, F2 and D-ID3, enabling end-to-end learning of thousands of domain-specific features on top of simple predictors. Across chess, image, and text domains, LeaPR achieves competitive accuracy with neural networks while remaining neural-network-free and interpretable; it also enables data exploration and failure debugging via SHAP. The work suggests a flexible paradigm where features and predictions are inspectable and can be adapted across domains.

Abstract

Classical models for supervised machine learning, such as decision trees, are efficient and interpretable predictors, but their quality is highly dependent on the particular choice of input features. Although neural networks can learn useful representations directly from raw data (e.g., images or text), this comes at the expense of interpretability and the need for specialized hardware to run them efficiently. In this paper, we explore a hypothesis class we call Learned Programmatic Representations (LeaPR) models, which stack arbitrary features represented as code (functions from data points to scalars) and decision tree predictors. We synthesize feature functions using Large Language Models (LLMs), which have rich prior knowledge in a wide range of domains and a remarkable ability to write code using existing domain-specific libraries. We propose two algorithms to learn LeaPR models from supervised data. First, we design an adaptation of FunSearch to learn features rather than directly generate predictors. Then, we develop a novel variant of the classical ID3 algorithm for decision tree learning, where new features are generated on demand when splitting leaf nodes. In experiments from chess position evaluation to image and text classification, our methods learn high-quality, neural network-free predictors often competitive with neural networks. Our work suggests a flexible paradigm for learning interpretable representations end-to-end where features and predictions can be readily inspected and understood.
Paper Structure (32 sections, 3 figures, 3 tables, 2 algorithms)

This paper contains 32 sections, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code generation with LLMs
  4. Black-box code evolution with LLMs
  5. Learning programmatic world models
  6. Mechanistic interpretability of neural networks
  7. Learning Programmatic Representations
  8. Black-box feature generation
  9. Dynamic splitting
  10. Experiments
  11. Chess position evaluation
  12. Image classification
  13. Text classification
  14. Case studies: understanding features and model predictions
  15. Limitations and Conclusion
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Learned Programmatic Representation models combine programmatic features, synthesized by LLMs as code, and decision tree predictors, yielding interpretable models. We give two algorithms for learning them end-to-end from supervised data in high-dimensional input domains.
  • Figure 2: Features FunSearch (F2)
  • Figure 3: (Left) Distribution of the top 2 LeaPR-learned features with highest SHAP values on the Ghostbuster dataset (gaussian jitter added to aid visualization). On Feature 1 (which measures fraction of "curly", or typographic characters, versus plain ASCII quotes), human text tends to cluster at the extremes, while AI-generated text features mid-range values. Feature 2 shows high values primarily for human-written text. (Right) A land bird misclassified by a LeaPR model on the Waterbird dataset; the top SHAP-valued features for this example show a clear reliance on the background.