GenZ: Foundational models as latent variable generators within traditional statistical models

TL;DR

GenZ tackles the gap between foundational-model knowledge and dataset-specific prediction by learning interpretable semantic features that connect semantics to real-valued targets through a generalized EM framework. A frozen foundation model classifies items using Discoveries in latent binary features ${\bf z}$, while a statistical model learns $p({\bf y}|{\bf z})$, updating both the semantic descriptors $\theta_f$ and feature uncertainties $\theta_e$ via a variational bound $L$. Across hedonic house-pricing and Netflix cold-start tasks, GenZ outperforms a strong 0-shot GPT-5 baseline and reveals human-interpretable patterns—such as location and architectural quality for housing, and franchises, prestige signals, and temporal cohorts for movies. The approach enables robust high-dimensional and multi-task predictions by translating data-driven splits into semantic prompts, providing practical gains in prediction accuracy and interpretability in real-world domains.

Abstract

We present GenZ, a hybrid model that bridges foundational models and statistical modeling through interpretable semantic features. While large language models possess broad domain knowledge, they often fail to capture dataset-specific patterns critical for prediction tasks. Our approach addresses this by discovering semantic feature descriptions through an iterative process that contrasts groups of items identified via statistical modeling errors, rather than relying solely on the foundational model's domain understanding. We formulate this as a generalized EM algorithm that jointly optimizes semantic feature descriptors and statistical model parameters. The method prompts a frozen foundational model to classify items based on discovered features, treating these judgments as noisy observations of latent binary features that predict real-valued targets through learned statistical relationships. We demonstrate the approach on two domains: house price prediction (hedonic regression) and cold-start collaborative filtering for movie recommendations. On house prices, our model achieves 12\% median relative error using discovered semantic features from multimodal listing data, substantially outperforming a GPT-5 baseline (38\% error) that relies on the LLM's general domain knowledge. For Netflix movie embeddings, our model predicts collaborative filtering representations with 0.59 cosine similarity purely from semantic descriptions -- matching the performance that would require approximately 4000 user ratings through traditional collaborative filtering. The discovered features reveal dataset-specific patterns (e.g., architectural details predicting local housing markets, franchise membership predicting user preferences) that diverge from the model's domain knowledge alone.

Figures (6)

• Figure 1: Feature classification $h(s,\theta_f)$ with extraction prompt strings shown in Python syntax. Items $s$ in the list are featurized using feature descriptions in $\theta_f$. The first prompt is used within a double loop ($t,i$) over items and features. The second is looped only over $i$ (and batches of items {s}) as each call retrieves the indices of items for which $z_i = 1$ (with $z_i = 0$ for all other items).
• Figure 2: Discovering feature descriptions for prompts in Fig. \ref{['fig:prompts']}. When the statistical model identifies a binary split in the data, we can prompt a foundational model to generate an explanation by providing example items from each group. This explanation can then be used as a feature descriptor in $\theta_f$. The prompt need not include all members of each group, only representative examples.
• Figure 3: House listing images (left) and corresponding MLLM-generated semantic item (right). Strikethrough price indicates that price is withheld from items $s$ in training our model. However, in the alternative to our approach, all items including prices were included in baseline prompting, where GPT 5 was given the entire dataset and asked to generate features predictive of the price.
• Figure 4: Hedonic regression results on log of house prices. Both plots show the median absolute error (MAE) (left axis) and number of features (right axis) over iterations. The horizontal dashed line shows GPT-5 0-shot baseline (MAE=0.38). The video showing which features are added (blue) or removed in (b) is available at: https://github.com/mjojic/genZ/tree/main/media.
• Figure 5: Cold start convergence simulation. Cosine similarity between embeddings computed from partial ratings vs. full ratings, as user ratings are gradually added in random order. The shaded region shows 90% confidence interval over 100 randomly selected movies. Hundreds of ratings are typically needed before embeddings converge, illustrating the severity of the cold start problem that our hybrid model addresses by predicting embeddings from semantic features alone.