LABOR-LLM: Language-Based Occupational Representations with Large Language Models

TL;DR

LABOR-LLM leverages a foundation-model–fine-tuning approach to predict the next occupation from lengthy career histories. By converting career data into text templates and fine-tuning Llama-2 models, FT-LABOR-LLM achieves state-of-the-art perplexity and calibration for granular next-occupation predictions, outperforming CAREER and off-the-shelf LLMs. The method demonstrates that non-representative data can substitute for model size, and that including textual job titles and rich demographic interactions enhances predictive accuracy. This enables more reliable causal inference, decompositions, and structural models in labor economics, while offering a scalable, data-friendly workflow that benefits replication and extension. The work also highlights limitations under systemic changes and suggests avenues for integrating LLM-based predictions with traditional econometric models.

Abstract

This paper builds an empirical model that predicts a worker's next occupation as a function of the worker's occupational history. Because histories are sequences of occupations, the covariate space is high-dimensional, and further, the outcome (the next occupation) is a discrete choice that can take on many values. To estimate the parameters of the model, we leverage an approach from generative artificial intelligence. Estimation begins from a ``foundation model'' trained on non-representative data and then ``fine-tunes'' the estimation using data about careers from a representative survey. We convert tabular data from the survey into text files that resemble resumes and fine-tune the parameters of the foundation model, a large language model (LLM), using these text files with the objective of predicting the next token (word). The resulting fine-tuned LLM is used to calculate estimates of worker transition probabilities. Its predictive performance surpasses all prior models, both for the task of granularly predicting the next occupation as well as for specific tasks such as predicting whether the worker changes occupations or stays in the labor force. We quantify the value of fine-tuning and further show that by adding more career data from a different population, fine-tuning smaller LLMs (fewer parameters) surpasses the performance of fine-tuning larger models. When we omit the English language occupational title and replace it with a unique code, predictive performance declines.

Figures (8)

• Figure 1: Distributions of individuals' birth years by survey dataset.
• Figure 2: Distribution of individuals' ages by calendar year of observation.
• Figure 3: Illustration of the model fine-tuning procedure.
• Figure 4: Calibration plots of baseline and fine-tuned models on the task of predicting staying in an occupation vs. moving occupations.
• Figure 5: Average covariate values within each quintile as defined by the predicted difference in log-likelihood on conditional prediction.