Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAREER: A Foundation Model for Labor Sequence Data

Keyon Vafa, Emil Palikot, Tianyu Du, Ayush Kanodia, Susan Athey, David M. Blei

TL;DR

CAREER addresses the challenge of leveraging large-scale resume data to improve predictive models of labor trajectories while maintaining generalization to survey data. It introduces a transformer-based two-stage representation model pretrained on 24M resume sequences and fine-tuned on small economic datasets, outperforming econometric baselines in occupation prediction and forecasting, and enhancing wage predictions when integrated into wage models. The approach demonstrates effective transfer learning across domains, reveals scaling laws for pretraining data, and provides a practical tool for downstream economic analyses. This work contributes to both methodology (foundation models for discrete labor sequences) and applied economics by enabling more accurate, data-driven labor market inferences.

Abstract

Labor economists regularly analyze employment data by fitting predictive models to small, carefully constructed longitudinal survey datasets. Although machine learning methods offer promise for such problems, these survey datasets are too small to take advantage of them. In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals. However, standard econometric models cannot take advantage of their scale or incorporate them into the analysis of survey data. To this end we develop CAREER, a foundation model for job sequences. CAREER is first fit to large, passively-collected resume data and then fine-tuned to smaller, better-curated datasets for economic inferences. We fit CAREER to a dataset of 24 million job sequences from resumes, and adjust it on small longitudinal survey datasets. We find that CAREER forms accurate predictions of job sequences, outperforming econometric baselines on three widely-used economics datasets. We further find that CAREER can be used to form good predictions of other downstream variables. For example, incorporating CAREER into a wage model provides better predictions than the econometric models currently in use.

CAREER: A Foundation Model for Labor Sequence Data

TL;DR

CAREER addresses the challenge of leveraging large-scale resume data to improve predictive models of labor trajectories while maintaining generalization to survey data. It introduces a transformer-based two-stage representation model pretrained on 24M resume sequences and fine-tuned on small economic datasets, outperforming econometric baselines in occupation prediction and forecasting, and enhancing wage predictions when integrated into wage models. The approach demonstrates effective transfer learning across domains, reveals scaling laws for pretraining data, and provides a practical tool for downstream economic analyses. This work contributes to both methodology (foundation models for discrete labor sequences) and applied economics by enabling more accurate, data-driven labor market inferences.

Abstract

Labor economists regularly analyze employment data by fitting predictive models to small, carefully constructed longitudinal survey datasets. Although machine learning methods offer promise for such problems, these survey datasets are too small to take advantage of them. In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals. However, standard econometric models cannot take advantage of their scale or incorporate them into the analysis of survey data. To this end we develop CAREER, a foundation model for job sequences. CAREER is first fit to large, passively-collected resume data and then fine-tuned to smaller, better-curated datasets for economic inferences. We fit CAREER to a dataset of 24 million job sequences from resumes, and adjust it on small longitudinal survey datasets. We find that CAREER forms accurate predictions of job sequences, outperforming econometric baselines on three widely-used economics datasets. We further find that CAREER can be used to form good predictions of other downstream variables. For example, incorporating CAREER into a wage model provides better predictions than the econometric models currently in use.
Paper Structure (23 sections, 17 equations, 5 figures, 8 tables)

This paper contains 23 sections, 17 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. CAREER
  3. Occupation Models
  4. Representation-Based Two-Stage Models
  5. CAREER Model
  6. Related Work
  7. Empirical Studies
  8. Conclusion
  9. Econometric Baselines
  10. Qualitative Analysis
  11. Transformer Details
  12. Exploratory Data Analysis
  13. One-Stage vs Two-Stage Prediction
  14. Data Preprocessing
  15. Resumes.
  16. ...and 8 more sections

Figures (5)

  • Figure 1: CAREER's computation graph. CAREER parameterizes a low-dimensional representation of an individual's career history with a transformer, which it uses to predict the next job.
  • Figure 2: Prediction results on longitudinal survey datasets and scaling law.
  • Figure 3: An example of a held-out job sequence on PSID along with CAREER's rationale. CAREER ranks the true next job (biological technician) as the most likely possible transition for this individual; in contrast, the regression and bag-of-jobs model rank it as 40th and 37th most likely, respectively. The rationale depicts the jobs in the history that were sufficient for CAREER's prediction.
  • Figure 4: Held-out perplexity as a function of career length on NLSY79. CAREER has a predictive advantage at all points of an individual's career. The magnitude of this advantage is largest mid-career.
  • Figure 5: The work experiences with the most similar CAREER representations (measured with cosine similarity) for individuals with no overlapping jobs in NLSY97.