The Career Interests of Large Language Models

TL;DR

This work investigates whether Large Language Models exhibit human-like career interests by applying the OIP short form within a linear mixed-effects framework, testing four LLMs (GPT-3.5, Gemini-1.5, ERNIE-3.5, Spark-3.5) across English and Chinese prompts with 20 replications per item. The results reveal a consistent tilt toward Social and Artistic tasks (with Investigative also prominent), while language and version changes induce shifts in patterns; crucially, there is a lack of alignment between interests and competence, as evidenced by weak correlations between self-rated and expert-rated performance. The study argues for more naturalistic evaluation environments and AI-specific psychometrics to guide the integration of LLMs into workplaces, and it highlights potential human-like personality-like traits in LLMs. Overall, the findings underscore the need to consider how AI systems’ perceived interests relate to actual capabilities when assigning professional roles. The work also highlights cross-language effects (English vs Chinese) and version-dependent shifts, informing future cross-cultural and longitudinal assessments of AI dispositions.

Abstract

Recent advancements in Large Language Models (LLMs) have significantly extended their capabilities, evolving from basic text generation to complex, human-like interactions. In light of the possibilities that LLMs could assume significant workplace responsibilities, it becomes imminently necessary to explore LLMs' capacities as professional assistants. This study focuses on the aspect of career interests by applying the Occupation Network's Interest Profiler short form to LLMs as if they were human participants and investigates their hypothetical career interests and competence, examining how these vary with language changes and model advancements. We analyzed the answers using a general linear mixed model approach and found distinct career interest inclinations among LLMs, particularly towards the social and artistic domains. Interestingly, these preferences did not align with the occupations where LLMs exhibited higher competence. This novel approach of using psychometric instruments and sophisticated statistical tools on LLMs unveils fresh perspectives on their integration into professional environments, highlighting human-like tendencies and promoting a reevaluation of LLMs' self-perception and competency alignment in the workforce.

Figures (21)

• Figure 1: Overall framework used to assess the career interest of LLMs. Sixty OIP short form English or Chinese version items were administered to each LLM. A prompt instructing the participant, i.e. LLM in this setting, was applied before each item. A total score for the interest category was calculated by summing the item score for corresponding work tasks.
• Figure 2: Interest scores of 6 categories of main LLMs containing gpt-3.5-turbo, Gemini-pro, ERNIE-3.5 and Spark-1.5, and Holland interest code of main LLMs and the corresponding occupations. Results showed significant differences in the career interest scores of the LLMs that, according to the OIP Short Form scoring rules, they would be recommended for different occupations. As recommended by O*NET, example occupations corresponding with each code are SAI: Art therapist, Education Teachers, English language and literature teachers, Marriage and family therapists; SIA: Environmental science teachers, family and consumer science teachers, genetic counselors, mental health and substance abuse social workers; ISA: clinical and counseling psychologists, psychiatrists; IAS: political scientists, sociologists. OIP does not have corresponding codes for ASI or AIS.
• Figure 5: The differences between expected values of Holland interest scores for Chinese and English languages for each LLM and Holland interest.
• Figure : (a)
• Figure : (a)