Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring a New Competency Modeling Process with Large Language Models

Silin Du, Manqing Xin, Raymond Jia Wang

TL;DR

The paper addresses the inefficiency and limited reproducibility of traditional competency modeling by redesigning the workflow with a full LLM-based framework (CoLLM). It combines in-context LLM extraction of behavioral and psychological signals, embedding-based mapping to predefined libraries, and a learnable weight $\alpha$ to fuse signals, all validated via an offline evaluation procedure. In a real-world software outsourcing setting, CoLLM achieves strong predictive validity (e.g., $AUC$ $>$ 0.7 and $\rho$ $>$ 0.3) and demonstrates robustness across competency libraries and LLM backends, while maintaining cross-library conceptual consistency. The approach promises a scalable, transparent, and evaluable alternative to expert-driven competency modeling, especially benefiting small and medium-sized organizations seeking cost-effective talent analytics.

Abstract

Competency modeling is widely used in human resource management to select, develop, and evaluate talent. However, traditional expert-driven approaches rely heavily on manual analysis of large volumes of interview transcripts, making them costly and prone to randomness, ambiguity, and limited reproducibility. This study proposes a new competency modeling process built on large language models (LLMs). Instead of merely automating isolated steps, we reconstruct the workflow by decomposing expert practices into structured computational components. Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data and map them to predefined competency libraries through embedding-based similarity. We further introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals. To address the long-standing challenge of validation, we develop an offline evaluation procedure that allows systematic model selection without requiring additional large-scale data collection. Empirical results from a real-world implementation in a software outsourcing company demonstrate strong predictive validity, cross-library consistency, and structural robustness. Overall, our framework transforms competency modeling from a largely qualitative and expert-dependent practice into a transparent, data-driven, and evaluable analytical process.

Exploring a New Competency Modeling Process with Large Language Models

TL;DR

The paper addresses the inefficiency and limited reproducibility of traditional competency modeling by redesigning the workflow with a full LLM-based framework (CoLLM). It combines in-context LLM extraction of behavioral and psychological signals, embedding-based mapping to predefined libraries, and a learnable weight to fuse signals, all validated via an offline evaluation procedure. In a real-world software outsourcing setting, CoLLM achieves strong predictive validity (e.g., 0.7 and 0.3) and demonstrates robustness across competency libraries and LLM backends, while maintaining cross-library conceptual consistency. The approach promises a scalable, transparent, and evaluable alternative to expert-driven competency modeling, especially benefiting small and medium-sized organizations seeking cost-effective talent analytics.

Abstract

Competency modeling is widely used in human resource management to select, develop, and evaluate talent. However, traditional expert-driven approaches rely heavily on manual analysis of large volumes of interview transcripts, making them costly and prone to randomness, ambiguity, and limited reproducibility. This study proposes a new competency modeling process built on large language models (LLMs). Instead of merely automating isolated steps, we reconstruct the workflow by decomposing expert practices into structured computational components. Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data and map them to predefined competency libraries through embedding-based similarity. We further introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals. To address the long-standing challenge of validation, we develop an offline evaluation procedure that allows systematic model selection without requiring additional large-scale data collection. Empirical results from a real-world implementation in a software outsourcing company demonstrate strong predictive validity, cross-library consistency, and structural robustness. Overall, our framework transforms competency modeling from a largely qualitative and expert-dependent practice into a transparent, data-driven, and evaluable analytical process.
Paper Structure (20 sections, 7 equations, 6 figures, 4 tables)

This paper contains 20 sections, 7 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Competency Modeling
  4. Large Language Models
  5. Proposed Method
  6. Traditional Competency Modeling Process
  7. New Process with LLMs
  8. Individual Level.
  9. Group Level.
  10. Offline Evaluation
  11. Experiments
  12. Data Collection and Settings
  13. Competency Modeling Results
  14. Determine the Number of Key Competencies
  15. Ablation and Robustness Analysis
  16. ...and 5 more sections

Figures (6)

  • Figure 1: The Workflow of CoLLM
  • Figure 2: Prompt Template for Extraction of Behavioral and Psychological Descriptions
  • Figure 3: Scores on Each Competence Item Derived from Behavioral (Left) and Psychological (Right) Descriptions of a Participant
  • Figure 4: Loss for Each Epoch
  • Figure 5: Differences in Scores between the High-performance Group and the Average-performance Group
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 1: Competency
  • Definition 2: Competency Modeling