Contrastive Bi-Encoder Models for Multi-Label Skill Extraction: Enhancing ESCO Ontology Matching with BERT and Attention Mechanisms

TL;DR

This work tackles the problem of mapping unstructured job ads to a large skill ontology (ESCO) in settings with scarce labeled data. It proposes a zero-shot XMLC pipeline that uses LLM-generated, hierarchy-conditioned synthetic supervision and a contrastive bi-encoder with a BiLSTM+attention enhanced backbone, preceded by a RoBERTa-based sentence filter. The approach yields fluent, discriminative synthetic data and strong zero-shot transfer to real Chinese job ads, achieving robust retrieval (e.g., F1@5 around 0.72) and outperforming TF-IDF and standard BERT baselines. The results demonstrate a scalable, data-efficient pathway for automated skill coding in labor economics and show promise for cross-lingual and taxonomy-expansion extensions.

Abstract

Fine-grained labor market analysis increasingly relies on mapping unstructured job advertisements to standardized skill taxonomies such as ESCO. This mapping is naturally formulated as an Extreme Multi-Label Classification (XMLC) problem, but supervised solutions are constrained by the scarcity and cost of large-scale, taxonomy-aligned annotations--especially in non-English settings where job-ad language diverges substantially from formal skill definitions. We propose a zero-shot skill extraction framework that eliminates the need for manually labeled job-ad training data. The framework uses a Large Language Model (LLM) to synthesize training instances from ESCO definitions, and introduces hierarchically constrained multi-skill generation based on ESCO Level-2 categories to improve semantic coherence in multi-label contexts. On top of the synthetic corpus, we train a contrastive bi-encoder that aligns job-ad sentences with ESCO skill descriptions in a shared embedding space; the encoder augments a BERT backbone with BiLSTM and attention pooling to better model long, information-dense requirement statements. An upstream RoBERTa-based binary filter removes non-skill sentences to improve end-to-end precision. Experiments show that (i) hierarchy-conditioned generation improves both fluency and discriminability relative to unconstrained pairing, and (ii) the resulting multi-label model transfers effectively to real-world Chinese job advertisements, achieving strong zero-shot retrieval performance (F1@5 = 0.72) and outperforming TF--IDF and standard BERT baselines. Overall, the proposed pipeline provides a scalable, data-efficient pathway for automated skill coding in labor economics and workforce analytics.

Figures (6)

• Figure 1: Evaluation of synthetic data quality. (a) GPT-2 perplexity distributions. (b) ROC curves from a separability test between synthetic skill sentences and generic non-skill text; the Level-2 constrained variant dominates across thresholds. (c) Word cloud illustrating broad topical coverage of the generated corpus.
• Figure 2: Binary sentence filtering. (a) Confusion matrix for our filter, showing balanced FP/FN counts. (b) Precision--Recall curves comparing our filter to keyword matching across thresholds. (c) Inference latency vs. accuracy for competing approaches.
• Figure 3: XMLC model evaluation. (a) MRR over training epochs. (b) Recall@K on the synthetic benchmark. (c) t-SNE projection of Model C embeddings, colored by ESCO Level-2 categories.
• Figure 4: Ablation analysis. (a) MRR vs. contrastive margin. (b) Trade-off between training efficiency and MRR as the number of negative samples increases. (c) Relative attention scores over tokens, highlighting emphasis on skill-bearing terms.
• Figure 5: End-to-end transfer performance on real-world test data. (a) Precision--Recall curves for Models A/B/C. (b) Correct vs. error ratio in qualitative case studies. (c) Error breakdown into false positives, false negatives, and taxonomy misalignment.