Application of LLM Agents in Recruitment: A Novel Framework for Resume Screening

TL;DR

The paper tackles the efficiency bottleneck of resume screening by introducing an LLM-agent framework that handles sentence-level information extraction, grading, summarization, and decision making. It demonstrates that fine-tuned instruction-format LLMs achieve high fidelity in sentence classification ($F1=87.73\%$) and that HR-agent modules can grade and summarize resumes with favorable ROUGE and BLEU scores, outperforming some baselines. Through extensive experiments on IT resumes, including long-context processing with GPT-4–family models and manual annotations, the study shows strong alignment between AI-driven decisions and human judgments (high correlations with manual rankings and consistent top picks). The framework yields substantial time savings (≈$11\times$ faster than manual) while maintaining decision quality, highlighting practical potential for automated, privacy-conscious recruitment workflows, albeit with limitations in cross-industry generalizability and data collection. The work suggests a viable path toward scalable, AI-assisted HR operations using open-source LLMs and agent-based reasoning.

Abstract

The automation of resume screening is a crucial aspect of the recruitment process in organizations. Automated resume screening systems often encompass a range of natural language processing (NLP) tasks. This paper introduces a novel Large Language Models (LLMs) based agent framework for resume screening, aimed at enhancing efficiency and time management in recruitment processes. Our framework is distinct in its ability to efficiently summarize and grade each resume from a large dataset. Moreover, it utilizes LLM agents for decision-making. To evaluate our framework, we constructed a dataset from actual resumes and simulated a resume screening process. Subsequently, the outcomes of the simulation experiment were compared and subjected to detailed analysis. The results demonstrate that our automated resume screening framework is 11 times faster than traditional manual methods. Furthermore, by fine-tuning the LLMs, we observed a significant improvement in the F1 score, reaching 87.73\%, during the resume sentence classification phase. In the resume summarization and grading phase, our fine-tuned model surpassed the baseline performance of the GPT-3.5 model. Analysis of the decision-making efficacy of the LLM agents in the final offer stage further underscores the potential of LLM agents in transforming resume screening processes.

Figures (14)

• Figure 1: The Process of automated resume screening.
• Figure 2: The illustration reprehsents the process of pre-training a language model and applying the pre-trained language model to a downstream task through fine-tuning method.
• Figure 3: The illustration depict LLM as the backbone of the agent system.
• Figure 4: The illustration depict the workflow of LLM agent base Automated Resume Screening Framework.
• Figure 5: The illustration depict the process of instruction tuning and RLHF for the LLaMA2 model.