Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations

Dingjie Fu, Dianxing Shi

TL;DR

This study evaluates whether state-of-the-art large language models can pass a standardized hiring evaluation used by tech firms, by administering a two-stage process (candidate questionnaire and HR-style recommendations) to 12 LLMs and two human volunteers. Model performance is quantified with $RMSE$ and Pearson correlation coefficient $\gamma$ against company-referenced answers, plus qualitative human analysis. The results show a robust misalignment: all evaluated LLMs fail to pass the hiring evaluation, with $RMSE$ typically exceeding $2.0$ for non-reasoning models and most $\gamma$ in the $0.3$–$0.5$ range; most models trend toward positive responses, and HR-style recommendations are not discriminative. An outlier, Qwen3-32B, diverges notably, while some reasoning-capable models (e.g., $0.6DeepSeek-R1$) approach better alignment but still do not pass. The work highlights a gap between web-scale training and company-specific hiring criteria, pointing to the need for alignment mechanisms and careful evaluation before deploying LLMs in hiring workflows.

Abstract

With the proliferation of the internet and the rapid advancement of Artificial Intelligence, leading technology companies face an urgent annual demand for a considerable number of software and algorithm engineers. To efficiently and effectively identify high-potential candidates from thousands of applicants, these firms have established a multi-stage selection process, which crucially includes a standardized hiring evaluation designed to assess job-specific competencies. Motivated by the demonstrated prowess of Large Language Models (LLMs) in coding and reasoning tasks, this paper investigates a critical question: Can LLMs successfully pass these hiring evaluations? To this end, we conduct a comprehensive examination of a widely used professional assessment questionnaire. We employ state-of-the-art LLMs to generate responses and subsequently evaluate their performance. Contrary to any prior expectation of LLMs being ideal engineers, our analysis reveals a significant inconsistency between the model-generated answers and the company-referenced solutions. Our empirical findings lead to a striking conclusion: All evaluated LLMs fails to pass the hiring evaluation.

"You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations

TL;DR

This study evaluates whether state-of-the-art large language models can pass a standardized hiring evaluation used by tech firms, by administering a two-stage process (candidate questionnaire and HR-style recommendations) to 12 LLMs and two human volunteers. Model performance is quantified with and Pearson correlation coefficient against company-referenced answers, plus qualitative human analysis. The results show a robust misalignment: all evaluated LLMs fail to pass the hiring evaluation, with typically exceeding for non-reasoning models and most in the – range; most models trend toward positive responses, and HR-style recommendations are not discriminative. An outlier, Qwen3-32B, diverges notably, while some reasoning-capable models (e.g., ) approach better alignment but still do not pass. The work highlights a gap between web-scale training and company-specific hiring criteria, pointing to the need for alignment mechanisms and careful evaluation before deploying LLMs in hiring workflows.

Abstract

With the proliferation of the internet and the rapid advancement of Artificial Intelligence, leading technology companies face an urgent annual demand for a considerable number of software and algorithm engineers. To efficiently and effectively identify high-potential candidates from thousands of applicants, these firms have established a multi-stage selection process, which crucially includes a standardized hiring evaluation designed to assess job-specific competencies. Motivated by the demonstrated prowess of Large Language Models (LLMs) in coding and reasoning tasks, this paper investigates a critical question: Can LLMs successfully pass these hiring evaluations? To this end, we conduct a comprehensive examination of a widely used professional assessment questionnaire. We employ state-of-the-art LLMs to generate responses and subsequently evaluate their performance. Contrary to any prior expectation of LLMs being ideal engineers, our analysis reveals a significant inconsistency between the model-generated answers and the company-referenced solutions. Our empirical findings lead to a striking conclusion: All evaluated LLMs fails to pass the hiring evaluation.
Paper Structure (21 sections, 2 equations, 8 figures, 4 tables)

This paper contains 21 sections, 2 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experimental Setup
  4. Results and Discussion
  5. Quantitative Analysis (RQ1)
  6. Finding 1:
  7. The Profile of LLM (RQ2)
  8. Finding 2:
  9. Qualitative Analysis (RQ3)
  10. Finding 3:
  11. Conclusion
  12. List of Questionnaire and Reference Answers
  13. Implementation Details
  14. Data Generation
  15. Models
  16. ...and 6 more sections

Figures (8)

  • Figure 1: The hiring evaluation process involves two stages: (1) candidates responding to a standardized questionnaire, (2) the system provides a recommendation based on their answers.
  • Figure 2: Average scores of LLMs versus human references on positive (P) and negative (N) questions. Abbreviated model names are used for clarity.
  • Figure 3: Pearson correlation coefficients across 12 LLMs. While LLMs exhibit highly consistent, the Qwen3-32B emerges as a notable outlier. Abbreviated model names are used for clarity.
  • Figure 4: HR-Candidates Recommend Matrix. Not Recommended is not shown in the legend because it does not appear in the matrix.
  • Figure 5: Candidates' prompt
  • ...and 3 more figures