Creating Healthy Friction: Determining Stakeholder Requirements of Job Recommendation Explanations

TL;DR

Job recommender systems in high-stakes recruitment require trustworthy, transparent explanations for diverse stakeholders. The authors implement a pre-registered mixed-design study (with $n=30$ participants) comparing real explanations generated by an explainable graph neural network (eGNN) against randomized baselines, across textual, bar chart, and graph-based modalities. They find that real explanations do not significantly improve decision speed or correctness, though there are non-significant trends toward higher perceived trust, usefulness, and transparency; preferences for explanation modality vary by stakeholder (text for candidates/recruiters, graphs for company representatives). The work contributes a stakeholder-specific, explainable JRS framework and emphasizes decision-support and healthy friction over persuasive explanations, offering design guidelines (e.g., direct CV/vacancy references, explicit negative weights, interactivity) for practical deployment and future cross-domain validation.

Abstract

The increased use of information retrieval in recruitment, primarily through job recommender systems (JRSs), can have a large impact on job seekers, recruiters, and companies. As a result, such systems have been determined to be high-risk in recent legislature. This requires JRSs to be trustworthy and transparent, allowing stakeholders to understand why specific recommendations were made. To fulfill this requirement, the stakeholders' exact preferences and needs need to be determined. To do so, we evaluated an explainable job recommender system using a realistic, task-based, mixed-design user study (n=30) in which stakeholders had to make decisions based on the model's explanations. This mixed-methods evaluation consisted of two objective metrics - correctness and efficiency, along with three subjective metrics - trust, transparency, and usefulness. These metrics were evaluated twice per participant, once using real explanations and once using random explanations. The study included a qualitative analysis following a think-aloud protocol while performing tasks adapted to each stakeholder group. We find that providing stakeholders with real explanations does not significantly improve decision-making speed and accuracy. Our results showed a non-significant trend for the real explanations to outperform the random ones on perceived trust, usefulness, and transparency of the system for all stakeholder types. We determine that stakeholders benefit more from interacting with explanations as decision support capable of providing healthy friction, rather than as previously-assumed persuasive tools.

Figures (3)

• Figure 1: The interface of the online environment with which the participants interacted. In this screenshot, all explanations are enabled. These can individually be toggled based on the user's preference. The web environment uses exclusively Dutch text, as the interviewees were all native Dutch speakers. The environment consists of the following components: (1) the list of recommended items, which were presented in a randomized order (i.e., the top item was not necessarily the best match); (2) the textual explanation; (3) the bar chart explanation; (4) the graph-based explanation. This example shows a real explanation. For an example of a random explanation, see \ref{['app:A']}.
• Figure 2: An example of a random explanation presented to participants. This explanation relates to the same match as that in \ref{['fig:interface']}.
• Figure 3: The same explanations as in \ref{['fig:interface']}, with the indicative numbers removed. This, again, shows the real explanations.