Building AI Agents to Improve Job Referral Requests to Strangers
Ross Chu, Yuting Huang
TL;DR
The paper tackles how to help job seekers obtain referrals by automatically rewriting and evaluating referral requests in an online community. It introduces a two-agent workflow (an improver and an evaluator) and augments it with Retrieval-Augmented Generation (RAG) to guide edits using well-written examples and editorial ratings. The basic workflow improves weaker requests but can harm stronger ones; adding RAG mitigates degradation and yields stronger gains for weaker requests, achieving roughly a $+14\%$ relative improvement in predicted success for weak requests. A reward-model-based proxy is used to assess quality, providing low-cost signals for prioritizing features before real-world field experiments. The findings offer a scalable method to prototype referral-enhancement tools and guide subsequent A/B tests on platforms like Blind.
Abstract
This paper develops AI agents that help job seekers write effective requests for job referrals in a professional online community. The basic workflow consists of an improver agent that rewrites the referral request and an evaluator agent that measures the quality of revisions using a model trained to predict the probability of receiving referrals from other users. Revisions suggested by the LLM (large language model) increase predicted success rates for weaker requests while reducing them for stronger requests. Enhancing the LLM with Retrieval-Augmented Generation (RAG) prevents edits that worsen stronger requests while it amplifies improvements for weaker requests. Overall, using LLM revisions with RAG increases the predicted success rate for weaker requests by 14\% without degrading performance on stronger requests. Although improvements in model-predicted success do not guarantee more referrals in the real world, they provide low-cost signals for promising features before running higher-stakes experiments on real users.