FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems

TL;DR

FACTER proposes a post hoc framework that unites conformal prediction with iterative prompt engineering to mitigate biases in black-box LLM-based recommender systems. It defines a fairness-aware non-conformity score that blends predictive discrepancy with cross-group embedding dissimilarities and uses offline calibration to set a robust threshold, updated online as violations occur. By injecting explicit bias-avoidance patterns into the system prompt and adjusting the threshold adaptively, FACTER achieves substantial reductions in fairness violations (up to 95.5%) on MovieLens-1M and Amazon datasets while preserving competitive recommendation accuracy. The approach requires no retraining and provides statistical guarantees, offering a scalable, black-box-friendly path to fair LLM-driven recommendations with practical impact for real-world deployment.

Abstract

We propose FACTER, a fairness-aware framework for LLM-based recommendation systems that integrates conformal prediction with dynamic prompt engineering. By introducing an adaptive semantic variance threshold and a violation-triggered mechanism, FACTER automatically tightens fairness constraints whenever biased patterns emerge. We further develop an adversarial prompt generator that leverages historical violations to reduce repeated demographic biases without retraining the LLM. Empirical results on MovieLens and Amazon show that FACTER substantially reduces fairness violations (up to 95.5%) while maintaining strong recommendation accuracy, revealing semantic variance as a potent proxy of bias.

Figures (7)

• Figure 1: FACTER's Iterative Prompt Engineering in Practice. (1) A user requests movie recommendations. (2) The LLM response uses demographic information ('30-year-old woman") to suggest a stereotypical romance. (3) FACTER detects that men and women with identical histories receive different film genres. (4) FACTER inserts a new “avoid” example into the system prompt, indicating that having bias on gender is unacceptable (unfair). (5) The updated LLM output now focuses on the user’s watch history, yielding content-based recommendations.
• Figure 2: FACTER Framework Workflow. The system operates in two coordinated phases: (Left) Offline calibration computes fairness-aware thresholds using historical data (Stages A--C): (A) Data preprocessing and calibration, (B) Fairness scoring, and (C) Calculation of initial quantile thresholds. (Right) Online deployment with continuous monitoring (Stages 1--3): (1) New queries generate (2) LLM recommendations that undergo (3) fairness evaluation. Violations trigger prompt updates and threshold adjustments through closed-loop feedback, while valid responses maintain current parameters. The dashed line indicates the persistence of unchanged settings.
• Figure 3: Fairness violation reduction trajectory vs. static baselines. FACTER progressively reduces violations while UP5 remains fixed. Zero-Shot (112) omitted for clarity.
• Figure 4: Violation reduction across LLMs. All models show monotonic improvement, with LLaMA3-8B converging near zero violations by Iteration 3.
• Figure 5: Fairness-accuracy tradeoff comparison. FACTER achieves strong fairness improvement while preserving recommendation quality.

Theorems & Definitions (7)

• Lemma 3.1: Conformal Coverage
• Theorem A.1: Embedding Shift Robustness
• proof
• Theorem A.2: Threshold Update Convergence
• proof
• Theorem A.3: Type II Error Bound
• proof : Sketch