Federated Fine-Tuning of Large Language Models: Kahneman-Tversky vs. Direct Preference Optimization

TL;DR

This work tackles the challenge of fine-tuning large language models in federated learning settings, where data is decentralized and non-IID. It compares Kahneman-Tversky Optimization (KTO) with Direct Preference Optimization (DPO) using Alpaca-7B with LoRA, and introduces a redistributed data regime (KTOR) where DPO is inapplicable. Across multiple FL aggregators and benchmarks (MT-Bench-1, Vicuna, AdvBench), KTOO and KTOR consistently outperform DPO, with KTOR showing robustness to data redistribution. The study demonstrates KTO as a flexible, privacy-preserving fine-tuning method suitable for heterogeneous FL environments, and outlines future directions including quantization, broader evaluation, and dedicated KTO datasets, with open-source artifacts to foster reproducibility.

Abstract

We evaluate Kahneman-Tversky Optimization (KTO) as a fine-tuning method for large language models (LLMs) in federated learning (FL) settings, comparing it against Direct Preference Optimization (DPO). Using Alpaca-7B as the base model, we fine-tune on a realistic dataset under both methods and evaluate performance using MT-Bench-1, Vicuna, and AdvBench benchmarks. Additionally, we introduce a redistributed dataset setup, where only KTO is applicable due to its ability to handle single-response feedback, unlike DPO's reliance on paired responses. Our results demonstrate that KTO, in both its original (KTOO) and redistributed (KTOR) configurations, consistently outperforms DPO across all benchmarks. In the redistributed setup, KTO further validates its flexibility and resilience by maintaining superior performance in scenarios where DPO cannot be applied. These findings establish KTO as a robust and scalable fine-tuning method for FL, motivating its adoption for privacy-preserving, decentralized, and heterogeneous environments.

Figures (1)

• Figure 1: Difference in data design between DPO, KTOO, and KTOR.