Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs

An Liu, Zonghan Yang, Zhenhe Zhang, Qingyuan Hu, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu

TL;DR

PANDA introduces a tuning-free, preference-based framework to boost domain-specific abilities of large language models by learning from domain expert preferences. It builds an insight pool from expert explanations of preference pairs during a learning stage and retrieves these insights at inference to adapt the LLM via in-context prompts. Across ScienceWorld interactive decision tasks and TweetEval text classification, PANDA substantially improves performance, even surpassing some expert models in certain settings and demonstrating weak-to-strong generalization. The approach avoids gradient-based fine-tuning, enabling applicability to closed-source LLMs, though it relies on effective retrieval and the LLM’s instruction-following capabilities. Overall, PANDA shows strong potential as a tuning-free pathway to domain specialization with notable cross-task transfer and robustness to data quality variations.

Abstract

While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. However, this method can be both resource and time-intensive, and not applicable to closed-source commercial LLMs. In this paper, we propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA), a method designed to augment the domain-specific capabilities of LLMs by leveraging insights from the response preference of expert models without requiring fine-tuning. Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks. Moreover, LLM with PANDA even outperforms the expert model that being learned on 4 tasks of ScienceWorld. This finding highlights the potential of exploring tuning-free approaches to achieve weak-to-strong generalization.

PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs

TL;DR

PANDA introduces a tuning-free, preference-based framework to boost domain-specific abilities of large language models by learning from domain expert preferences. It builds an insight pool from expert explanations of preference pairs during a learning stage and retrieves these insights at inference to adapt the LLM via in-context prompts. Across ScienceWorld interactive decision tasks and TweetEval text classification, PANDA substantially improves performance, even surpassing some expert models in certain settings and demonstrating weak-to-strong generalization. The approach avoids gradient-based fine-tuning, enabling applicability to closed-source LLMs, though it relies on effective retrieval and the LLM’s instruction-following capabilities. Overall, PANDA shows strong potential as a tuning-free pathway to domain specialization with notable cross-task transfer and robustness to data quality variations.

Abstract

While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. However, this method can be both resource and time-intensive, and not applicable to closed-source commercial LLMs. In this paper, we propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA), a method designed to augment the domain-specific capabilities of LLMs by leveraging insights from the response preference of expert models without requiring fine-tuning. Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks. Moreover, LLM with PANDA even outperforms the expert model that being learned on 4 tasks of ScienceWorld. This finding highlights the potential of exploring tuning-free approaches to achieve weak-to-strong generalization.
Paper Structure (37 sections, 1 equation, 3 figures, 18 tables)

This paper contains 37 sections, 1 equation, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs
  3. Overview
  4. Learning from Expert Preferences
  5. Preference Adaptation during Inference
  6. Experiments
  7. Interactive Decision Making
  8. Benchmark.
  9. Baselines.
  10. Results.
  11. Text Classification
  12. Benchmark.
  13. Baselines.
  14. Results.
  15. Ablation and Analysis
  16. ...and 22 more sections

Figures (3)

  • Figure 1: PANDA aims to enhance domain-specific capability of LLMs, which possess superior general capability, by learning from domain expert models that have inferior general capability with a tuning-free way. While conventional knowledge distillation usually leverage superior models to teach inferior models via gradient-based methods. The direction of the arrow represents the direction of knowledge transfer.
  • Figure 2: PANDA consists of two main stages: (a) the learning stage acquires insights from expert preferences and forms an insight pool; (b) the inference stage retrieves relevant insights from the insight pool and perform preference adaptation via in-context learning.
  • Figure 3: Ablation results on sentiment classification. For ablation study, we replace the retrieved insights with the corresponding few-shot examples ("w/ Ablation"), whose label is provided by the expert model.