Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DEEM: Dynamic Experienced Expert Modeling for Stance Detection

Xiaolong Wang, Yile Wang, Sijie Cheng, Peng Li, Yang Liu

TL;DR

DEEM addresses stance detection by moving beyond fully generated or fixed-role multi-agent reasoning to a semi-parametric framework that dynamically retrieves experienced experts. It generates a large pool of candidate experts from training data, filters them by occurrence and accuracy, and uses a sentence-expert repository to retrieve the most relevant experts for each new sentence, guiding the LLM’s final decision. Across three benchmark datasets and two LLMs, DEEM achieves superior performance and demonstrates reduced bias compared with self-consistency and other expert-based methods. The approach leverages domain knowledge distilled from data and a retrieval mechanism to maintain robustness and generalization while mitigating hallucinations. This semi-parametric, expert-driven reasoning has practical implications for improving trustworthy stance analysis in social media and related NLP tasks.

Abstract

Recent work has made a preliminary attempt to use large language models (LLMs) to solve the stance detection task, showing promising results. However, considering that stance detection usually requires detailed background knowledge, the vanilla reasoning method may neglect the domain knowledge to make a professional and accurate analysis. Thus, there is still room for improvement of LLMs reasoning, especially in leveraging the generation capability of LLMs to simulate specific experts (i.e., multi-agents) to detect the stance. In this paper, different from existing multi-agent works that require detailed descriptions and use fixed experts, we propose a Dynamic Experienced Expert Modeling (DEEM) method which can leverage the generated experienced experts and let LLMs reason in a semi-parametric way, making the experts more generalizable and reliable. Experimental results demonstrate that DEEM consistently achieves the best results on three standard benchmarks, outperforms methods with self-consistency reasoning, and reduces the bias of LLMs.

DEEM: Dynamic Experienced Expert Modeling for Stance Detection

TL;DR

DEEM addresses stance detection by moving beyond fully generated or fixed-role multi-agent reasoning to a semi-parametric framework that dynamically retrieves experienced experts. It generates a large pool of candidate experts from training data, filters them by occurrence and accuracy, and uses a sentence-expert repository to retrieve the most relevant experts for each new sentence, guiding the LLM’s final decision. Across three benchmark datasets and two LLMs, DEEM achieves superior performance and demonstrates reduced bias compared with self-consistency and other expert-based methods. The approach leverages domain knowledge distilled from data and a retrieval mechanism to maintain robustness and generalization while mitigating hallucinations. This semi-parametric, expert-driven reasoning has practical implications for improving trustworthy stance analysis in social media and related NLP tasks.

Abstract

Recent work has made a preliminary attempt to use large language models (LLMs) to solve the stance detection task, showing promising results. However, considering that stance detection usually requires detailed background knowledge, the vanilla reasoning method may neglect the domain knowledge to make a professional and accurate analysis. Thus, there is still room for improvement of LLMs reasoning, especially in leveraging the generation capability of LLMs to simulate specific experts (i.e., multi-agents) to detect the stance. In this paper, different from existing multi-agent works that require detailed descriptions and use fixed experts, we propose a Dynamic Experienced Expert Modeling (DEEM) method which can leverage the generated experienced experts and let LLMs reason in a semi-parametric way, making the experts more generalizable and reliable. Experimental results demonstrate that DEEM consistently achieves the best results on three standard benchmarks, outperforms methods with self-consistency reasoning, and reduces the bias of LLMs.
Paper Structure (19 sections, 5 equations, 7 figures, 6 tables)

This paper contains 19 sections, 5 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Generating Diverse Experts
  5. Filtering Experienced Experts
  6. Retrieving Related Experts
  7. Comparison with Other Methods
  8. Experiments
  9. Experimental Setups
  10. Main Results
  11. Analyses and Discussion
  12. Expert Distributions
  13. Filtering Strategies
  14. Dynamic Experts vs. Fixed Experts
  15. Impact of Demonstrations, Experts, and Discussion Turns
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Illustration of our DEEM method. Top: Vanilla multi-agent reasoning for stance detection through generation. Bottom: Our method first generates and filters experienced experts by leveraging training data, then retrieves the related ones during reasoning.
  • Figure 2: An example of a few-shot prompt with expert modeling for stance detection. The underlined parts are the sentence, target, and label, respectively. The green texts indicate the manually written experts.
  • Figure 3: The overall pipeline of our DEEM method. We first use the prompt and training sentences to generate diverse experts ($\S$\ref{['section:31']}). Then we filter experienced experts according to their occurrence numbers and performance ($\S$\ref{['section:32']}). Finally, we build the sentence-expert pairs and retrieve the experienced experts for each new sentence ($\S$\ref{['section:33']}).
  • Figure 4: Impact of filtering strategies according to accuracy (Top) and frequency (Bottom).
  • Figure 5: Impact of (a) demonstrations, (b) retrieved experts, and (c) discussion turns during reasoning.
  • ...and 2 more figures