QANA: LLM-based Question Generation and Network Analysis for Zero-shot Key Point Analysis and Beyond

TL;DR

QANA tackles information overload in social media by replacing traditional summaries with a Question-Answering Network Analysis framework. It converts each argument into multiple questions via LLMs, builds a QA network where edges reflect cosine-based embedding similarity between arguments and questions, and then uses centrality measures to identify important questions as key points. In zero-shot KPM, QANA achieves competitive performance with supervised models and reduces matching complexity from quadratic to linear; in KPG, high-centrality questions align with manually crafted key points, enabling multi-perspective analysis. The framework enhances impartiality and fairness in opinion mining by letting analysts choose centrality notions and question styles to surface diverse, important viewpoints.

Abstract

The proliferation of social media has led to information overload and increased interest in opinion mining. We propose "Question-Answering Network Analysis" (QANA), a novel opinion mining framework that utilizes Large Language Models (LLMs) to generate questions from users' comments, constructs a bipartite graph based on the comments' answerability to the questions, and applies centrality measures to examine the importance of opinions. We investigate the impact of question generation styles, LLM selections, and the choice of embedding model on the quality of the constructed QA networks by comparing them with annotated Key Point Analysis datasets. QANA achieves comparable performance to previous state-of-the-art supervised models in a zero-shot manner for Key Point Matching task, also reducing the computational cost from quadratic to linear. For Key Point Generation, questions with high PageRank or degree centrality align well with manually annotated key points. Notably, QANA enables analysts to assess the importance of key points from various aspects according to their selection of centrality measure. QANA's primary contribution lies in its flexibility to extract key points from a wide range of perspectives, which enhances the quality and impartiality of opinion mining.

Figures (4)

• Figure 1: Overview of the proposed Question-Answering Network Analysis (QANA) framework for key point analysis. The methodology involves generating questions from comments using Large Language Models (LLMs), constructing a Question-Answering (QA) network based on the answerability of the questions by the arguments, and extracting important question nodes as key points using centrality measures.
• Figure 2: Example of argument-to-question transformation using GPT-4 for the topic of mandatory child vaccinations. The prompt template provides guidelines for generating closed-ended questions that capture similar or opposing viewpoints to the given argument. The GPT-4 output successfully extracts multiple perspectives from a single comment, creating a series of questions that represent diverse stances on the topic.
• Figure 3: Comparison of different question types, language models, and embedding models in terms of their impact on the quality of the Question-Answering (QA) network in the Key Point Matching (KPM) setting. The graph shows the strict mAP scores for various configurations, with the combination of "text-embedding-3-large" embedding and closed questions generated by GPT-4 achieving the highest performance, comparable to state-of-the-art supervised models.
• Figure 4: Coverage of human-generated key points by top questions with high centrality scores, comparing different question types and centrality metrics across various model configurations. The graph illustrates the coverage of human-generated key points as the number of top questions considered increases. The combination of GPT-4 for question generation, the "text-embedding-3-large" model for embedding, and PageRank as the centrality metric achieves the highest coverage, with 85% of key points covered by the top 10 questions.