CAUS: A Dataset for Question Generation based on Human Cognition Leveraging Large Language Models

TL;DR

This work introduces the CAUS dataset to study question generation by Large Language Models under epistemic uncertainty, aiming to mimic human cognitive questioning. It combines scene Description generation with reasoning and five-question querying, followed by two-dimensional K-type and Q-type classifications, validated against human ground truth. The GPT-4-0613 driven pipeline demonstrates that LLMs can identify uncertainties and generate diverse, relevant questions with high classification reliability. The findings suggest that incorporating human-like questioning strategies can enhance AI's handling of uncertainty and point to model-agnostic prompting and future work incorporating social-pragmatic factors.

Abstract

We introduce the Curious About Uncertain Scene (CAUS) dataset, designed to enable Large Language Models, specifically GPT-4, to emulate human cognitive processes for resolving uncertainties. Leveraging this dataset, we investigate the potential of LLMs to engage in questioning effectively. Our approach involves providing scene descriptions embedded with uncertainties to stimulate the generation of reasoning and queries. The queries are then classified according to multi-dimensional criteria. All procedures are facilitated by a collaborative system involving both LLMs and human researchers. Our results demonstrate that GPT-4 can effectively generate pertinent questions and grasp their nuances, particularly when given appropriate context and instructions. The study suggests that incorporating human-like questioning into AI models improves their ability to manage uncertainties, paving the way for future advancements in Artificial Intelligence (AI).

Figures (4)

• Figure 1: The main concept of the CAUS dataset aligned with human cognition. The uncertain Scene Description provides a context that causes "epistemic curiosity." The Reasoning sentences point out the uncertain point in a given scene. And the Questions represent efforts to resolve the uncertainty.
• Figure 2: The pipeline for creating a dataset involves collecting, classifying, and evaluating potential questions evoked from scene descriptions. Symbols in the bottom box represent frameworks used in the pipeline.
• Figure 3: Question classification results. Each row indicates the scene class (i.e., (Top)Object Uncertainty. (Middle)Intention Uncertainty. (Bottom)Event Uncertainty.). Each column shows the generation order of the question. In each plot, the position of the bars corresponds to the question category, coordinated by k-type and q-type. The height of the bars indicates the frequency of the question for that type pair.
• Figure 4: Denoting the question space for displaying question classification