The Impact of AI Generated Content on Decision Making for Topics Requiring Expertise

TL;DR

This study investigates how domain-specific knowledge and AI-generated content (AIGC) influence online decision-making. Using a lab-based explanatory sequential design with 30 university students across domain-general and domain-specific topics, it compares AI-generated versus human-written decision-support information and analyzes opinion changes with a generalized linear mixed-effects model (GLMM), complemented by semi-structured interviews. The refined GLMM for opinion change achieved $R^2=0.47$ (theoretical) with $ ext{Delta }R^2=0.27$, and the model for confidence changes yielded $R^2_m=0.29$ and $R^2_c=0.44$, indicating meaningful explanatory power of the identified predictors. Key findings show that domain-specific knowledge reduces opinion shifts on domain-specific topics and that AIGC is perceived as comparably helpful to human-written content, highlighting both the potential for AI to bridge knowledge gaps and the need for regulation to mitigate risks in domain-critical decision-making.

Abstract

Modelling users' online decision-making and opinion change is a complex issue that needs to consider users' personal determinants, the nature of the topic and the information retrieval activities. Furthermore, generative-AIbased products like ChatGPT gradually become an essential element for the retrieval of online information. However, the interaction between domainspecific knowledge and AI-generated content during online decision-making is unclear. We conducted a lab-based explanatory sequential study with university students to overcome this research gap. In the experiment, we surveyed participants about a set of general domain topics that are easy to grasp and another set of domain-specific topics that require adequate levels of chemical science knowledge to fully comprehend. We provided participants with decision-supporting information that was either produced using generative AI or collected from selected expert human-written sources to explore the role of AI-generated content compared to ordinary information during decision-making. Our result revealed that participants are less likely to change opinions on domain-specific topics. Since participants without professional knowledge had difficulty performing in-depth and independent reasoning based on the information, they favoured relying on conclusions presented in the provided materials and tended to stick to their initial opinion. Besides, information that is labelled as AI-generated is equivalently helpful as information labelled as dedicatedly human-written for participants in this experiment, indicating the vast potential as well as concerns for AI replacing human experts to help users tackle professional topics or issues.

Figures (2)

• Figure 1: Plots of confidence changes versus opinion changes under different categorical variables. From left to right: (i) educational degree, (ii) whether the topic is in-domain or out-of-domain, (iii) gender, and (iv) whether the topic is under the source-sensitive pattern or not.
• Figure 2: Inspection visualisation of the comparison between the full model (red) and the refined model (blue) on opinion changes. It illustrates the combination of gender plus pattern versus different continuous variables. From left to right then top to bottom: (a) perceived helpfulness of the decision-supporting information, (b) confidence before revealing information sources, (c) perceived familiarity regarding the topic, and (d) the final confidence.