Rethinking Search: A Study of University Students' Perspectives on Using LLMs and Traditional Search Engines in Academic Problem Solving

TL;DR

The paper investigates how university students use LLMs and traditional search engines for academic problem solving, revealing a complementary pattern: LLMs enable fast summaries and drafting, while search engines provide credible, source-rich verification. Using a mixed-methods design (n=$109$ survey participants and $n=12$ interviews), the study shows a clear overall preference for LLMs across usability dimensions, yet demonstrates that a balanced, hybrid approach yields the highest task accuracy ($ ext{≈}90 ext{%}$) at the cost of longer task durations. These findings motivate a contextual, integrated tool that embeds GPT-like generation within the search interface to reduce cognitive load while preserving source transparency through embedded links. The proposed prototype aims to bridge generation and verification, with implications for design of AI-assisted academic workflows and future work expanding participant diversity and validating an implemented system in real settings.

Abstract

With the increasing integration of Artificial Intelligence (AI) in academic problem solving, university students frequently alternate between traditional search engines like Google and large language models (LLMs) for information retrieval. This study explores students' perceptions of both tools, emphasizing usability, efficiency, and their integration into academic workflows. Employing a mixed-methods approach, we surveyed 109 students from diverse disciplines and conducted in-depth interviews with 12 participants. Quantitative analyses, including ANOVA and chi-square tests, were used to assess differences in efficiency, satisfaction, and tool preference. Qualitative insights revealed that students commonly switch between GPT and Google: using Google for credible, multi-source information and GPT for summarization, explanation, and drafting. While neither tool proved sufficient on its own, there was a strong demand for a hybrid solution. In response, we developed a prototype, a chatbot embedded within the search interface, that combines GPT's conversational capabilities with Google's reliability to enhance academic research and reduce cognitive load.

Figures (3)

• Figure 1: Overview of the study methodology. A mixed-methods approach was employed in this study. (1) The survey phase $(n = 109)$ captured quantitative data and analyzed using different statistical tests. (2) The qualitative phase included in-person interviews $(n = 12)$, where participants completed six academic tasks and were grouped based on tool usage. Thematic analysis of open-ended responses and interview transcripts led to four themes.
• Figure 2: Boxplot of Quantitative Features: This figure presents a comparative analysis of key usability factors between traditional search engines and LLM-based tools. The top four features—Search_Use_Frequency, Search_Satisfaction, Search_Efficiency, and Search_Ease—represent user responses related to traditional search engines. The bottom four—LLM_Use_Frequency, LLM_Satisfaction, LLM_Efficiency, and LLM_Ease—correspond to user experiences with large language models.
• Figure 3: Class Distribution of the Preferred Tool among the Students