Search in Transition: A Study of University Students Perspectives on Using LLMs and Traditional Search Engines in English Test Problem Solving for Higher Study

TL;DR

The paper investigates how university students solve English language test problems using large language models (LLMs) and traditional search engines, examining usability, efficiency, and task strategies through a mixed-methods design (n=140 survey, n=20 interviews). Quantitative results show LLMs are rated higher across usability dimensions, while qualitative analysis reveals students adopt a hybrid workflow that leverages LLMs for generation and explanation and search engines for verification and credible sourcing. The highest task accuracy emerges from a tool-balancing approach, albeit with longer completion times, underscoring a speed–reliability trade-off. Based on these findings, the authors propose a conceptual hybrid prototype—embedding an LLM assistant within a search interface with explicit source attribution—to reduce cognitive load and promote evidence-based learning in English test preparation.

Abstract

As Artificial Intelligence (AI) becomes increasingly integrated into education, university students preparing for English language tests are frequently shifting between traditional search engines like Google and large language models (LLMs) to assist with problem-solving. This study explores students perceptions of these tools, particularly in terms of usability, efficiency, and how they fit into English test preparation practices. Using a mixed-methods design, we collected survey data from 140 university students across various academic fields and conducted in-depth interviews with 20 participants. Quantitative analyses, including ANOVA and chi-square tests, were applied to assess differences in perceived efficiency, satisfaction, and overall tool preference. The qualitative results reveal that students strategically alternate between GPT and Google based on task requirements. Google is primarily used for accessing reliable, multi-source information and verifying rules, whereas GPT is favored for summarizing content, providing explanations, paraphrasing, and drafting responses for English test tasks. Since neither tool independently satisfies all aspects of English language test preparation, students expressed a clear preference for an integrated approach. In response, this study proposes a prototype chatbot embedded within a search interface, combining GPTs interactive capabilities with Googles credibility to enhance test preparation and reduce cognitive load.

Figures (5)

• Figure 1: Overview of the study methodology. A mixed-methods approach was employed in this study. (1) The survey phase (n = 140) captured quantitative data and was analyzed using different statistical tests. (2) The qualitative phase included in-person interviews (n = 20), where participants completed seven English language test problem tasks and were grouped based on tool usage. Thematic analysis of open-ended responses and interview transcripts led to four themes.
• Figure 2: Boxplot of quantitative features. This figure presents a comparative analysis of key usability dimensions, usage frequency, satisfaction, efficiency, and ease of use between traditional search engines and LLM-based tools in the context of English language test problem-solving for higher studies. The top four features (Search_Use_Frequency, Search_Satisfaction, Search_Efficiency, and Search_Ease) represent participant responses related to traditional search engines, while the bottom four features (LLM_Use_Frequency, LLM_Satisfaction, LLM_Efficiency, and LLM_Ease) correspond to user experiences with large language models.
• Figure 3: Trade-off between task accuracy and completion time across different tool usage strategies in English language test problem solving.
• Figure 4: Distribution of participants’ overall tool preference for English language test problem solving. The majority of students preferred a hybrid approach combining LLM-based tools and traditional search engines, followed by exclusive use of LLMs, while comparatively few relied solely on traditional search engines.
• Figure 5: Conceptual prototype of the proposed hybrid search–LLM interface for English language test preparation. The interface integrates traditional search results with an embedded LLM assistant that provides explanations, summaries, and draft responses, while explicitly linking generated outputs to underlying sources to support verification and reduce cognitive load.