LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

TL;DR

The first large-scale survey of 816 verified research article authors is presented, finding that traditionally disadvantaged groups in academia (non-White, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity.

Abstract

The rise of large language models (LLMs) has led many researchers to consider their usage for scientific work. Some have found benefits using LLMs to augment or automate aspects of their research pipeline, while others have urged caution due to risks and ethical concerns. Yet little work has sought to quantify and characterize how researchers use LLMs and why. We present the first large-scale survey of 816 verified research article authors to understand how the research community leverages and perceives LLMs as research tools. We examine participants' self-reported LLM usage, finding that 81% of researchers have already incorporated LLMs into different aspects of their research workflow. We also find that traditionally disadvantaged groups in academia (non-White, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity. However, women, non-binary, and senior researchers have greater ethical concerns, potentially hindering adoption.

Figures (5)

• Figure 1: An example of data we collected per participant from sections of the survey that relate to our statistical modeling. Each participant provides answers to (up to) five demographic questions and (up to) 36 Likert ratings in response to questions about LLM usage frequency, perceptions, and usage types. Participants are not required to report on every question. In this example, the gender information is missing from the participant.
• Figure 2: Overview of Usage Frequency Divided by LLM Usage Type (N=816). The left diverging bar chart displays the distribution of usage frequency across different types of LLM usage, with each type represented by a separate row. The frequency levels, from left to right, are: Very Rarely, Rarely, Occasionally, Frequently, and Very Frequently, with the midpoint of the chart centered at "Occasionally." The grey bar chart on the right indicates the percentage of responses that report "Never" using LLMs for each corresponding type. From this plot, we can tell that researchers report using LLMs for Information Seeking and Editing most frequently, and for Data Cleaning & Analysis and Data Generation the least frequently.
• Figure 3: LLM Usage Breakdown Under Each Usage Type (N=816). Each bar chart shows the number of participants who reported using LLMs for tasks that were subcategories of each usage type. Every participant could select multiple subcategories. The total number next to each title shows the number of participants who indicated using LLMs for the broad usage type.
• Figure 4: Overview of our survey results ($N$ is shown in Table \ref{['tab:data-overview']}), broken apart by demographic characteristics. Each heat map square represents the average rating of this demographic group on the usage frequency or perception for the particular type of LLM usage. The stars (***) are the significance levels of the differences indicated by p values from regression results. The brackets on the left indicate this difference is significant across all types of usage whereas the lines between squares indicate this difference is only significant for certain types of usage.
• Figure 5: Overview of the relation between usage frequency and perceptions of LLM. Each heatmap represents one type of perception, and each cell represents the number of responses (log scaled) that fall under this level of frequency of perception. The Kendall's tau coefficient on the bottom indicates how strong the correlation is between the usage frequency and the perception of that usage. All perceptions are significantly correlated with usage ($p$ < .0001). Tests performed using cor.test in R and corrected with p.adjust.