Measuring the State of Open Science in Transportation Using Large Language Models

TL;DR

This study develops an automated, scalable pipeline that uses Large Language Models to measure data and code availability in Transportation Research journals, validating against manual annotations and applying it to over 10k papers from 2019–2024. It finds low openness (about 5% code, 4% data) with no clear citation or review-time incentives, implying a need for policy-driven interventions by journals and funders. The analysis reveals geographic, topical, and journal-based patterns in openness and highlights an incentive gap that may hinder broader adoption. The authors provide an interactive explorer and a scalable framework that can be extended to other journals, enabling continuous monitoring of open science practices in transportation research.

Abstract

Open science initiatives have strengthened scientific integrity and accelerated research progress across many fields, but the state of their practice within transportation research remains under-investigated. Key features of open science, defined here as data and code availability, are difficult to extract due to the inherent complexity of the field. Previous work has either been limited to small-scale studies due to the labor-intensive nature of manual analysis or has relied on large-scale bibliometric approaches that sacrifice contextual richness. This paper introduces an automatic and scalable feature-extraction pipeline to measure data and code availability in transportation research. We employ Large Language Models (LLMs) for this task and validate their performance against a manually curated dataset and through an inter-rater agreement analysis. We applied this pipeline to examine 10,724 research articles published in the Transportation Research Part series of journals between 2019 and 2024. Our analysis found that only 5% of quantitative papers shared a code repository, 4% of quantitative papers shared a data repository, and about 3% of papers shared both, with trends differing across journals, topics, and geographic regions. We found no significant difference in citation counts or review duration between papers that provided data and code and those that did not, suggesting a misalignment between open science efforts and traditional academic metrics. Consequently, encouraging these practices will likely require structural interventions from journals and funding agencies to supplement the lack of direct author incentives. The pipeline developed in this study can be readily scaled to other journals, representing a critical step toward the automated measurement and monitoring of open science practices in transportation research.

Figures (6)

• Figure 1: Features considered for data and code availabilities.
• Figure 2: Number of articles published in Transportation Research journals (TR-A, TR-B, TR-C, TR-D, TR-E, TR-F, and TR-IP) from 2019 to 2024 ($N=10,724$).
• Figure 3: Overview of the features extracted from our pipeline
• Figure 4: Citations per year of each paper over time by code and data-sharing practice, with a LOESS moving average line applied for mean comparison.
• Figure 5: Flowchart of the postprocessing pipeline. Starting from raw XML metadata, LLM-extracted JSON features, and human annotations, the pipeline (i) parses and flattens inputs into paper- and dataset-level tables, (ii) normalizes types and encodings, (iii) canonizes and classifies URLs into code and data host categories, and (iv) aggregates and validates features to produce the final analysis-ready dataset.