Simplify, Consolidate, Intervene: Facilitating Institutional Support with Mental Models of Learning Management System Use

TL;DR

This work introduces Depth of Use (DOU), an ordinal, vendor-agnostic metric to quantify learning management system engagement across tens of thousands of courses. By analyzing Virginia Tech’s Canvas metadata from 2021–2023, the authors link DOU to course modality, participation, outcomes, and logistics, revealing drivers such as scale, interoperability, and ubiquitous access for LMS adoption. The study demonstrates DOU’s utility for identifying low-adoption cohorts, informing resource allocation, professional development, and LMS evangelism, and it documents a practical response to the COVID-19 shift to remote teaching. Overall, DOU provides a scalable, cross-stakeholder framework to measure LMS use, test policy-relevant claims, and guide institutional decisions about teaching and technology investments.

Abstract

Measuring instructors' adoption of learning management system (LMS) tools is a critical first step in evaluating the efficacy of online teaching and learning at scale. Existing models for LMS adoption are often qualitative, learner-centered, and difficult to leverage towards institutional support. We propose depth-of-use (DOU): an intuitive measurement model for faculty's utilization of a university-wide LMS and their needs for institutional support. We hypothesis-test the relationship between DOU and course attributes like modality, participation, logistics, and outcomes. In a large-scale analysis of metadata from 30000+ courses offered at Virginia Tech over two years, we find that a pervasive need for scale, interoperability and ubiquitous access drives LMS adoption by university instructors. We then demonstrate how DOU can help faculty members identify the opportunity-cost of transition from legacy apps to LMS tools. We also describe how DOU can help instructional designers and IT organizational leadership evaluate the impact of their support allocation, faculty development and LMS evangelism initiatives.

Figures (6)

• Figure 1: Study methodology overview (clockwise from top left): DOU Estimation (data sources and methods), Contextual Inquiry (hypothesis variables and methods), key user groups, and measurement objectives fulfilled by DOU. (Top-left) We estimate a DOU score of overall LMS use - low, medium or high - for each course in our analysis. We then test claims (top-right) about how DOU is linked to course modality, participation, logistics, and outcomes. Finally, we discuss DOU use-cases (bottom-left and bottom-right) for faculty, instructional designers, and LMS administrators.
• Figure 2: Brainstorming session aids for Canvas depth-of-use development: three early, data-driven attempts to capture Canvas use from page request logs ((a)-(c)), and a preliminary model-driven summary of Canvas usage tiers (d).
• Figure 3: A schematic (left), and descriptions of all steps (right) involved in the calculation of course-level DOU. S, F, D, etc. refer to LMS resource labels in the taxonomy in table \ref{['tab_taxonomy']}. These resource-level DOUs are aggregated into the overall DOU ranking (low, medium, or high) for the course.
• Figure 4: A simple illustration of how two LMS resource DOUs are paired using equation \ref{['main_dou_equation']}. In this example, assignment delivery DOU is 1 (link to DOC, ZIP or app), and assignment submission DOU is 0 (no file upload, likely paper or app). (Left) Setting $\beta$ to 0 computes the MAX of the two resource DOUs, which is useful when only one is needed for overall DOU. (Right) Setting $\beta$ to 1 computes the floored average, which is useful when both need to contribute to DOU equally.
• Figure 5: Third-party app use at Virginia Tech by frequency of use, overall (right) and in the Computer Science department (left). Discussion forums and course content management (Piazza), and programming instruction (OpenDSA) combined account for nearly half of all third-party app use for the latter. Significant, often LMS-unaccounted use of third-party apps makes it challenging for department and IT leadership to evaluate and scale pedagogies.