Measuring fidelity of implementation of named active learning methods in physics

TL;DR

This study addresses how faithfully instructors implement named active learning methods in introductory physics using direct COPUS observations and classroom observation networks. It identifies instructional critical components for SCALE-UP, ISLE, and Tutorials, then compares 18 broader implementations to high-fidelity CALEP data, assessing time allocation and temporal patterns via network analysis, and relates fidelity to student conceptual gains using concept inventories. The findings show broad implementations generally deliver the critical components at similar levels as high-fidelity ones, with substantial variation in how activities are sequenced, and no consistent link between fidelity and learning gains, though SCALE-UP often exhibits higher transition fidelity and learning gains. The work demonstrates a robust framework for measuring fidelity in STEM education and suggests that flexibility in how critical components are enacted might still support student understanding, while calling for larger, more diverse samples to refine guidelines for effective implementation across disciplines and contexts.

Abstract

Various active learning methods have been developed for introductory physics, and these methods are increasingly being adopted by instructors. However, instructors often do not implement these methods exactly as was originally intended by the developers, as they may face issues related to funding and institutional support for active learning and/or have different instructional contexts (e.g., student populations) and environments (e.g., physical classroom layouts) than the developers. Existing research does not sufficiently capture the range of variation in instructor implementation of established active learning methods, especially in comparison to high-fidelity implementations. In this study, we first identify the critical components (i.e., components without which the active learning method cannot be said to have been implemented) of three named active learning methods: SCALE-UP, ISLE, and Tutorials. We then evaluate the fidelity with which 18 different introductory physics instructors implement these methods by analyzing classroom observations and comparing the extent to which these broader implementations use each critical component in their classroom to high-fidelity implementations. We find across all three active learning methods that broader implementations spend similar amounts of class time on the critical components as high-fidelity implementations. At the same time, we observe substantial variation in the specific styles that broader implementers operationalize these critical components (e.g., doing a few long activities versus many short activities). Finally, we find no clear relationship between fidelity of implementation and student conceptual learning gains for our study's sample of instructors, providing preliminary evidence that different ways of implementing the critical components of active learning method may all effectively improve student understanding.

Figures (5)

• Figure 1: Types of critical components as defined by Stains and colleagues stains2017fidelity. The outlined black box highlights the focus of this study: instructional critical components.
• Figure 2: The proportion of two-minute time intervals spent on each COPUS code in high-fidelity and broader implementations of each active learning method. The COPUS codes corresponding to critical components are bolded (Table \ref{['criticalcomponents-table']}). The gray boxplots indicate interquartile range, with the bold lines representing the medians and whiskers denoting 1.5 times the interquartile range.
• Figure 3: Cosine similarity values comparing (a) the proportion of class time spent on all COPUS activities and (b) patterns of transitions between all COPUS activities (i.e., using the edge weights of classroom observation networks) in broader implementations to high-fidelity implementations of each active learning method. The gray boxplots indicate interquartile range, with the bold lines representing the medians and whiskers denoting 1.5 times the interquartile range.
• Figure 4: Example classroom observation networks, only considering instructor codes, for (a) the high-fidelity implementation of SCALE-UP and (b) one broader implementation of SCALE-UP with high cosine similarity values for duration fidelity and low cosine similarity values for transition fidelity. Node color represents the fraction of observed two-minute time intervals spent on each code, with darker shades indicating larger fractions. Edges point from an initial code to the code that occurs in the following two-minute time interval of the COPUS observation. Edge width indicates the number of transitions occurring between those two codes normalized by the total number of two-minute time intervals observed. Networks for all courses in this study are available at Ref. github2025.
• Figure 5: Effect sizes of student concept inventory scores versus (a) duration fidelity, measured using the proportion of class time spent on all COPUS codes, and (b) transition fidelity, measured using classroom observation networks (Fig. \ref{['cosine-sim']}). Dots indicate Hedges' g values and error bars indicate 95% confidence intervals.