Can Theory-Informed Message Framing Drive Honest and Motivated Performance with Better Assessment Experiences in a Remote Assessment?

TL;DR

The paper tackles cheating in remote, unproctored assessments by developing theory-grounded, motivational messages drawn from self-determination theory, cognitive dissonance, social norms, and self-efficacy. Through a two-part design—an expert workshop generating 45 messages across 15 concepts and a large online study using an incentivized anagram task (N=1232)—the authors show that concept-based messages reduce full-cheating by 42% without harming performance or user experience, and that effects are broadly similar across concepts. Analyses of self-reported mechanisms reveal modest, multi-mechanism processing, with mechanisms predicting outcomes only modestly and no single mechanism dominating discrimination between integrity groups. The findings advocate for motivational framing over deterrence and demonstrate that integrity interventions can foster honest, motivated performance while preserving assessment quality, informing scalable, theory-grounded practices for remote assessments.

Abstract

Remote unproctored assessments increasingly use messaging interventions to reduce cheating, but existing approaches lack theoretical grounding, focus narrowly on cheating suppression while overlooking performance and experience, and treat cheating as binary rather than continuous. This study examines whether messages based on 15 psychological concepts from self-determination, cognitive dissonance, social norms, and self-efficacy theories can reduce cheating while preserving performance and experience. Through an expert workshop (N=5), we developed 45 theory-informed messages and tested them with online participants (N=1232) who completed an incentivized anagram task. Participants were classified as non-cheaters (0% items cheated), partial-cheaters (1-99% cheated), or full-cheaters (100% cheated). Results show that concept-based messages reduced full-cheating occurrence by 42% (33% to 19%), increased non-cheating by 19% (53% to 63%), with no negative effects on performance or experience across integrity groups. Surprisingly, messages grounded in different theoretical concepts produced virtually identical effects. Analyses of self-rated psychological mechanisms revealed that messages influenced multiple mechanisms simultaneously rather than their intended targets, though these mechanisms predicted behavior, performance, and experience. These findings show that causal pathways are more complex than current theories predict. Practically, integrity interventions using supportive motivation rather than rule enforcement can reduce cheating without impairing performance or experience.

Figures (12)

• Figure 1: Sequential study flow for evaluating concept-based messaging interventions in a remote unproctored anagram challenge. Participants rated their ability to solve anagram (2), practiced the task (3), viewed a theory-based message to motivate honest performance (4), completed two 2-minute rounds (5) where longer and more numerous words earned higher rewards, then filled out a post-task survey (6) and were debriefed, including disclosure of self-reported cheating (7).
• Figure 2: Screenshots of the online study interface. Left: Message interventions displayed for a mandatory 10-second period. Right: Participants created words using drag-and-drop interactions.
• Figure 3: Methodology for operationalizing concepts into messages. Experts selected a concept via an in-house web app and generated messages with assistance from a tailored LLM primed with concept definitions and examples. Messages were iteratively refined via an evaluator-optimizer workflow madaan2023self, with expert feedback incorporated into subsequent LLM-assisted revisions.
• Figure 4: Comprehensive measurement framework for evaluating psychological concept-based messages in remote unproctored assessments. Three measurement phases: Phase 1 captures baseline perceived ability before message exposure (1); Phase 2 tracks behavioral outcomes during and after the task (cheating behavior (2), performance (3), experience (4)); Phase 3 assesses self-rated psychological states or mechanisms (5), supplemented by qualitative open-ended questions (6).
• Figure 5: Multi-phase cheating detection for remote anagram challenge. The algorithm processes logged user interactions through three main phases: (1) baseline speed establishment, (2) identification of suspicious event sequences, and (3) multi-rule evaluation of flagged behaviors. An additional self-report validation phase was conducted but proved unreliable (only 4.2% of behaviorally flagged participants admitted cheating), confirming the necessity of behavioral detection methods. Complete algorithmic specifications in Appendix.