MaRginalia: Enabling In-person Lecture Capturing and Note-taking Through Mixed Reality

TL;DR

MaRginalia presents a novel MR- and tablet-based note-taking system for in-person lectures that enables dual-channel capture: handwritten notes on a tablet and MR-based captures of lecture slides and transcripts. By leveraging a Gaze+Pen interaction model and spatial panels, the system promotes engagement with the lecture while facilitating rapid capture and organization of content, including a navigable lecture history. An exploratory user study (n=12) reports good usability (SUS ≈ 77) and positive feedback on the Slides and Transcripts Panels, with participants noting flexible workflows and time savings, though some found indirect Gaze+Pen input unfamiliar. The work demonstrates MR's potential to augment note-taking in education and outlines directions for larger-scale deployments, dynamic content capture, and multi-speaker contexts.

Abstract

Students often take digital notes during live lectures, but current methods can be slow when capturing information from lecture slides or the instructor's speech, and require them to focus on their devices, leading to distractions and missing important details. This paper explores supporting live lecture note-taking with mixed reality (MR) to quickly capture lecture information and take notes while staying engaged with the lecture. A survey and interviews with university students revealed common note-taking behaviors and challenges to inform the design. We present MaRginalia to provide digital note-taking with a stylus tablet and MR headset. Students can take notes with an MR representation of the tablet, lecture slides, and audio transcript without looking down at their device. When preferred, students can also perform detailed interactions by looking at the physical tablet. We demonstrate the feasibility and usefulness of MaRginalia and MR-based note-taking in a user study with 12 students.

Figures (14)

• Figure 1: Survey results on digital note-taking with 45 university students. (a) The frequency of overall digital note-taking and specific digital note-taking methods. (b) Subjective ratings on statements on respondents' digital note-taking satisfaction, utility, and issues based on their existing digital note-taking methods.
• Figure 2: Tablet interface. A: User notes on the canvas. B-C: User captured snapshots of lecture slide (B) and transcript (C). D: Drawing tool palette, including pen, highlighter, and stroke eraser. E: Navigator buttons to scroll between captures.
• Figure 3: Note-taking on the tablet (A-B) and the spatial Notes Panel (C-E). A: Looking at the tablet enables direct handwriting. B: Looking forward, the tablet becomes an indirect input. C: View notes on the Notes Panel. D: Selecting the scroll button navigates between captures. E: Taking notes on the Notes Panel. Notes are synced between the Notes Panel and the tablet.
• Figure 4: Gaze+Pen creating strokes with spatial panels. A: The pen is idle when far from the tablet. The system continuously detects the user's gaze position. B: Bringing the pen closer to the tablet reveals the cursor. C: Moving the pen while hovering adjusts the cursor position. D: Touching the tablet with the pen engages in an interaction. In this case, begin drawing. E: Moving the pen while touching the tablet draws a stroke. F: Lifting the pen into a hover state completes a stroke and enables cursor position adjustment. G: Lifting the pen resets the system to an idle state and hides the cursor.
• Figure 5: Cursor icons. The cursor hides when idle and dynamically reflects drawing tools and button selection states during hover and pen-down.