VIBES: Exploring Viewer Spatial Interactions as Direct Input for Livestreamed Content

TL;DR

This paper investigates whether viewers can directly influence livestreamed content through spatially mapped mouse interactions, addressing the gap beyond text-based chat. It introduces VIBES, a lightweight technology probe that captures viewer mouse events, augments them with Twitch extension data, and delivers bidirectional feedback to streamer applications while compensating for broadcast latency with a camera-state buffer. Through formative streamer studies and two application deployments (Terraria Interaction Mod and Storm the Village), the work demonstrates that spatial input can enhance engagement and co-creation, though it raises concerns about latency, scalability, and streamer agency. The findings inform design guidelines and future work on scalable, privacy-aware, platform-integrated spatial-input systems for livestreaming with potential benefits for audience participation and prosocial engagement.

Abstract

Livestreaming has rapidly become a popular online pastime, with real-time interaction between streamer and viewer being a key motivating feature. However, viewers have traditionally had limited opportunity to directly influence the streamed content; even when such interactions are possible, it has been reliant on text-based chat. We investigate the potential of spatial interaction on the livestreamed video content as a form of direct, real-time input for livestreamed applications. We developed VIBES, a flexible digital system that registers viewers' mouse interactions on the streamed video, i.e., clicks or movements, and transmits it directly into the streamed application. We used VIBES as a technology probe; first designing possible demonstrative interactions and using these interactions to explore streamers' perception of viewer influence and possible challenges and opportunities. We then deployed applications built using VIBES in two livestreams to explore its effects on audience engagement and investigate their relationships with the stream, the streamer, and fellow audience members. The use of spatial interactions enhances engagement and participation and opens up new avenues for both streamer-viewer and viewer-viewer participation. We contextualize our findings around a broader understanding of motivations and engagement in livestreaming, and we propose design guidelines and extensions for future research.

Figures (10)

• Figure 1: Simplified architecture diagram of VIBES. The browser extension captures a viewer's mouse input; the Twitch extension provides added expressiveness to the spatial data. This data is passed to the websocket server, which transmits it to the streamer's application. The bi-directional nature of the architecture allows the application to send data back to influence the Twitch extension.
• Figure 2: The left is an application for drawing on a shared canvas. In (a), the viewers use the Twitch extension to select colour, undo, or delete. In (b), the user uses their mouse to draw the letters "w" and "o". Although small, note the spinner circles used to denote latency. In (c), the user's drawing has been passed to the Unity application, where it renders the line with the correct colour onto the application. The right is an interaction for spawning an enemy into a game. In (d) viewers can select which enemy or debuff to apply. In (e), viewers click into the game where they want the enemy to spawn and in (f), the viewers see the enemy spawned.
• Figure 3: The left interaction surveys clicks. (a) illustrates the initial scene, where viewers are prompted to vote on a move. In (b), the viewer clicks to vote for their selection (the middle-left square); in (c), the number at this square increments and the username of the viewer is added to a list on the right of the stream. The right interaction surveys gestures. (d) illustrates an initial scene, where a different viewer has already "voted" for a force. In (e), the viewer drags from the ball to add another force, which is then represented as a line in (f). Once the timer has completed, the average of the forces is applied, as seen in (g).
• Figure 4: VIBES can be used to input mouse gestures as control. For example, we developed a simple arrow gesture decoder that can be used to control the streamer's Spotify application.
• Figure 5: VIBES can be used to annotate on the streamer's screen. On the stream, we see a heart, checkmark, and star icon that has been spawned in. This tool can also be used to spawn textboxes with viewer-specified text or to freehand draw by the viewers.