Objestures: Everyday Objects Meet Mid-Air Gestures for Expressive Interaction

Abstract

Everyday object-based interactions (EOIs) and mid-air gesture interactions (MAIs) have been widely explored, yet prior work on their integration often targets narrow use cases or specific technologies, leaving designers and developers with limited guidance that generalizes across diverse EOIs and MAIs. We introduce Objestures ("Obj" + "Gestures") -- five interaction types spanning EOIs and MAIs, forming a design space for expressive uni- and bimanual interaction. To evaluate the usefulness of Objestures, we conducted an exploratory user study (N=12) on basic 3D tasks (rotation and scaling), which showed performance comparable to the headset's native freehand manipulation. To understand the user experience, we conducted case studies with the same participants across three applications (Sound, Draw, and Shadow), where participants found the interactions intuitive, engaging, and expressive, and indicated interest in everyday use. We further demonstrate the potential of Objestures across diverse contexts through 30 examples, and discuss limitations and implications. https://www.zhuoyuelyu.com/objestures

Figures (8)

• Figure 1: Formation of the five interaction types (Binary, Linear, Rotational, Nonlinear, and Free), as well as the design space. Follow ❶--❺, then read upward for definitions, examples, and functions. Continue with ❻--❼, and finally $\bullet$i . EOI = everyday object-based interaction; MAI = mid-air gesture interaction. Details in \ref{['sec:objestures']}.
• Figure 2: Overview of study setup and conditions. The image on the left shows the study environment. The sub-figures (A1--C4) on the right illustrate the three Approaches (rows: Obj, NObj, Hands) across four Task $\times$ Distance conditions (columns: Near-Scaling, Near-Rotation, Far-Scaling, Far-Rotation). Each sub-figure shows the moment immediately before the cube was pinch-selected to initiate manipulation. Users manipulate the cubes to match the target angle or scale shown in the white wireframe. Refer to \ref{['sec:des']} and the accompanying video for details.
• Figure 3: Plots of performance metrics (Error, Movement, Time) for Scaling and Rotation Tasks across different Approaches (Hands, Obj, NObj) and Distances (Near, Far). Error bars represent standard errors. Significant differences (main effects of Approach only) are indicated by * (p < 0.05), ** (p < 0.01), *** (p < 0.001), and **** (p < 0.0001); pairwise differences at individual Distance levels are not implied. See \ref{['sec:performance']} for details.
• Figure 4: Median preference ranks across three Approaches rated by all participants (lower values indicate higher preference). The error bars represent the interquartile range (IQR).
• Figure 5: Stacked bar plots showing user responses across three examples (\ref{['fig:teaser-full']}A--C) on seven Likert items, color-coded from strongly disagree to strongly agree, with median values displayed to the right of the bars. Percentages for segments $\geq$ 4 users (33 %) are indicated for clarity. All applications had median ratings $\geq$ 5.5 on all items.