SnapNCode: An Integrated Development Environment for Programming Physical Objects Interactions
Xiaoyan Wei, Zijian Yue, Hsiang-Ting Chen
TL;DR
SnapNCode addresses the perceptual gap in spatial computing by representing physical object states as images within code, while preserving a text-based core for compatibility with existing workflows. The approach combines a CodeMirror-based frontend, a Flask backend, and YOLOv8-based object detection to capture, display, and trigger code tied to physical objects via live video streams. A usability study with $N=12$ programmers demonstrates generally positive acceptance, highlighting ease of use and potential workflow integration, alongside concrete feedback on features and collaboration. The work suggests a viable path toward more intuitive spatial programming tools and outlines clear directions for improving detection, 3D reasoning, and integration with broader IDE ecosystems.
Abstract
Spatial computing technologies have the potential to revolutionize how we interact with the world around us. However, most modern integrated development environments (IDEs) have not fully adapted to this paradigm shift. For example, physical 3D objects in the real world are still represented as 2D text variables in code, creating a significant perceptual distance between these representations. In response to this challenge, we introduce SnapNCode, a novel IDE for spatial programming. SnapNCode enables programmers to capture various states of physical objects through live video streams from cameras and directly insert these visual representations into their code. Moreover, users can augment physical objects by attaching code snippets onto objects, which are opportunistically triggered when observed by cameras. We conducted a user study (N=12) to assess the usability of SnapNCode. Feedback from participants indicates that the system is easy-to-use and holds promise for daily casual uses and integration into a broader range of workflows.