PixelGen: Rethinking Embedded Camera Systems

TL;DR

PixelGen rethinks embedded camera systems by combining a low-power multimodal sensor array with edge-driven LLMs and diffusion-based image synthesis to produce high-resolution representations from low-bit, low-res inputs. It expands the operational envelope of embedded cameras to visualize phenomena beyond visible light, such as acoustic emissions, and to support mixed-reality visualization. The system components—PixelSense hardware, edge prompting workflow, and diffusion-based reconstruction—achieve significant bandwidth and energy savings while enabling novel applications in robotics and AR/VR. Overall, PixelGen demonstrates a scalable pathway to long-endurance, richly informative imaging beyond traditional cameras.

Abstract

Embedded camera systems are ubiquitous, representing the most widely deployed example of a wireless embedded system. They capture a representation of the world - the surroundings illuminated by visible or infrared light. Despite their widespread usage, the architecture of embedded camera systems has remained unchanged, which leads to limitations. They visualize only a tiny portion of the world. Additionally, they are energy-intensive, leading to limited battery lifespan. We present PixelGen, which re-imagines embedded camera systems. Specifically, PixelGen combines sensors, transceivers, and low-resolution image and infrared vision sensors to capture a broader world representation. They are deliberately chosen for their simplicity, low bitrate, and power consumption, culminating in an energy-efficient platform. We show that despite the simplicity, the captured data can be processed using transformer-based image and language models to generate novel representations of the environment. For example, we demonstrate that it can allow the generation of high-definition images, while the camera utilises low-power, low-resolution monochrome cameras. Furthermore, the capabilities of PixelGen extend beyond traditional photography, enabling visualization of phenomena invisible to conventional cameras, such as sound waves. PixelGen can enable numerous novel applications, and we demonstrate that it enables unique visualization of the surroundings that are then projected on extended reality headsets. We believe, PixelGen goes beyond conventional cameras and opens new avenues for research and photography.

Figures (13)

• Figure 1: (b) Xreal Air-2 extended reality headset, which we leverage to visualize the acoustic emissions. (a) and (c) exhibit the views from inside the AR glasses, where the user can see the acoustic emission with their eyes. The images are generated using PixelGen. PixelGen can facilitate visualization of otherwise invisible sensor streams with applications to extended reality headsets and beyond.
• Figure 3: Comparison of ECS architectures with PixelGen. Conventional architecture capture environmental representations using image sensors that track areas illuminated by visible light. As a result, they do not capture other fields, such as ambient radio waves or vibrations. In contrast, the PixelGen architecture employs diverse sensors, including a low-resolution image sensor and radio transceivers, to capture various fields, phenomena, and emissions. Utilizing an LLM, it generates appropriate natural language prompts based on captured data and user input. These prompts are then employed with an image model to generate novel representations of the environment. These representations can also visualize fields, phenomena and emissions that a conventional ECS fail to capture.
• Figure 4: PixelSense is composed of sensors, microcontroller and transceivers, possibly supplemented with image sensors. Its primary function is to gather diverse environmental data. The system is designed for energy efficiency, potentially using a backscatter mechanism to enable low-power communication and operation on harvested energy. A key feature of PixelSense is its use of a diffusion model to reconstruct images from the collected sensor data. Unlike conventional embedded cameras, PixelSense may not even sport a image sensor on the platform.
• Figure 5: PixelSense platform - Custom board that enable PixelGen to generate an image from a low-power camera and an array of sensor data.
• Figure 6: The edge computer facilitates interaction between the end-user and the PixelGen. It receives inputs from the end-user through natural language prompts, providing instructions on utilizing the collected sensor data. These prompts are paired with sensor data and fed into a language model, which generates specific prompts for the diffusion model. Using the image, sensor data, and generated prompts, the diffusion model processes this information to create a rich representation of the physical environment. As an example, the end-user may want to provide prompts to generate a high-resolution image even though PixelSense only captured low-resolution image.