eNCApsulate: NCA for Precision Diagnosis on Capsule Endoscopes

TL;DR

This work addresses the challenge of heavy data and localization in Wireless Capsule Endoscopy by introducing eNCApsulate, a lean Neural Cellular Automata (NCA) pipeline for on-device bleeding segmentation and monocular depth estimation, distilled from a large foundation model and ported to the ESP32-S3 microcontroller. The approach yields ultra-small models (<70 kB) that deliver competitive segmentation (Dice around 0.58) and convincing depth maps, while achieving substantial on-chip speedups through optimized inference and temporal regularization. Key contributions include the first on-device NCA deployment for capsule endoscopy, a distillation strategy from foundation models, and practical runtime improvements enabling capsule-scale visualization and navigation. The work lays the groundwork for sensor-less capsule localization via visual odometry, reducing transmission needs and enabling precise diagnosis directly on the capsule.

Abstract

Wireless Capsule Endoscopy is a non-invasive imaging method for the entire gastrointestinal tract, and is a pain-free alternative to traditional endoscopy. It generates extensive video data that requires significant review time, and localizing the capsule after ingestion is a challenge. Techniques like bleeding detection and depth estimation can help with localization of pathologies, but deep learning models are typically too large to run directly on the capsule. Neural Cellular Automata (NCA) for bleeding segmentation and depth estimation are trained on capsule endoscopic images. For monocular depth estimation, we distill a large foundation model into the lean NCA architecture, by treating the outputs of the foundation model as pseudo ground truth. We then port the trained NCA to the ESP32 microcontroller, enabling efficient image processing on hardware as small as a camera capsule. NCA are more accurate (Dice) than other portable segmentation models, while requiring more than 100x fewer parameters stored in memory than other small-scale models. The visual results of NCA depth estimation look convincing, and in some cases beat the realism and detail of the pseudo ground truth. Runtime optimizations on the ESP32-S3 accelerate the average inference speed significantly, by more than factor 3. With several algorithmic adjustments and distillation, it is possible to eNCApsulate NCA models into microcontrollers that fit into wireless capsule endoscopes. This is the first work that enables reliable bleeding segmentation and depth estimation on a miniaturized device, paving the way for precise diagnosis combined with visual odometry as a means of precise localization of the capsule -- on the capsule.

Figures (6)

• Figure 1: Experimental hardware setup for eNCApsulate, featuring a tiny version of the ESP32-S3 as the main component. The chip itself is miniaturized and can thus fit into a capsule endoscope (white contour overlay). An RGB display, connected directly to the chip, displays the segmentation of a capsule endoscopic image generated by an NCA.
• Figure 2: eNCApsulate architecture for lightweight segmentation or depth estimation. (1) The channels of the input RGB image are augmented to the match the input + hidden + output channel dimension $C$. (2) The input image is then processed by a learned bank of $3\times 3$ filters, and for each pixel, the concatenated result (3) is fed as an input to the NCA MLP network (4). The NCA MLP computes the image update for each cell. The result is an update vector (5) that is added to the input image buffer by a chance of 50% (stochastic cell update).
• Figure 3: Accuracy of different lightweight segmentation models (blue) vs. eNCApsulateS (green), and their model size in kilobytes, on a logarithmic scale.
• Figure 4: Qualitative segmentation results for eNCApsulateS, compared to other lightweight segmentation models based on CNN.
• Figure 5: Visual comparison of different monocular depth estimation approaches on a part of the benchmark dataset (subset of KvasirCapsule). eNCApsulate was trained on the KID2 dataset, whereas the other models are foundation models.