PortionNet: Distilling 3D Geometric Knowledge for Food Nutrition Estimation
Darrin Bright, Rakshith Raj, Kanchan Keisham
TL;DR
PortionNet addresses the challenge of estimating nutrition from a single RGB image by injecting 3D geometric reasoning during training through cross-modal distillation. It uses a PointNet-based geometry encoder as a teacher and a lightweight RGB-to-geometry adapter as a student, with dual-mode training to balance 3D guidance with RGB-only inference. On MetaFood3D, it achieves state-of-the-art performance in volume and energy estimation, while cross-dataset evaluation on SimpleFood45 shows strong generalization. This work demonstrates that transferring geometric knowledge from 3D to 2D representations enables accurate nutrition estimation on standard smartphones without depth sensors.
Abstract
Accurate food nutrition estimation from single images is challenging due to the loss of 3D information. While depth-based methods provide reliable geometry, they remain inaccessible on most smartphones because of depth-sensor requirements. To overcome this challenge, we propose PortionNet, a novel cross-modal knowledge distillation framework that learns geometric features from point clouds during training while requiring only RGB images at inference. Our approach employs a dual-mode training strategy where a lightweight adapter network mimics point cloud representations, enabling pseudo-3D reasoning without any specialized hardware requirements. PortionNet achieves state-of-the-art performance on MetaFood3D, outperforming all previous methods in both volume and energy estimation. Cross-dataset evaluation on SimpleFood45 further demonstrates strong generalization in energy estimation.