Learning to Segment Liquids in Real-world Images

TL;DR

Segmenting liquids in real-world images is difficult due to transparency, reflectance, and deformable shapes. The authors introduce LQDS, a 5000-image, 14-class dataset, and LQDM, a dual-branch ViT-based network that uses boundary-guided cross-attention to refine liquid masks. The approach is trained with a combined loss $L = L_m + \omega L_b$ and boundary supervision, achieving state-of-the-art performance on LQDS and demonstrating generalization to ADE20K. These results establish a strong baseline for liquid segmentation with clear implications for robotics, enabling safer navigation and manipulation around liquids.

Abstract

Different types of liquids such as water, wine and medicine appear in all aspects of daily life. However, limited attention has been given to the task, hindering the ability of robots to avoid or interact with liquids safely. The segmentation of liquids is difficult because liquids come in diverse appearances and shapes; moreover, they can be both transparent or reflective, taking on arbitrary objects and scenes from the background or surroundings. To take on this challenge, we construct a large-scale dataset of liquids named LQDS consisting of 5000 real-world images annotated into 14 distinct classes, and design a novel liquid detection model named LQDM, which leverages cross-attention between a dedicated boundary branch and the main segmentation branch to enhance segmentation predictions. Extensive experiments demonstrate the effectiveness of LQDM on the test set of LQDS, outperforming state-of-the-art methods and establishing a strong baseline for the semantic segmentation of liquids.

Figures (7)

• Figure 1: Examples of liquids in real-world settings. The images illustrate the wide variation in appearance, color, and shape across different liquid types such as coffee, water, chemical, and juice. These examples highlight the challenges of liquid segmentation: liquids can reflect their surroundings, exhibit transparency, and deform freely to fit containers of any geometry. Such variability makes distinguishing liquid regions and classes difficult for standard segmentation methods trained on rigid and opaque objects.
• Figure 2: Examples of pairs of images and masks for each of the 14 classes in the LQDS dataset, demonstrating the variety of forms and appearances that liquids come in.
• Figure 3: Statistics of the LQDS dataset.
• Figure 4: Overview of the proposed dual-branch architecture, LQDM. A ViT backbone first extracts shared feature embeddings through $L_1$ transformer encoder blocks. The network then splits into two branches: the mask branch for semantic segmentation and the boundary branch for boundary prediction. Each branch adds $K$ learnable queries to its feature embeddings, which pass through $L_2$ cross-attention blocks that apply cross-attention as per Equation \ref{['eq:cab']}, while the combined query and feature embeddings pass through $L_2$ transformer encoder blocks, enabling the mask branch to attend to boundary features. The outputs of each branch are processed by the mask module which produces mask, boundary, and class logits, which are used to compute training loss as per Equation \ref{['eq:loss']}.
• Figure 5: Visual comparison of LQDM to other semantic segmentation methods on images from the LQDS testing set.