USA-Net: Unified Semantic and Affordance Representations for Robot Memory

TL;DR

USA-Net addresses the need for a memory representation that encodes both scene semantics and affordances to support open-ended instructions. It introduces a unified differentiable map learned from RGB-D data with CLIP-based semantic supervision and SDF-based affordance supervision. It then proposes two planners: a grid-based baseline and a gradient-based planner that jointly optimize semantic goals and path feasibility. Experiments across diverse scenes show the gradient planner achieves shorter trajectories and higher semantic alignment than grid-based planners. This enables more flexible, language-driven robotic navigation in cluttered environments.

Abstract

In order for robots to follow open-ended instructions like "go open the brown cabinet over the sink", they require an understanding of both the scene geometry and the semantics of their environment. Robotic systems often handle these through separate pipelines, sometimes using very different representation spaces, which can be suboptimal when the two objectives conflict. In this work, we present USA-Net, a simple method for constructing a world representation that encodes both the semantics and spatial affordances of a scene in a differentiable map. This allows us to build a gradient-based planner which can navigate to locations in the scene specified using open-ended vocabulary. We use this planner to consistently generate trajectories which are both shorter 5-10% shorter and 10-30% closer to our goal query in CLIP embedding space than paths from comparable grid-based planners which don't leverage gradient information. To our knowledge, this is the first end-to-end differentiable planner optimizes for both semantics and affordance in a single implicit map. Code and visuals are available at our website: https://usa.bolte.cc/

Figures (4)

• Figure 1: USA-Net lets us jointly optimize for satisfying language goals, and finding collision-free trajectories. Above, see an example trajectory from a random starting point sampled from the camera trajectory, navigating to two different target queries, "Jackets hanging in the closet" and "a conference room." The camera poses are shown in pink and blue. The generated trajectory is shown starting in red and ending in green. High semantic similarity points are shown in bright red. For more examples, see our website.
• Figure 2: A high-level diagram of the system. A single network encodes both the semantic and affordance information for the environment, enabling the robot to navigate using open-ended vocabulary queries.
• Figure 3: Simulation showing how sampling more points systematically increases the bias in the final SDF estimate. In this case, we consider the SDF computed using the nearest neighbor point sampled from $N$ points centered at the same location 1 meter from the reference point (i.e., a true SDF value of 1 meter), drawn from the distribution described in Equation \ref{['eq:point_distribution']} with $\sigma_C = 1 cm$.
• Figure 4: An illustration of the underestimation modeled by Equation \ref{['eq:point_distribution']}. In the top image, we show the navigable space found by thresholding the SDF to 50 centimeters, overlaid on navigable space found by taking the Minkowski sum of the point cloud-based occupancy grid with a 50 centimeter diameter column. The SDF underestimates the navigable space relative to the point cloud-based occupancy grid. On the bottom, we instead threshold the SDF to 30 centimeters, estimated as in Figure \ref{['fig:distribution_simulation']}, and find that it aligns much more closely with the point cloud-based navigable space.