OVSegDT: Segmenting Transformer for Open-Vocabulary Object Goal Navigation

TL;DR

This work introduces OVSegDT, a lightweight transformer policy that tackles end-to-end policies overfit small simulator datasets with two synergistic components, cutting the sample complexity of training by 33%, and reducing collision count in two times while keeping inference cost low.

Abstract

Open-vocabulary Object Goal Navigation requires an embodied agent to reach objects described by free-form language, including categories never seen during training. Existing end-to-end policies overfit small simulator datasets, achieving high success on training scenes but failing to generalize and exhibiting unsafe behaviour (frequent collisions). We introduce OVSegDT, a lightweight transformer policy that tackles these issues with two synergistic components. The first component is the semantic branch, which includes an encoder for the target binary mask and an auxiliary segmentation loss function, grounding the textual goal and providing precise spatial cues. The second component consists of a proposed Entropy-Adaptive Loss Modulation, a per-sample scheduler that continuously balances imitation and reinforcement signals according to the policy entropy, eliminating brittle manual phase switches. These additions cut the sample complexity of training by 33%, and reduce collision count in two times while keeping inference cost low (130M parameters, RGB-only input). On HM3D-OVON, our model matches the performance on unseen categories to that on seen ones and establishes state-of-the-art results (40.1% SR, 20.9% SPL on val unseen) without depth, odometry, or large vision-language models. Code is available at https://github.com/CognitiveAISystems/OVSegDT.

Figures (10)

• Figure 1: We demonstrate that the explicit integration of semantic segmentation through a target binary mask and an auxiliary semantic segmentation loss function significantly improves the quality of end-to-end training of a transformer-based model for open-vocabulary navigation.
• Figure 2: OVSegDT model encodes the current visual observation $I_t$, the goal object category $G$, the previous action $a_{t-1}$ and binary mask of the target object (goal mask) $M_t$ to form observation embedding $o_t$. At each timestep, a transformer receives the embedding sequence from the previous 100 steps. The action head is used to sample the action $a_t$. During training, a segmentation DCGAN head generates a binary target mask $m_t$ for the auxiliary segmentation loss function, and a critic head is employed to predict the value $v_t$ of the current state.
• Figure 3: Qualitative comparison of navigation trajectories between our method OVSegDT and the baseline DagRL on categories from the val unseen split of the HM3D-OVON benchmark. Left: The DagRL model fails to recognize the target object a pillow and thus ends the episode unsuccessfully, while OVSegDT successfully reaches the goal thanks to the binary goal mask. Right: The use of the binary goal mask enables OVSegDT to search for the target object stairs more efficiently compared to DagRL, which takes extra steps in the environment and passes by the goal at the beginning of the episode.
• Figure 4: Training curves of Success Rate (higher is better) for the considered switching strategies. Our entropy-adaptive EALM reaches top performance with the fewest samples.
• Figure 5: Training curves of Collision Count (lower is better). Collisions serve as a safety proxy; EALM steadily reduces collisions whereas other methods eventually over-fit and regress.