OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields

TL;DR

OpenHype addresses open-vocabulary segmentation in Neural Radiance Fields by encoding scene hierarchies in a continuous hyperbolic latent space, enabling geodesic traversal to sample multi-scale semantics with a single rendering pass. It introduces a hyperbolic auto-encoder that maps language-aligned features and scene relations into the Lorentz model $\mathbb{L}^n$, and supervises the NeRF to predict hierarchical hyperbolic features along with color and density. Through extensive experiments on adapted Search3D/ScanNet++ and the LERF dataset, OpenHype achieves consistent improvements over state-of-the-art baselines, demonstrates robust hierarchical reasoning, and highlights the benefits of continuous hierarchy representations over discrete, multi-pass approaches. The work suggests broad practical impact for 3D scene understanding in robotics and AR/VR, while acknowledging dependencies on 2D segmentation quality and vision-language biases, and pointing to future directions in probabilistic hyperbolic modeling and generalization to other 3D representations.

Abstract

Modeling the inherent hierarchical structure of 3D objects and 3D scenes is highly desirable, as it enables a more holistic understanding of environments for autonomous agents. Accomplishing this with implicit representations, such as Neural Radiance Fields, remains an unexplored challenge. Existing methods that explicitly model hierarchical structures often face significant limitations: they either require multiple rendering passes to capture embeddings at different levels of granularity, significantly increasing inference time, or rely on predefined, closed-set discrete hierarchies that generalize poorly to the diverse and nuanced structures encountered by agents in the real world. To address these challenges, we propose OpenHype, a novel approach that represents scene hierarchies using a continuous hyperbolic latent space. By leveraging the properties of hyperbolic geometry, OpenHype naturally encodes multi-scale relationships and enables smooth traversal of hierarchies through geodesic paths in latent space. Our method outperforms state-of-the-art approaches on standard benchmarks, demonstrating superior efficiency and adaptability in 3D scene understanding.

Figures (6)

• Figure 1: Method overview figure.(a) Our auto-encoder transforms a hierarchy of language-aligned features into hyperbolic space. (b) Then, we supervise a Nerf model to predict the hierarchical hyperbolic features together with color and density. (c) Lastly, during inference, we can traverse in a continuous manner the scene hierarchy by following a geodesic path in hyperbolic space and perform open-vocabulary queries for a given text prompt.
• Figure 2: Qualitative results on Search3D dataset. While other methods struggle to segment object parts, our method successfully segments these complex compositional prompts.
• Figure 3: Visualization of hierarchies embedded on hyperbolic features. We visualize the hyperbolic embeddings of the hierarchies of two different images of the same scene using CO-SNE guo2022cosne. By color-coding the embeddings, we can see that hierarchies are properly embedded in the hyperbolic space even if the embeddings come from two different images of the same scene.
• Figure :
• Figure E.1: Qualitative results on Search3D dataset of relevancy maps for different levels of the hierarchy of LangSplat qin2024langsplat and OpenHype.