YOLOE-26: Integrating YOLO26 with YOLOE for Real-Time Open-Vocabulary Instance Segmentation

TL;DR

YOLOE-26 tackles real-time open-vocabulary instance segmentation by unifying the deployment-efficient YOLOv26 detector with YOLOE's open-vocabulary embedding framework. It introduces RepRTA for text alignment, SAVPE for visual prompting, and LRPC for prompt-free inference, all operating in a common object embedding space to enable text, visual, and autonomous prompting within a single model. The approach preserves end-to-end, NMS-free YOLO-like speed while achieving strong open-world performance across model sizes, with $mAP_{50\text{--}95}$ reaching up to 39.5 (text) and 36.2 (visual) at 640 px when prompting is used. It remains compatible with Ultralytics tooling for training, validation, and deployment, and lays out a roadmap toward edge-aware, robust, and agentic open-world perception for dynamic environments.

Abstract

This paper presents YOLOE-26, a unified framework that integrates the deployment-optimized YOLO26(or YOLOv26) architecture with the open-vocabulary learning paradigm of YOLOE for real-time open-vocabulary instance segmentation. Building on the NMS-free, end-to-end design of YOLOv26, the proposed approach preserves the hallmark efficiency and determinism of the YOLO family while extending its capabilities beyond closed-set recognition. YOLOE-26 employs a convolutional backbone with PAN/FPN-style multi-scale feature aggregation, followed by end-to-end regression and instance segmentation heads. A key architectural contribution is the replacement of fixed class logits with an object embedding head, which formulates classification as similarity matching against prompt embeddings derived from text descriptions, visual examples, or a built-in vocabulary. To enable efficient open-vocabulary reasoning, the framework incorporates Re-Parameterizable Region-Text Alignment (RepRTA) for zero-overhead text prompting, a Semantic-Activated Visual Prompt Encoder (SAVPE) for example-guided segmentation, and Lazy Region Prompt Contrast for prompt-free inference. All prompting modalities operate within a unified object embedding space, allowing seamless switching between text-prompted, visual-prompted, and fully autonomous segmentation. Extensive experiments demonstrate consistent scaling behavior and favorable accuracy-efficiency trade-offs across model sizes in both prompted and prompt-free settings. The training strategy leverages large-scale detection and grounding datasets with multi-task optimization and remains fully compatible with the Ultralytics ecosystem for training, validation, and deployment. Overall, YOLOE-26 provides a practical and scalable solution for real-time open-vocabulary instance segmentation in dynamic, real-world environments.

Figures (6)

• Figure 1: Bird’s-eye diagram: simplified architecture of YOLOE-26 for open-vocabulary instance segmentation.
• Figure 2: Performance comparison of YOLO26 under TensorRT FP16 on an NVIDIA T4 GPU (https://docs.ultralytics.com/models/yolo26/). (a) COCO mAP(50–95) versus inference latency (ms/image), comparing YOLO26 with earlier YOLO versions and other real-time detectors, highlighting its improved accuracy–speed trade-off. (b) COCO mAP(50–95) versus end-to-end latency, comparing YOLO26 with YOLOv10 and RT-DETR variants, illustrating its advantage in overall pipeline efficiency.
• Figure 3: Comparison of performance, training cost, and inference efficiency between YOLOE and advanced YOLO-Worldv2 in terms of open text prompts. LVIS AP is evaluated on minival set and FPS w/ TensorRT and w/ CoreML is measured on T4 GPU and iPhone 12, respectively. The results highlight our superiority. (Source: YOLOE paper wang2025yoloe
• Figure 4: Architectural diagram of YOLOE-26 for open-vocabulary instance segmentation. The upper part illustrates the core YOLOv26 end-to-end detection and segmentation pipeline, while the lower part depicts the YOLOE (wang2025yoloe) components that enable text-prompted, visual-prompted, and prompt-free open-vocabulary learning.
• Figure 5: Unified object embedding space in YOLOE-26. Visual features are encoded into object embeddings and matched with text prompts, visual prompts, or prompt-free vocabulary embeddings via similarity-based inference for open-vocabulary instance segmentation.