OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

TL;DR

OpenVTON-Bench addresses a critical gap in evaluating high-fidelity, controllable virtual try-on by introducing a large-scale, high-resolution benchmark with dense semantic captions across 20 garment categories. The authors combine a VLM-based semantic assessment with a Multi-Scale Representation Metric that uses SAM3 segmentation and morphological erosion to separate boundary errors from interior texture artifacts, achieving strong alignment with human judgments (e.g., Kendall's tau 0.83 vs 0.61 for SSIM). The dataset, built from ~100K image pairs up to $1536×1536$, is constructed via DINOv3 semantic clustering and Gemini-based dense captioning, ensuring balanced distribution and commercial-grade annotations. Experiments reveal a texture-realism gap in diffusion-based VTON models and demonstrate that the proposed hybrid evaluation outperforms traditional pixel metrics in correlating with human preferences. OpenVTON-Bench thus provides a robust, open benchmark to drive commercial fidelity in VTON systems and enables fine-grained diagnosis of texture and semantic shortcomings.

Abstract

Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck. Traditional metrics struggle to quantify fine-grained texture details and semantic consistency, while existing datasets fail to meet commercial standards in scale and diversity. We present OpenVTON-Bench, a large-scale benchmark comprising approximately 100K high-resolution image pairs (up to $1536 \times 1536$). The dataset is constructed using DINOv3-based hierarchical clustering for semantically balanced sampling and Gemini-powered dense captioning, ensuring a uniform distribution across 20 fine-grained garment categories. To support reliable evaluation, we propose a multi-modal protocol that measures VTON quality along five interpretable dimensions: background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism. The protocol integrates VLM-based semantic reasoning with a novel Multi-Scale Representation Metric based on SAM3 segmentation and morphological erosion, enabling the separation of boundary alignment errors from internal texture artifacts. Experimental results show strong agreement with human judgments (Kendall's $τ$ of 0.833 vs. 0.611 for SSIM), establishing a robust benchmark for VTON evaluation.

Figures (8)

• Figure 1: The Proposed Hybrid Evaluation Framework. We move beyond single-scalar metrics by decomposing VTON quality into five human-aligned dimensions. Uniquely, our framework combines (Top) a VLM-as-a-Judge module for semantic auditing with (Bottom) a Multi-Scale Representation Metric that verifies semantic structural consistency. This synergy ensures both semantic plausibility and accurate garment replication.
• Figure 2: Data Construction Pipeline of OpenVTON-Bench. The process consists of three stages: (1) Large-scale raw data aggregation from diverse sources; (2) Hybrid annotation combining human verification for pair alignment and VLM-based dense captioning; (3) Semantic-aware filtering using DINOv3 clustering to ensure a balanced distribution across 20 fine-grained categories.
• Figure 3: Dataset Analysis of OpenVTON-Bench.Left (a-d): t-SNE visualizations of the full dataset and the train/validation/test splits. Right (e): Category distribution of the dataset.
• Figure 4: Representative examples from OpenVTON-Bench. More examples are provided in Appendix \ref{['app:data_vis']}.
• Figure 5: Qualitative comparison of state-of-the-art methods on OpenVTON-Bench.