FitControler: Toward Fit-Aware Virtual Try-On

TL;DR

FitControler delivers a practical, plug-in solution to integrate fit-aware control into diffusion-based virtual try-on, addressing the gap between garment fit and realistic rendering. It introduces a two-stage pipeline: a fit-aware layout generator that reshapes body-garment layouts conditioned on fit, and a multi-scale fit injector that passes layout cues into existing VTON models, leveraging a garment-agnostic preprocessor to avoid shape leakage. A new Fit4Men dataset with labeled fits and two fit-consistency metrics (Hu moments and Hausdorff distance) enables robust evaluation, with extensive experiments showing consistent improvements across multiple VTON backbones and notable data-efficiency. The work demonstrates that explicit modeling of garment fit improves realism and user control, offering a path toward scalable, fit-consistent VTON across diverse garments and poses.

Abstract

Realistic virtual try-on (VTON) concerns not only faithful rendering of garment details but also coordination of the style. Prior art typically pursues the former, but neglects a key factor that shapes the holistic style -- garment fit. Garment fit delineates how a garment aligns with the body of a wearer and is a fundamental element in fashion design. In this work, we introduce fit-aware VTON and present FitControler, a learnable plug-in that can seamlessly integrate into modern VTON models to enable customized fit control. To achieve this, we highlight two challenges: i) how to delineate layouts of different fits and ii) how to render the garment that matches the layout. FitControler first features a fit-aware layout generator to redraw the body-garment layout conditioned on a set of delicately processed garment-agnostic representations, and a multi-scale fit injector is then used to deliver layout cues to enable layout-driven VTON. In particular, we build a fit-aware VTON dataset termed Fit4Men, including 13,000 body-garment pairs of different fits, covering both tops and bottoms, and featuring varying camera distances and body poses. Two fit consistency metrics are also introduced to assess the fitness of generations. Extensive experiments show that FitControler can work with various VTON models and achieve accurate fit control. Code and data will be released.

Figures (19)

• Figure 1: Comparison between existing VTON models and our proposed FitControler. (a) Existing models produce only a fixed fit, often leading to unnatural results due to mismatched fit. (b) With FitControler, the same inputs can produce try-on results with customized fits such as slim, regular, and loose T-shirts, which better matches the overall style and user preference.
• Figure 2: Overview of FitControler. The person image is first processed by (a) the garment-agnostic preprocessor to extract the mask and dense pose. These are concatenated with the noise map and garment image---along both channel and spatial dimensions as in CatVTON chong2024catvton---before being fed into (b) the fit-aware layout generator to produce a fit-sensitive segmentation map. The layout features are then delivered by (c) the multi-scale fit injector to VTON models via the ControlNet zhang2023adding interface. The layout generator is pre-trained and remains frozen when integrating FitControler into different VTON models, where only the fit injector requires model-specific training.
• Figure 3: Impact of shape leakage in mask and dense pose. (a) Commonly used masks and dense poses implicitly encode the original garment boundaries, which (b) biases the model to follow these cues when rendering garment. To address this, (c) our preprocessor reconstructs them from human keypoints to form standardized representations. See supplementary material for the ablation on this preprocessor.
• Figure 4: Qualitative results of VTON models with FitControler. Regions showing the most prominent fit variations are highlighted with dashed boxes. Additional examples on other VTON models are provided in the supplementary material.
• Figure 5: Correlation analysis of fit consistency metrics. For each target fit, three types of fits (slim, regular, and loose) are generated and compared.