AnyRefill: A Unified, Data-Efficient Framework for Left-Prompt-Guided Vision Tasks

TL;DR

This work introduces Left-Prompt-Guided (LPG) as a unified input scheme that places a left reference image next to a right masked target, reframing perception, editing, and conditional generation as LPG-inpainting tasks. AnyRefill extends the LeftRefill idea by leveraging a DiT-based diffusion backbone (FLUX.Fill) and injecting task-specific LoRA adapters, enabling diverse vision tasks with minimal fine-tuning and no extra visual encoders. Through extensive experiments and ablations, AnyRefill delivers high-quality, reference-aligned results across editing, generation, and perception tasks, often matching or surpassing open-source and commercial baselines while maintaining data efficiency. The approach demonstrates strong potential for practical deployment, offering a single, adaptable framework that scales to new tasks with limited training data and modest compute.

Abstract

In this paper, we present a novel Left-Prompt-Guided (LPG) paradigm to address a diverse range of reference-based vision tasks. Inspired by the human creative process, we reformulate these tasks using a left-right stitching formulation to construct contextual input. Building upon this foundation, we propose AnyRefill, an extension of LeftRefill, that effectively adapts Text-to-Image (T2I) models to various vision tasks. AnyRefill leverages the inpainting priors of advanced T2I model based on the Diffusion Transformer (DiT) architecture, and incorporates flexible components to enhance its capabilities. By combining task-specific LoRAs with the stitching input, AnyRefill unlocks its potential across diverse tasks, including conditional generation, visual perception, and image editing, without requiring additional visual encoders. Meanwhile, AnyRefill exhibits remarkable data efficiency, requiring minimal task-specific fine-tuning while maintaining high generative performance. Through extensive ablation studies, we demonstrate that AnyRefill outperforms other image condition injection methods and achieves competitive results compared to state-of-the-art open-source methods. Notably, AnyRefill delivers results comparable to advanced commercial tools, such as IC-Light and SeedEdit, even in challenging scenarios. Comprehensive experiments and ablation studies across versatile tasks validate the strong generation of the proposed simple yet effective LPG formulation, establishing AnyRefill as a unified, highly data-efficient solution for reference-based vision tasks.

Figures (23)

• Figure 1: An image, generated by DALL·E 3 betker2023improving, vividly illustrates the motivation behind AnyRefill. A robot, representing the T2I model in AnyRefill, acts as an experienced painter, using the left image as a reference to create content on the right canvas.
• Figure 2: AnyRefill unifies various vision tasks by generating the right canvas conditioned by left references. We can re-formulate several existing tasks in the Left-Prompt-Guided (LPG) manner, including perception tasks, image editing tasks, and conditional generation tasks.
• Figure 3: Overview and comparison of (a) our previous LeftRefill cao2024leftrefill and (b) AnyRefill under the LPG formulation. The task prompt embedding is infused into CLIP-H for cross-attention learning in U-net, while task-specific LoRAs are adopted to the rectified flow-based DiT for more diverse vision tasks. For the output of LeftRefill and AnyRefill, we discard the left-side reference and take the right-side generation.
• Figure 4: Inputs of (a) Partially masked targets for reference inpainting based tasks; and (b) Fully masked right canvas for various other vision tasks. Masked regions are indicated with semi-transparent blue.
• Figure 5: Qualitative results of the deblurring task. AnyRefill restores content and maintains consistency with the reference.