EVCtrl: Efficient Control Adapter for Visual Generation

TL;DR

EVCtrl addresses the high computational cost of controllable diffusion-based generation by identifying and exploiting spatial and temporal redundancies. It introduces Local Focused Caching (LFoC) to concentrate computation on salient control regions and Denoising Step Skipping (DSS) to skip noncritical denoising steps, all without retraining. The method is validated on multiple ControlNet-enabled models (CogVideo-ControlNet, Wan2.1-ControlNet, Flux-ControlNet), achieving substantial speedups (up to around 2.16x) with minimal degradation in quality across image and video tasks. These results offer a practical, training-free path to real-time or near-real-time controllable generation in diverse visual domains.

Abstract

Visual generation includes both image and video generation, training probabilistic models to create coherent, diverse, and semantically faithful content from scratch. While early research focused on unconditional sampling, practitioners now demand controllable generation that allows precise specification of layout, pose, motion, or style. While ControlNet grants precise spatial-temporal control, its auxiliary branch markedly increases latency and introduces redundant computation in both uncontrolled regions and denoising steps, especially for video. To address this problem, we introduce EVCtrl, a lightweight, plug-and-play control adapter that slashes overhead without retraining the model. Specifically, we propose a spatio-temporal dual caching strategy for sparse control information. For spatial redundancy, we first profile how each layer of DiT-ControlNet responds to fine-grained control, then partition the network into global and local functional zones. A locality-aware cache focuses computation on the local zones that truly need the control signal, skipping the bulk of redundant computation in global regions. For temporal redundancy, we selectively omit unnecessary denoising steps to improve efficiency. Extensive experiments on CogVideo-Controlnet, Wan2.1-Controlnet, and Flux demonstrate that our method is effective in image and video control generation without the need for training. For example, it achieves 2.16 and 2.05 times speedups on CogVideo-Controlnet and Wan2.1-Controlnet, respectively, with almost no degradation in generation quality.Codes are available in the supplementary materials.

Figures (5)

• Figure 1: Showcase of EVCtrl. We propose the EVCtrl, a lightweight, plug-and-play control adapter.We present the outcomes of controllable image and video generation under various control conditions, along with an efficiency comparison.
• Figure 2: Motivation illustration. We observe significant spatial redundancy (a) and temporal redundancy (b) in controllable image and video generation. Spatially(a), large regions in the control image or video that carry no control information need not be computed. Temporally(b), skipping most highly similar adjacent diffusion timesteps does not impair controllability.
• Figure 3: Overall framework.The pipeline takes initial noise and control conditions as inputs. Specifically, full caches are refreshed at fixed timestep intervals and at a handful of pivotal timesteps. For cached intermediate steps, only selected tokens within the mid-to-late fresh_blocks that capture local salient control cues are updated.
• Figure 4: The qualitative comparisons with existing methods.Visualization comparing acceleration methods on Wan2.1-ControlNet: while others sacrifice control-condition details and introduce visual distortion at high speed-up ratios, ours preserves fine-grained details and maintains high-quality generation.
• Figure 5: The qualitative ablation study. We select the canny as the input condition to ablate the performance.