ProxyImg: Towards Highly-Controllable Image Representation via Hierarchical Disentangled Proxy Embedding

TL;DR

ProxyImg introduces a hierarchical proxy-based parametric image representation that decouples semantic, geometric, and texture attributes into independent spaces, enabling precise, component-wise editing and high-fidelity reconstruction across natural images. It constructs semantic layers via a Bezier-guided geometry pipeline with adaptive boundary proxies and multi-scale internal proxies, while embedding multi-scale implicit textures onto distributed proxy nodes indexed by a shared locality-aware feature grid. A lightweight decoder φθ decodes coordinates to RGB values, supporting geometry edits, texture edits, and real-time animation when combined with Position-Based Dynamics. Empirical results on ImageNet, OIR-Bench, and HumanEdit demonstrate state-of-the-art rendering quality with far fewer parameters, along with intuitive interactive editing and physics-based animation capabilities that surpass generative-model-based approaches in controllability and efficiency.

Abstract

Prevailing image representation methods, including explicit representations such as raster images and Gaussian primitives, as well as implicit representations such as latent images, either suffer from representation redundancy that leads to heavy manual editing effort, or lack a direct mapping from latent variables to semantic instances or parts, making fine-grained manipulation difficult. These limitations hinder efficient and controllable image and video editing. To address these issues, we propose a hierarchical proxy-based parametric image representation that disentangles semantic, geometric, and textural attributes into independent and manipulable parameter spaces. Based on a semantic-aware decomposition of the input image, our representation constructs hierarchical proxy geometries through adaptive Bezier fitting and iterative internal region subdivision and meshing. Multi-scale implicit texture parameters are embedded into the resulting geometry-aware distributed proxy nodes, enabling continuous high-fidelity reconstruction in the pixel domain and instance- or part-independent semantic editing. In addition, we introduce a locality-adaptive feature indexing mechanism to ensure spatial texture coherence, which further supports high-quality background completion without relying on generative models. Extensive experiments on image reconstruction and editing benchmarks, including ImageNet, OIR-Bench, and HumanEdit, demonstrate that our method achieves state-of-the-art rendering fidelity with significantly fewer parameters, while enabling intuitive, interactive, and physically plausible manipulation. Moreover, by integrating proxy nodes with Position-Based Dynamics, our framework supports real-time physics-driven animation using lightweight implicit rendering, achieving superior temporal consistency and visual realism compared with generative approaches.

Figures (11)

• Figure 1: (Left): ProxyImg enables precise instance-level multi-turn image editing, offering significantly greater control over edits compared to advanced commercial models. (Right): ProxyImg also supports physics-based image-to-animation generation, producing animations with superior physical realism and temporal consistency while Sora and Jimeng either fail to demonstrate clear dynamic effects or exhibit significant temporal inconsistencies. "Gemini" means Gemini 2.5 Flash Image comanici2025gemini and Jimeng uses Seedance 1.0 gao2025seedance.
• Figure 2: Overview of our framework. We propose a novel vectorized image representation to embed multi-scale image texture features on multi-layer hierarchically related geometric control points. With such representation, the texture at arbitrary position in the continuous image domain can be decoded with a geometry-aware interpolation method and a lightweight decoding function. Building upon our representation, we can enable a variety of controllable and precise image manipulations, such as geometric editing, texture editing, and image animation, with only minimal computational cost.
• Figure 3: (a). Diagram of proxy nodes construction and distributed embedding of image textures. (b). Schematic of the coordinate-to-texture network training process based on proxy embeddings and illustration of background hole completion using a learnable spatial feature grid (represented with green dots). (c). Illustration of the differences between the proposed feature assignment method and the commonly used random feature assignment. Compared with prior methods, the proposed algorithm effectively ensures distributional consistency of local texture codes, which not only enhances texture stability during image manipulation but also accelerates the parameter optimization process. Please zoom in and refer to the legends for more details.
• Figure 4: Qualitative comparison on natural images. We use red boxes to emphasize the differences. Our representation can express complex image details. The results of S-SVG hu2024supersvg are directly obtained from their paper. Please zoom in for more details.
• Figure 5: Qualitative comparison on natural images. We visualize examples of methods with the same order of magnitude of parameters. Please zoom in for more details.