Elastic-DETR: Making Image Resolution Learnable with Content-Specific Network Prediction

TL;DR

Elastic-DETR introduces a learnable image-resolution mechanism for DETR-based detectors by attaching a lightweight scale predictor that outputs a per-image scale factor $\phi$ and scales inputs via $Scale(\cdot,\phi)$. The scale predictor is trained with two novel losses: a scale loss $\mathcal{L}_{scale}$ that ties the factor to object sizes through a probability $P_{up}$ and a distribution loss $\mathcal{L}_{dist}$ that aligns the overall scaling tendency with detection performance using a Beta distribution and Wasserstein distance. By enabling content-specific resolution adjustments and end-to-end optimization, Elastic-DETR demonstrates up to $3.5$ percentage points AP improvement or a $26\%$ reduction in computation on MS-COCO, with robust gains across backbones and small-object scales. The approach provides a general framework for learnable hyperparameter optimization and suggests avenues to extend resolution learning to other components and tasks, offering practical flexibility for diverse deployment constraints.

Abstract

Multi-scale image resolution is a de facto standard approach in modern object detectors, such as DETR. This technique allows for the acquisition of various scale information from multiple image resolutions. However, manual hyperparameter selection of the resolution can restrict its flexibility, which is informed by prior knowledge, necessitating human intervention. This work introduces a novel strategy for learnable resolution, called Elastic-DETR, enabling elastic utilization of multiple image resolutions. Our network provides an adaptive scale factor based on the content of the image with a compact scale prediction module (< 2 GFLOPs). The key aspect of our method lies in how to determine the resolution without prior knowledge. We present two loss functions derived from identified key components for resolution optimization: scale loss, which increases adaptiveness according to the image, and distribution loss, which determines the overall degree of scaling based on network performance. By leveraging the resolution's flexibility, we can demonstrate various models that exhibit varying trade-offs between accuracy and computational complexity. We empirically show that our scheme can unleash the potential of a wide spectrum of image resolutions without constraining flexibility. Our models on MS COCO establish a maximum accuracy gain of 3.5%p or 26% decrease in computation than MS-trained DN-DETR.

Figures (14)

• Figure 1: A preliminary experiment for the effect of image resolution selection with DETR-R50 (50 epoch) on MS COCO. We trained the MS strategy while expanding the resolution range with an increase in the maximum size.
• Figure 2: Comparison on COCO val. The marker size indicates the maximum size of the image resolution, which is employed 800$\times$1333 for MS training. The base backbone network is R50.
• Figure 3: Overview of Elastic-DETR. The image resolution is scaled according to the scale factor, which is obtained from the scale predictor. The scale factor optimization is achieved from scale loss and distribution loss. These functions modify the scale factor based on the relative size of the object between two size boundaries, determined by the per-scale performance of the network.
• Figure 4: Scale loss produces a low value for high (low)-scale factor for small (large) objects. When $y_{up}=0$, the input probability converges to ${\tau_{min}}/{\tau_{max}}$, indicating that scale loss can function effectively with scale factor clipping.
• Figure 5: Comparison between two forms of the loss function utilized in distribution loss. In this graph, we assume the plain loss values follow a normal distribution.