Text2Street: Controllable Text-to-image Generation for Street Views

TL;DR

Text2Street tackles the challenge of text-to-image generation for street views by introducing a controllable diffusion-based framework that separately handles road topology, traffic object layout, and weather conditions. The lane-aware road topology generator produces a local semantic map with lane lines using a counting adapter and diffusion with CLIP guidance, while the position-based object layout generator creates object layouts via an object-level bounding-box diffusion steered by a local semantic map. A final multiple control image generator projects these components into a perspective view and fuses topology, layout, and weather through a multi-branch ControlNet-enabled diffusion process, trained with dedicated losses $\mathcal{L}_{SD}$, $\mathcal{L}_{CA}$, $\mathcal{L}_{POLG}$, and $\mathcal{L}_{MCIG}$. Evaluations on nuScenes show superior performance over strong baselines in both image fidelity and attribute-level controllability (road structure, lane counting, object counting, and weather), and the generated data can enhance downstream tasks such as object detection. The practical impact lies in providing fine-grained, text-driven street-view synthesis useful for data augmentation, map construction, and scenario planning in autonomous driving contexts.

Abstract

Text-to-image generation has made remarkable progress with the emergence of diffusion models. However, it is still a difficult task to generate images for street views based on text, mainly because the road topology of street scenes is complex, the traffic status is diverse and the weather condition is various, which makes conventional text-to-image models difficult to deal with. To address these challenges, we propose a novel controllable text-to-image framework, named \textbf{Text2Street}. In the framework, we first introduce the lane-aware road topology generator, which achieves text-to-map generation with the accurate road structure and lane lines armed with the counting adapter, realizing the controllable road topology generation. Then, the position-based object layout generator is proposed to obtain text-to-layout generation through an object-level bounding box diffusion strategy, realizing the controllable traffic object layout generation. Finally, the multiple control image generator is designed to integrate the road topology, object layout and weather description to realize controllable street-view image generation. Extensive experiments show that the proposed approach achieves controllable street-view text-to-image generation and validates the effectiveness of the Text2Street framework for street views.

Figures (7)

• Figure 1: Challenges of text-to-image generation for street views. There are three primary challenges: (1) complex road topology, including road structure in the first row and topological marks in the second row, (2) diverse traffic status, e.g., varying traffic objects in the third row, and (3) various weather conditions like the rainy day in the last row. Note that Reference are original images from nuScenes caesar2020nuscenes, Stable Diffusion rombach2022high/Midjourney midjourney/DALLE3 dalle3 are tested on their official APIs, and Stable Diffusion$^*$ and Ours are finetuned on nuScenes.
• Figure 2: Framework of Text2Street. We begin by introducing the lane-aware road topology generator, which utilizes textual input to create a local semantic map representing the intricate road topology with lane information. Next, we present the position-based object layout generator, which captures the diversity of traffic status and generate the traffic object layout. Subsequently, the road topology and object layout are projected into the camera's perspective through pose sampling. Finally, the projected road topology, object layout, and textual weather description are integrated through the multiple control image generator to produce the ultimate street-view image.
• Figure 3: Architecture of the lane-aware road topology generator.
• Figure 4: Architecture of the position-based object layout generator. Note $\oplus$ means element-wise addition.
• Figure 5: Architecture of the multiple control image generator. Note $\otimes$, $\oplus$ means the concatenation and element-wise addition.