CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AI

TL;DR

This work proposes CreativeConnect, a system with generative AI pipelines that helps users discover useful elements from the reference image using keywords, recommends relevant keywords, generates diverse recombination options with user-selected keywords, and shows recombinations as sketches with text descriptions.

Abstract

Graphic designers often get inspiration through the recombination of references. Our formative study (N=6) reveals that graphic designers focus on conceptual keywords during this process, and want support for discovering the keywords, expanding them, and exploring diverse recombination options of them, while still having room for designers' creativity. We propose CreativeConnect, a system with generative AI pipelines that helps users discover useful elements from the reference image using keywords, recommends relevant keywords, generates diverse recombination options with user-selected keywords, and shows recombinations as sketches with text descriptions. Our user study (N=16) showed that CreativeConnect helped users discover keywords from the reference and generate multiple ideas based on them, ultimately helping users produce more design ideas with higher self-reported creativity compared to the baseline system without generative pipelines. While CreativeConnect was shown effective in ideation, we discussed how CreativeConnect can be extended to support other types of tasks in creativity support.

Figures (11)

• Figure 1: Screenshot of CreativeConnect. (a) Keyword Extraction Panel: The system automatically extracts keywords in four categories (subject matter, action & pose, theme & mood, and arrangement) from the reference image. Users can select these keywords or add keywords manually. (b) Interactive Mood Board with Keyword Suggestion Panel: Users can organize the reference images along with the selected keywords. Users can import the keywords shown below, which are suggested based on all keywords on the board or the keywords that users selected on this mood board. (c) Keyword Merge Panel: When users select keywords they want to recombine on the mood board, the system generates sketches and their respective descriptions, including all selected keywords. Users can view more generated sketches by clicking the "More Sketches" button.
• Figure 2: Technical pipeline of CreativeConnect. (a) Keyword extraction from image: The caption generated from the image captioning model goes into the LLM to extract subject matter, action & pose, and theme & mood. The segmentation model is used to detect the image's arrangement. (b) Keyword-based image generation: the LLM generates descriptions based on the given keywords, and the layout variator generates similar arrangements. The image generation model generates the image, and the style transfer model converts this into a sketch.
• Figure 3: User study process. The 2-hour user study consists of two sessions with different tools, each including a 30-minute ideation phase utilizing the given tool. The order of the tool and the design tasks are counterbalanced. After the two sessions, they had 20-minute semi-structured interviews about their experiences.
• Figure 4: Survey results on the user-perceived efficiency during each recombination step with a 95% confidence interval. CreativeConnect was significantly helpful in discovering elements from the reference image and generating multiple ideas.
• Figure 5: Comparison of the count of two different actions (adding keywords, generating image through generation model) taken to generate each sketch (from the first sketch to the fifth sketch) in CreativeConnect and baseline. The results showed that users use the add keyword action more in CreativeConnect compared to the baseline, where users only add keywords for the initial sketches. There was no significant difference in the count of generated image action.