PRISM: Learning Design Knowledge from Data for Stylistic Design Improvement

TL;DR

PRISM addresses the challenge of stylistic design improvement from natural language instructions by learning a design knowledge base from real designs. It infers style-specific knowledge through style space partitioning using GRAD-based graph distances, then distills this into compact, discriminative knowledge via a contrastive learning objective, and finally applies this knowledge at inference through a retrieval-augmented generation pipeline to produce style-aware edits. On Crello data, PRISM achieves the best balance of fidelity and diversity across 15 styles, outperforming data-agnostic prompts and prior data-driven baselines, with user studies showing designers prefer PRISM’s outputs. The work demonstrates that combining data-curated style knowledge with retrieval-based editing yields outputs that more closely reflect authentic design data and user stylistic intents, offering practical gains for real-world design workflows.

Abstract

Graphic design often involves exploring different stylistic directions, which can be time-consuming for non-experts. We address this problem of stylistically improving designs based on natural language instructions. While VLMs have shown initial success in graphic design, their pretrained knowledge on styles is often too general and misaligned with specific domain data. For example, VLMs may associate minimalism with abstract designs, whereas designers emphasize shape and color choices. Our key insight is to leverage design data -- a collection of real-world designs that implicitly capture designer's principles -- to learn design knowledge and guide stylistic improvement. We propose PRISM (PRior-Informed Stylistic Modification) that constructs and applies a design knowledge base through three stages: (1) clustering high-variance designs to capture diversity within a style, (2) summarizing each cluster into actionable design knowledge, and (3) retrieving relevant knowledge during inference to enable style-aware improvement. Experiments on the Crello dataset show that PRISM achieves the highest average rank of 1.49 (closer to 1 is better) over baselines in style alignment. User studies further validate these results, showing that PRISM is consistently preferred by designers.

Figures (13)

• Figure 1: A user provides a design to improve and a natural language instruction. PRISM leverages existing design data to generate diverse improvements that align with the requested style. In contrast, approaches that solely rely on VLMs' pretrained knowledge on styles produce outputs that fail to match the design data.
• Figure 2: PRISM Overview. Given design data with a style "abstract," the (1) Style Space Partitioning stage identifies visually distinctive clusters. Then, (2) Knowledge Extraction focuses on learning concise, actionable knowledge from these clusters. Under a contrastive framework, the VLM compares and contrast positive and negative examples before generating the knowledge. We highlight two examples of how positive/negative designs influence specific lines in the guideline. During inference, (3) Prior-Informed Edits proportionally retrieves relevant design knowledge based on the original design data distribution, thereby outputting diverse improvements that also align with data.
• Figure 3: Main Results. We report the average fidelity and diversity with standard error across 15 styles. PRISM achieve the highest value for both metrics. On the left, we also visualize different methods' input/output.
• Figure 4: Qualitative Results. We show 3 styles, each representative of PRISM's different performance. "abstract" is one where PRISM achieves the highest rank for both fidelity and diversity. "artistic" is where one is high (fidelity). "modern" is where both ranks are lower.
• Figure 5: Expected Ranks. We report the expected ranks across all styles. PRISM achieves the best rank in both fidelity and diversity, showing its ability to balance the two metrics.