Leveraging Human Revisions for Improving Text-to-Layout Models

TL;DR

This work targets misalignment in text-to-layout generation by leveraging detailed human revisions. It introduces Revision-Aware Reward Models (RARE) that learn from full revision sequences collected via a Figma plugin, and uses RLHF to fine-tune a text-conditioned layout model (PLay) toward designer preferences. The approach, including Chamfer-distance and Keystroke variants, yields stronger alignment and more modern layouts than supervised finetuning or preference-based RLHF, demonstrating the value of richer human feedback. The findings suggest broader applicability of revision-based rewards for aligning generative models with human values across structured output domains.

Abstract

Learning from human feedback has shown success in aligning large, pretrained models with human values. Prior works have mostly focused on learning from high-level labels, such as preferences between pairs of model outputs. On the other hand, many domains could benefit from more involved, detailed feedback, such as revisions, explanations, and reasoning of human users. Our work proposes using nuanced feedback through the form of human revisions for stronger alignment. In this paper, we ask expert designers to fix layouts generated from a generative layout model that is pretrained on a large-scale dataset of mobile screens. Then, we train a reward model based on how human designers revise these generated layouts. With the learned reward model, we optimize our model with reinforcement learning from human feedback (RLHF). Our method, Revision-Aware Reward Models ($\method$), allows a generative text-to-layout model to produce more modern, designer-aligned layouts, showing the potential for utilizing human revisions and stronger forms of feedback in improving generative models.

Figures (14)

• Figure 1: $\text{RARE}$ Method Overview Our method consists of three parts: (a) Collecting human revision sequences, (b) Training our reward model on sequence data, and (c) Using the reward model in an RLHF framework.
• Figure 2: Layouts Visualizations We use CLAY (left) as our pretraining dataset. PLay (middle) is a generative, text-conditioned layout model. We ask designers to edit layouts generated by PLay, leading to more modern, cohesive layouts (right).
• Figure 3: Result Comparison We compare layouts generated by a PLay model, a supervised finetuned model, and a model trained with RLHF with $\text{RARE}$. In these examples, RLHF w/ $\text{RARE}$ produces the most cohesive and aligned layouts.
• Figure 4: Figma Plugin Our Figma plugin renders a PLay layout with the corresponding text description. Designers are asked to revise the layout by adding, modifying, and deleting elements.
• Figure 5: Element Distribution The element distribution from PLay samples (left) becomes more diverse after revisions (right). Designers add more elements (mean number of elements increases from 11.02 to 13.05 after revisions), particularly images and labels.