No Evidence for LLMs Being Useful in Problem Reframing

TL;DR

This study evaluates whether large language models can aid designers in problem reframing by comparing manual framing with direct, structured, and free-form LLM usage across three design problems in a large online experiment (N=456, with 280 frames evaluated by 15 experts). Across four conditions, the authors measure frame novelty, usefulness, designer agency/ownership, and perceived helpfulness of LLMs, using expert ratings, a competence quiz, CSI, and NASA-TLX. The primary finding is that LLMs do not improve the quality of problem frames and can widen the competence gap between expert and novice designers, though expert designers may derive greater agency and generate more novel frames when using LLMs, particularly under structured usage. The results suggest that LLMs are not ready to replace designer reasoning in problem reframing, but can function as auxiliary tools to illuminate problem space or surface prior ineffective solutions, guiding future research toward more targeted, competence-sensitive AI-assisted design practices.

Abstract

Problem reframing is a designerly activity wherein alternative perspectives are created to recast what a stated design problem is about. Generating alternative problem frames is challenging because it requires devising novel and useful perspectives that fit the given problem context. Large language models (LLMs) could assist this activity via their generative capability. However, it is not clear whether they can help designers produce high-quality frames. Therefore, we asked if there are benefits to working with LLMs. To this end, we compared three ways of using LLMs (N=280): 1) free-form, 2) direct generation, and 3) a structured approach informed by a theory of reframing. We found that using LLMs does not help improve the quality of problem frames. In fact, it increases the competence gap between experienced and inexperienced designers. Also, inexperienced ones perceived lower agency when working with LLMs. We conclude that there is no benefit to using LLMs in problem reframing and discuss possible factors for this lack of effect.

Figures (15)

• Figure 1: Problem reframing helps designers attack thorny design problems by rethinking what they are about. In this paper, we study whether LLMs can help designers arrive at better problem frames.
• Figure 2: A structured problem-reframing workflow based on Dorst's reframing process dorst:book:reframing. From the starting point of a description of problems (in the yellow box), intermediate content (in the white boxes) is generated, to lead to diverse problem frames (in the green box). We tested this flow as one of the approaches to LLM use in this study.
• Figure 3: Compared to the manual approach where designers reframe problems by themselves (left), We tested three potential ways of using LLMs in problem reframing found in the literature about co-creation with LLMs. In the direct approach, designers can build on LLM-generated problem frames only. In the free-form approach, designers can freely ask LLMs to perform the tasks they require in their own process. In the structured approach, designers can also build on the intermediate content that LLMs generate following Dorst's nine-step reframing process dorst:book:reframing.
• Figure 4: We conducted an empirical study with 456 participants. Prior to the main task, all participants went through the same introduction, pre-survey, and quiz. Each was randomly assigned an approach and instructed to reframe a problem. The study concluded with a post-survey. We collected 456 problem frames and filtered out the low-quality responses. In phase 2, 280 frames were evaluated, by 15 expert designers.
• Figure 5: Using LLMs in problem reframing increased neither the novelty nor the usefulness of the problem frames, as evaluated by expert designers.