Indexing Analytics to Instances: How Integrating a Dashboard can Support Design Education

TL;DR

In design education, instructors struggle to provide timely, meaningful feedback on open-ended, multiscale design work. The authors build a research artifact that integrates AI-based multiscale design analytics with a live design environment and study it across five courses using a Research through Design approach, collecting qualitative data from 236 students and 9 instructors. They find that linking analytics to specific design instances via indexical visualizations improves mutual intelligibility, supports pedagogical action, and enhances assessment and student self-reflection, while acknowledging limitations where AI mislabels structures. The work demonstrates how context-aware, annotated analytics can scale design education practices and informs future dashboard design and broader educational analytics efforts.

Abstract

We investigate how to use AI-based analytics to support design education. The analytics at hand measure multiscale design, that is, students' use of space and scale to visually and conceptually organize their design work. With the goal of making the analytics intelligible to instructors, we developed a research artifact integrating a design analytics dashboard with design instances, and the design environment that students use to create them. We theorize about how Suchman's notion of mutual intelligibility requires contextualized investigation of AI in order to develop findings about how analytics work for people. We studied the research artifact in 5 situated course contexts, in 3 departments. A total of 236 students used the multiscale design environment. The 9 instructors who taught those students experienced the analytics via the new research artifact. We derive findings from a qualitative analysis of interviews with instructors regarding their experiences. Instructors reflected on how the analytics and their presentation in the dashboard have the potential to affect design education. We develop research implications addressing: (1) how indexing design analytics in the dashboard to actual design work instances helps design instructors reflect on what they mean and, more broadly, is a technique for how AI-based design analytics can support instructors' assessment and feedback experiences in situated course contexts; and (2) how multiscale design analytics, in particular, have the potential to support design education. By indexing, we mean linking which provides context, here connecting the numbers of the analytics with visually annotated design work instances.

Figures (4)

• Figure 1: Multiscale design produced by a student in instructor I4's human-computer interaction project. The student organized different phases of the project as clusters of design elements, nested across scales of magnification. At the outer scale, the crown, its caption, and spikes connected with each phase together form an overall cluster. In the bottom left, we see the view as the user zooms in. Elements deeply nested in one scale become legible at the next. We observe nine nested clusters at the inner scale.
• Figure 2: Each multiscale design document— a free-form web curation (FFWC)— is stored as JSON jain2021recognizing. It is comprised of properties such as title, description, creator, and a collection of elements. Each element includes a transforms property— with sub-properties position, scale, and rotation— which allows determining the element's spatial location with respect to the origin. Each element also stores a clipping property, which in turn stores semantics extracted at the time of collecting the element.
• Figure 3: We integrated a dashboard with the underlying design environment with the goal of conveying the meaning of analytics to users. Clicking an analytic presented on the dashboard interface opens the actual design environment and shows the corresponding AI identified nested scales, with all clusters at a particular scale rendered in the same background color. In the above figure, the outermost scale comprises one cluster— including all design elements— which is rendered in yellow color. The next inner scale has three clusters (one at top and two at bottom), which are rendered in blue. The innermost scale has three clusters— within the top blue cluster— which are rendered in brown. The visualization makes relationships visible, between particular design element assemblages and analytics that describe and measure them. It enables instructors to understand what the analytics mean.
• Figure 4: Another example of scale and cluster recognition. The outermost scale comprises one cluster— including all design elements— which is rendered in yellow color. The next inner scale has seven clusters, which are rendered in blue. However, AI is not perfect. Here, the two clusters rendered in brown are incorrectly recognized to be at the next inner level, as their content is similarly legible as the content within the clusters rendered in blue.