Umwelt: Accessible Structured Editing of Multimodal Data Representations

TL;DR

Umwelt introduces an accessible authoring environment that de-centers the visual modality by deriving visualization, textual description, and sonification from a shared abstract data model. A structured editor, coupled with a linked viewer, enables BLV users to rapidly prototype and compare multimodal representations, with default heuristics guiding initial configurations and traversal-based sonification. User studies with expert BLV participants show that multimodal representations offer complementary overviews and details, sustained by synchronized predicates that coordinate across modalities, while the editor reduces semantic and articulatory distance in representation creation. The work advances independent, self-guided data exploration for BLV analysts and motivates future research on native non-visual modalities and collaborative, interdependent access.

Abstract

We present Umwelt, an authoring environment for interactive multimodal data representations. In contrast to prior approaches, which center the visual modality, Umwelt treats visualization, sonification, and textual description as coequal representations: they are all derived from a shared abstract data model, such that no modality is prioritized over the others. To simplify specification, Umwelt evaluates a set of heuristics to generate default multimodal representations that express a dataset's functional relationships. To support smoothly moving between representations, Umwelt maintains a shared query predicated that is reified across all modalities -- for instance, navigating the textual description also highlights the visualization and filters the sonification. In a study with 5 blind / low-vision expert users, we found that Umwelt's multimodal representations afforded complementary overview and detailed perspectives on a dataset, allowing participants to fluidly shift between task- and representation-oriented ways of thinking.

Figures (6)

• Figure 1: An analyst's workflow in Umwelt: 1--6 illustrate an analyst's process of creating an initial multimodal data representation (shown in 7-9). 10 shows their initial exploration, before (11) making edits and then (12--14) continuing their analysis .
• Figure 2: The Umwelt interface. A) The data, visual, and audio tabs of the editor. B) The editor's fields tab, where users specify field definitions and encodings. C) The viewer, where users analyze data with interactive multimodal data representations.
• Figure 3: A) Fragments of an internal declarative specification shown next to their corresponding Umwelt editor states. B) The output multimodal representation for that specification.
• Figure 5: Prototypes of encoding- and field-oriented specifications of a scatterplot with concatenated sonification, illustrating the role-inexpressivenessblackwell_cognitive_2001 of field-oriented textual specification. Color-coded spans on the left side of each text prototype show the lines of code that pertain to each modality: green represents visual while blue represents audio. A) Encoding-oriented specification groups each modality into unit specifications. B) Field-oriented specification fragments each unit's encodings across the spec.
• Figure 6: An example of linked interaction across modalities, driven by the textual modality. Navigating to a node in the textual structure emits a query predicate. The visualization reifies this predicate as a conditional encoding, and the sonification reifies it as a filter.