Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions

TL;DR

SightationCounts addresses the gap between sighted annotation and BLV user needs by introducing Sightation, a large-scale BLV-aligned diagram-description dataset built with latent supervision and validated by BLV educators. The authors implement a two-pass guided generation pipeline that produces a guide of question-answer pairs to steer diagram descriptions, and they collect extensive sighted and BLV evaluations across completions, preference, retrieval, VQA, and reasoning tasks. The dataset comprises 5k diagrams and 137k samples, and they demonstrate that fine-tuning and latent supervision yield measurable improvements across multiple tasks and metrics, including substantial gains for smaller models. Sightation is shown to bolster BLV accessibility training data, reduce annotator bias, and enhance model alignment to BLV needs, with broad implications for educational diagram descriptions and multimodal AI systems in accessibility contexts. The work also discusses limitations, ethical considerations, and directions for future improvements in supervision strategies and diagram detail capture.

Abstract

Often, the needs and visual abilities differ between the annotator group and the end user group. Generating detailed diagram descriptions for blind and low-vision (BLV) users is one such challenging domain. Sighted annotators could describe visuals with ease, but existing studies have shown that direct generations by them are costly, bias-prone, and somewhat lacking by BLV standards. In this study, we ask sighted individuals to assess -- rather than produce -- diagram descriptions generated by vision-language models (VLM) that have been guided with latent supervision via a multi-pass inference. The sighted assessments prove effective and useful to professional educators who are themselves BLV and teach visually impaired learners. We release Sightation, a collection of diagram description datasets spanning 5k diagrams and 137k samples for completion, preference, retrieval, question answering, and reasoning training purposes and demonstrate their fine-tuning potential in various downstream tasks.

Figures (10)

• Figure 1: The key benefit of utilizing sighted user feedback lies in their assessments, which are based on solid visual grounding. The compiled assessments prove an effective training substance for steering VLMs towards more accessible descriptions. Dataset use and the subsequent validation are described in Sec. \ref{['sec:perf']}. A complete list of use cases is provided in Appendix \ref{['app:rest']}.
• Figure 2: The qualities assessed by their respective groups.
• Figure 3: Tuning VLMs on Sightation enhanced various qualities of the diagram descriptions, evaluated by BLV educators, and shown here as normalized ratings averaged in each aspect. The capability of the dataset is most strongly pronounced with the 2B variant, shown above. Full results across 4 models and 22 metrics are reported in Tables \ref{['tab:big1a']}, \ref{['tab:big1b']}, \ref{['tab:big2B']}, and \ref{['tab:big7B']}.
• Figure 4: Percentage distribution of the quality of question-answer pairs in AI2D and SightationVQA
• Figure 5: Less can be more for BLV users. Our approach streamlines details to highlight the core information while emphasizing key details to increase information density and maximize information efficiency per unit length.