Did somebody say "Gest-IT"? A pilot exploration of multimodal data management

TL;DR

A pilot exploration of the construction, management and analysis of a multimodal corpus is presented and a unified CoNLL-U corpus is proposed to investigate the variation of gesture-making patterns in conversations between sighted people and people with visualimpairment.

Abstract

The paper presents a pilot exploration of the construction, management and analysis of a multimodal corpus. Through a three-layer annotation that provides orthographic, prosodic, and gestural transcriptions, the Gest-IT resource allows to investigate the variation of gesture-making patterns in conversations between sighted people and people with visual impairment. After discussing the transcription methods and technical procedures employed in our study, we propose a unified CoNLL-U corpus and indicate our future steps

Figures (4)

• Figure 1: Room setting for the recording sessions, Logitech Brio Stream Webcam were employed for recordings
• Figure 2: Recording scenarios: unmasked and masked, same and different sight conditions
• Figure 3: ELAN tiers showing verbal and gestural transcriptions.
• Figure 4: CoNLL-U extract