See-Through Face Display for DHH People: Enhancing Gaze Awareness in Remote Sign Language Conversations with Camera-Behind Displays

TL;DR

The paper addresses the challenge of maintaining natural eye contact in remote sign language conversations by introducing a See-Through Face Display, which places a camera behind a transparent screen so interlocutors appear to gaze directly at each other. The authors implement a multi-party videoconferencing system using this display, compare gaze behavior to conventional videoconferencing, and discuss limitations such as resolution and flicker. They propose concrete future applications, including gaze-aware online environments, paired gaze–sign-language corpora, and AI-driven sign language avatars, underscoring the importance of collaboration with DHH communities for real-world deployment. Overall, the approach aims to preserve authentic gaze cues to improve turn-taking, engagement, and sign-language comprehension in remote settings, with potential impact on interpretation workflows and data collection for sign language research.

Abstract

This paper presents a sign language conversation system based on the See-Through Face Display to address the challenge of maintaining eye contact in remote sign language interactions. A camera positioned behind a transparent display allows users to look at the face of their conversation partner while appearing to maintain direct eye contact. Unlike conventional methods that rely on software-based gaze correction or large-scale half-mirror setups, this design reduces visual distortions and simplifies installation. We implemented and evaluated a videoconferencing system that integrates See-Through Face Display, comparing it to traditional videoconferencing methods. We explore its potential applications for Deaf and Hard of Hearing (DHH), including multi-party sign language conversations, corpus collection, remote interpretation, and AI-driven sign language avatars. Collaboration with DHH communities will be key to refining the system for real-world use and ensuring its practical deployment.

Figures (4)

• Figure 1: The screenshots of our videoconferencing software. (a) The user accesses the dashboard page to establish one client connection per conversational partner. (b) The client interface as it appears on the See-Through Face Display. For debugging purposes, the user can access the shared room, as well as control the camera and microphone on the See-Through Face Display. Since the display renders only the top-left 320$\times$360 pixels of the screen, it exclusively shows the remote user's face.
• Figure 2: (a) Example of a P2P mesh communication configuration in a three-party conversation. User A establishes P2P connections with Users B and C via Displays $D_{AB}$ and $D_{AC}$, respectively. Displays $D_{AB}$ and $D_{AC}$ show images of Users B and C captured by cameras positioned behind $D_{BA}$ and $D_{CA}$, respectively, while Displays $D_{BA}$ and $D_{CA}$ show images of User A captured by cameras behind $D_{AB}$ and $D_{AC}$. (b) User A is speaking; Users B and C are displayed separately in $D_{AB}$ and $D_{AC}$ and can make eye contact with each other.
• Figure 3: (a) In a typical videoconferencing setup, the user is in fact looking at their conversation partner, but the transmitted image depicts them as if they are looking away. (b-1) In See-Through Face Display, for any conversation partner the user is not looking at, the video stream appears as though the user is looking elsewhere. (b-2) Conversely, for the partner the user is actually looking at, the video stream appears as though the user is looking directly at them.
• Figure 4: Implementation of eye contact communication with AI avatars by Izumi et al. (reproduced from izumi2025animegazerealtimemutualgaze). © ZEPTO002, licensed under VN3 license version 1.10.