Computational Trichromacy Reconstruction: Empowering the Color-Vision Deficient to Recognize Colors Using Augmented Reality

TL;DR

This work tackles color naming for color vision deficiency by transforming the problem from pure discrimination to learnable recognition. It introduces a smartphone AR app that rotates colors in linear RGB about the gray axis via user swipes, creating a temporal 3D color space that augments the native 2D percept. Psychophysical experiments show that rotational shifts have discriminative power, and a 9-day longitudinal study demonstrates learning, generalization to unseen colors, and durable recall. Real-world tasks, including Lego-block sorting and interpretation of AI-generated art, indicate practical usability and potential for daily-life assistance. The paper discusses design considerations, extensions to head-mounted displays, and the need for broader, geographically distributed studies to validate impact at scale.

Abstract

We propose an assistive technology that helps individuals with Color Vision Deficiencies (CVD) to recognize/name colors. A dichromat's color perception is a reduced two-dimensional (2D) subset of a normal trichromat's three dimensional color (3D) perception, leading to confusion when visual stimuli that appear identical to the dichromat are referred to by different color names. Using our proposed system, CVD individuals can interactively induce distinct perceptual changes to originally confusing colors via a computational color space transformation. By combining their original 2D precepts for colors with the discriminative changes, a three dimensional color space is reconstructed, where the dichromat can learn to resolve color name confusions and accurately recognize colors. Our system is implemented as an Augmented Reality (AR) interface on smartphones, where users interactively control the rotation through swipe gestures and observe the induced color shifts in the camera view or in a displayed image. Through psychophysical experiments and a longitudinal user study, we demonstrate that such rotational color shifts have discriminative power (initially confusing colors become distinct under rotation) and exhibit structured perceptual shifts dichromats can learn with modest training. The AR App is also evaluated in two real-world scenarios (building with lego blocks and interpreting artistic works); users all report positive experience in using the App to recognize object colors that they otherwise could not.

Figures (11)

• Figure 1: Illustration demonstrating color naming confusion for CVD individuals and how our system resolves such confusion. The xy-chromaticity space in \ref{['fig:idea_f']} can be obtained by a $3\times3$ linear transformation of the 3D LMS space in \ref{['fig:idea']} followed by a perspective projection. The xy diagram is commonly used in color science, because it conveniently allows us to visualize colors in a 2D representation sharma2017digitalchromaticitytutorial. Note that the Deuteranopia confusion lines, which are parallel to the M-axis in the LMS space, are now converging in the xy space.
• Figure 2: User swipes the finger to shifts colors of objects in the physical world. Initially confusing colors have different shift patterns (bottom blow-ups), which the user learns over time and uses to name/recognize colors. The slider at the top indicates the amount of shift applied, which our users report very useful for learning color naming. The button at the bottom right resets the shift to the initial colors.
• Figure 3: \ref{['fig:rot_rgb']} -- \ref{['fig:rot_xy']}: Rotating colors about the gray axis cycles the colors through different hues. Notice how the sRGB gamut clips the trajectories. \ref{['fig:intuition']}: Under rotational shifts, initially confusing colors are uniquely recognizable: colors on the opposite side of the iso-chrome lines (e.g., A and B) shift in opposite directions, and colors on the same side of the iso-chrome lines (e.g., B and C) shift in the same direction but with different amounts. The iso-chrome line contains all the colors a Deuteranope can see brettel1997computerizedjudd1948color (disregarding luminance variations, which is inherent in the chromaticity representation), so the intersection of a confusion line and the iso-chrome line represents the color that a dichromat actually sees when presented with all the colors on the confusion line. For instance, a Deuteranope sees colors A, B, C as the color $\alpha$.
• Figure 4: \ref{['fig:pilot_setup']} A trial in the discrimination task. There are four patches, three of which have the same base color and a randomly placed patch has the odd color. The participant is asked to identify the odd color (4AFC). In the first half of the study, they use the left/right arrow keys to induce color shifts, which are not available in the second half of the study. \ref{['fig:staircase_new']} An example of the 1-up-2-down staircase procedure to narrow down the color-discrimination threshold. As a participant completes a 4AFC sequence in \ref{['fig:pilot_setup']}, the staircase procedure automatically adjusts the odd color to make it harder if the participant identifies the odd color correctly and vice-versa. The $y$-axis is a relatively measure of the distance between the current color and the base color; 0 means the base color. \ref{['fig:pilot_example']} For a given base color (left), we experimentally obtain eight discrimination thresholds without using the shifts (top) and eight thresholds while using the shifts (bottom).
• Figure 5: Color discrimination results averaged across Deuteranomaly participants (N=8). We have four base colors, each of which has two sets of discrimination thresholds, one without using the color shifts and the other using the shifts. For each set of thresholds, we regress an ellipse inspired by the classic MacAdam's ellipses macadam1942visual for modeling color discrimination thresholds. The discrimination threshold reduction is statistically significant for green, gray, and blue base colors.