Visual Bias in Simulated Users: The Impact of Luminance and Contrast on Reinforcement Learning-based Interaction

TL;DR

This work provides the first systematic analysis of how luminance and contrast affect behavior by training 247 simulated users using RL on pointing and tracking tasks, revealing critical insights into what RL-driven simulated users actually learn.

Abstract

Reinforcement learning (RL) enables simulations of HCI tasks, yet their validity is questionable when performance is driven by visual rendering artifacts distinct from interaction design. We provide the first systematic analysis of how luminance and contrast affect behavior by training 247 \RV{simulated users using RL} on pointing and tracking tasks. We vary the luminance of task-relevant objects, distractors, and background under no distractor, static distractor, and moving distractor conditions, and evaluate task performance and robustness to unseen luminances. Results show luminance becomes critical with static distractors, substantially degrading performance and robustness, whereas motion cues mitigate this issue. Furthermore, robustness depends on preserving relational ordering between luminances rather than matching absolute values. Extreme luminances, especially black, often yield high performance but poor robustness. Overall, seemingly minor luminance changes can strongly shape learned behavior, revealing critical insights into what RL-driven simulated users actually learn.

Figures (8)

• Figure 1: Illustration of the tasks. (a) Pointing task with a black background and white task-relevant objects. (b) Pointing task with a target (bright object) and a distractor (dark object). (c) Tracking task with a moving target and distractor right of it.
• Figure 2: Pointing performance. Note the different success rate scales. (Left) Performance across background-object luminance combinations. Lower background-object contrast yields better performance. (Middle) Performance across object-static-distractor luminance combinations. (Right) Performance across object-moving-distractor luminance combinations. High object-distractor contrast helps with static distractors, while distractor luminance has virtually no effect when the distractor moves. Background luminance is fixed to 0.25 in the presence of distractors.
• Figure 3: Tracking performance (lower distance = better). Note the different distance-to-target scales. (Left) Black background and object yield the best performance, with little overall deviation. (Middle) High contrast between objects and static distractors improves performance. (Right) In performance across object-moving-distractor luminance combinations, luminance has little effect. We recall that background luminance is fixed to 0.25 in the presence of distractors.
• Figure 4: Robustness across background-object luminance combinations for (left) pointing and (right) tracking. Agents trained on black-on-black achieve high training performance but exhibit poor robustness. Robustness improves when the relational luminance ordering learned during training is preserved: for heatmaps in the upper left, in each individual heatmap the best performance is also in the upper left; analogously for heatmaps in the lower right.
• Figure 5: Robustness across static distractor–object luminance combinations for pointing. Agents trained with object luminance similar to the background (0.25) show higher robustness, generalizing well to evaluation conditions with luminances close to those seen during training.