HIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices

TL;DR

HIDAgent presents an open-source hardware/software toolkit that externalizes UI agent control by emulating HID inputs on target devices, enabling agents to observe via HDMI screen capture and act across mobile and desktop platforms. The architecture uses three off-the-shelf components plus a Python library to support flexible agent development, including GUI recognition, multi-model LM processing, and cross-platform compatibility. Five use-case prototypes illustrate capabilities such as extensible UI agents, screen-reader-friendly interfaces, data-collection pipelines, cross-device interactions, and assistive observer roles, highlighting both practical potential and current limitations like high-resolution performance. This work advances research in Personal Agents by enabling agent control on devices where software installation is restricted or undesirable, promoting research into new interaction paradigms, accessibility applications, and cross-device workflows across heterogeneous devices.

Abstract

UI Agents powered by increasingly performant AI promise to eventually use computers the way that people do - by visually interpreting UIs on screen and issuing appropriate actions to control them (e.g., mouse clicks and keyboard entry). While significant progress has been made on interpreting visual UIs computationally, and in sequencing together steps to complete tasks, controlling UIs is still done with system-specific APIs or VNC connections, which limits the platforms and use cases that can be explored. This paper introduces HIDAgent, an open-source hardware/software toolkit enabling UI agents to operate HID-compatible computing systems by emulating the physical keyboard and mouse. HIDAgent is built using three off-the-shelf components costing less than $30 and a Python library supporting flexible integration. We validated the HIDAgent toolkit by building five diverse use case prototypes across mobile and desktop platforms. As a hardware device, HIDAgent supports research into new interaction scenarios where the agents are separated from the devices they control.

Figures (1)

• Figure 1: HID space to pixel space calibration is done by observing the cursor's movement in response to HID mouse move events. In this example, the quite faint cursor of the Apple iPhone screen is shown before (i) and after (ii). The pixel locations that were automatically detected to have changed are shown in the third image marked with red x's (iii), which includes pixel changes detected in the clock icon because the seconds hand has moved. To differentiate which changes are important, HIDAgent moves the cursor only in one direction at a time and looks for changes only in that direction. In this case, left to right with no vertical movement.