cotomi Act: Learning to Automate Work by Watching You

Abstract

What if a browser agent could learn your work simply by watching you do it? We present cotomi Act, a browser-based computer-using agent that combines reliable multi-step task execution with persistent organizational knowledge learned from user behavior. For execution, an agent scaffold with adaptive lazy observation, verbal-diff-based history compression, coarse-grained actions, and test-time scaling via best-of-N action selection achieves 80.4% on the 179-task WebArena human-evaluation subset, exceeding the reported 78.2% human baseline. For organizational knowledge, a behavior-to-knowledge pipeline passively observes the user's browsing and progressively abstracts it into artifacts (task boards, wiki) exposed through a shared workspace editable by both user and agent. A controlled proxy evaluation confirms that task success improves as behavior-derived knowledge accumulates. In our live demonstration, attendees interact with the system in a real browser, issuing tasks and observing end-to-end autonomous execution and shared knowledge management.

Figures (4)

• Figure 1: Unlike conventional computer-using agents that execute tasks with no knowledge of the user's organization, cotomi Act continuously learns from ordinary browsing. The Behavior Logger (top) monitors user activity and distills it via an agentic ETL pipeline into structured artifacts---tasks, timelines, and wiki pages---stored in a shared Knowledge Workspace (bottom) that both the user and the agent can read and edit. The Agent (right) consults these artifacts during its ReAct loop, acting as a situated co-worker rather than a stateless executor.
• Figure 2: Observation and history design trajectory on WebArena-Verified (Gemma-4-31B-IT, 82 tasks, 3 runs). Each arrow traces one design change in accuracy--token-cost space. Applying verbal diffs to the current observation improves accuracy, while using them to replace stale history reduces cost, jointly moving the scaffold toward the upper-left (higher accuracy, lower cost).
• Figure 3: Bidirectional workspace curation in action. The agent reviews the user's recent browsing sessions, proposes a task status update (not_started $\to$ completed), and the user approves the change---keeping the shared workspace aligned with reality.
• Figure 4: Change in success rate relative to the zero-coverage baseline (i.e., the agent uses no behavioral knowledge) as domain coverage of user behavior increases. More coverage improves downstream task success, while the most effective format depends on how much user behavior is available.