Permission Manifests for Web Agents

TL;DR

The paper tackles the governance gap created by LLM-based web agents that perform complex UI interactions. It proposes agent-permissions.json, a lightweight, robots.txt–style JSON manifest that separates resource-level constraints from high-level action guidelines and promotes API-first interactions when available. The design emphasizes low friction, broad applicability, and compatibility with existing signaling efforts like AIPref, MCP, and A2A, while acknowledging non-enforceability and the need for layered enforcement. Practical implementations include a Python parsing library and an automated manifest generator, illustrating a feasible path toward widespread adoption that could reduce unnecessary blocking and enable compliant, beneficial automation on the web.

Abstract

The rise of Large Language Model (LLM)-based web agents represents a significant shift in automated interactions with the web. Unlike traditional crawlers that follow simple conventions, such as robots.txt, modern agents engage with websites in sophisticated ways: navigating complex interfaces, extracting structured information, and completing end-to-end tasks. Existing governance mechanisms were not designed for these capabilities. Without a way to specify what interactions are and are not allowed, website owners increasingly rely on blanket blocking and CAPTCHAs, which undermine beneficial applications such as efficient automation, convenient use of e-commerce services, and accessibility tools. We introduce agent-permissions.json, a robots.txt-style lightweight manifest where websites specify allowed interactions, complemented by API references where available. This framework provides a low-friction coordination mechanism: website owners only need to write a simple JSON file, while agents can easily parse and automatically implement the manifest's provisions. Website owners can then focus on blocking non-compliant agents, rather than agents as a whole. By extending the spirit of robots.txt to the era of LLM-mediated interaction, and complementing data use initiatives such as AIPref, the manifest establishes a compliance framework that enables beneficial agent interactions while respecting site owners' preferences.

Figures (2)

• Figure 1: agent-permissions.json is a minimalist, straightforward permission manifest for agents interacting with web pages.
• Figure 2: agent-permissions.json examples: resource rules (top), action guidelines (bottom left), and API references (bottom right).