ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet
Yuekang Li, Wei Song, Bangshuo Zhu, Dong Gong, Yi Liu, Gelei Deng, Chunyang Chen, Lei Ma, Jun Sun, Toby Walsh, Jingling Xue
TL;DR
ai.txt addresses the lack of fine-grained governance for AI interactions with the Internet by introducing a domain-specific language that extends robots.txt to element-level regulation and supports natural language guidance. It pairs a formal syntax and an integrated development environment with an XML generator to encode permissions and agent instructions, plus two enforcement pathways: programmatic XML-based enforcement and prompt-level guidance embedded in prompts. Preliminary experiments and case studies suggest ai.txt is expressive and can improve compliance across diverse scenarios. The work offers practical tooling and a governance-oriented framework to regulate AI-Internet interactions in digital ecosystems.
Abstract
We introduce ai.txt, a novel domain-specific language (DSL) designed to explicitly regulate interactions between AI models, agents, and web content, addressing critical limitations of the widely adopted robots.txt standard. As AI increasingly engages with online materials for tasks such as training, summarization, and content modification, existing regulatory methods lack the necessary granularity and semantic expressiveness to ensure ethical and legal compliance. ai.txt extends traditional URL-based access controls by enabling precise element-level regulations and incorporating natural language instructions interpretable by AI systems. To facilitate practical deployment, we provide an integrated development environment with code autocompletion and automatic XML generation. Furthermore, we propose two compliance mechanisms: XML-based programmatic enforcement and natural language prompt integration, and demonstrate their effectiveness through preliminary experiments and case studies. Our approach aims to aid the governance of AI-Internet interactions, promoting responsible AI use in digital ecosystems.