Google, Microsoft, and OpenAI Pledge Support for Unified AI Scraping Controls
Major tech companies have pledged to support a shared robots.txt signal allowing publishers to block data scraping for AI training.
Tech companies Google, Microsoft, OpenAI, and Anthropic have pledged to support a new common standard in robots.txt files, designed to allow websites to block data crawling used for training artificial intelligence models, tech outlet Engadget reported.
According to a report by Unite.AI, Google plans to implement the shared control mechanism to provide content creators and web publishers with clearer oversight regarding how their material is used online. The initiative is intended to establish an agreed-upon, unified industry signal, replacing the disparate controls currently maintained by individual companies.
Further details regarding the timeline for the standard's full implementation and its precise technical specifications have not yet been released.