The AI Giants' New 'Stop Sign' is a Corporate Handshake, Not a Paycheck
AI-generated image · US National Wire
Google, OpenAI, Anthropic, and Microsoft agree to a common scraping directive, but the move does nothing for creators already plundered.
OPINION: Let’s be clear about what is happening here. The tech titans are finally agreeing to a 'do not enter' sign for our data, but don't be fooled—this is a corporate handshake to carve up the internet while the actual creators are still left without a dime in royalties.
As Engadget first reported, Google, OpenAI, Anthropic, and Microsoft have committed to honoring a new robots.txt directive. This move allows publishers to signal their preferences regarding the scraping of their content for generative AI training. The companies are adopting a proposal from the IETF (the open standards organization for the internet) authored by Kevin Kelley, who serves as the technical lead for standards at Anthropic.
Under the current system, Engadget reports that website owners are forced to identify individual crawlers and create separate rules for every single provider. Kelley's proposal aims to streamline this by getting multiple AI firms to recognize one common signal. To implement this, publishers must create a specific user agent in their robots.txt file—labeled as `User-Agent: __usage_ai-training`—to block crawlers dedicated to AI training. The system also includes an "Allow" section, giving operators the choice to let AI scrape specific areas, such as a press kit.
However, the fine print reveals the limits of this "concession." Engadget notes that this directive does not stop crawling entirely, nor does it prevent content from appearing in Google search results. Specifically, Google will treat the signal as a stop sign for its "Google-Extended" user agent, which is used to gather materials for Gemini models.
Most egregiously, this new rule is not retroactive. Engadget highlights that the directive cannot undo any scraping that has already occurred; if the companies have already crawled a page, they keep the content.
As for the timeline of this rollout, Anthropic has already begun supporting the directive. OpenAI has pledged to implement the signal in the coming months. Microsoft and Google have both set their sights on 2027, with Microsoft committing to February 1, 2027, and Google stating support will arrive in early 2027.