Four AI labs agree on one robots.txt token letting publishers opt out of training in a single line

International Technology · Oct. 30, 2026 · OpenAI, Google, Microsoft and Anthropic announced on Friday that they will honor a single shared robots.txt token letting a website publisher opt its content out of AI model training across all four companies’ crawlers in one directive, instead of setting separate rules for each company. It is the first time the four major Western labs have converged on one opt-out standard for training data.

One line: __usage_ai-training

The token collapses the previous stack into one line: User-Agent: __usage_ai-training followed by Disallow: /, with Allow lines available for carve-outs. Previously, publishers had to set different directives for each company’s crawler. The mechanism comes from an individual Internet-Draft at the Internet Engineering Task Force titled “Machine-Readable Training Preference Signal,” authored by Anthropic technical lead for standards Kevin Kelley and dated Oct. 30, 2026, with an expiry of May 3, 2027.

Implementation timelines differ

Rollout timing is not uniform: Anthropic has already begun implementing support, OpenAI says it aims to implement the signal in the coming months, Microsoft targets Feb. 1, 2027, and Google says 2027. Meta expressed support for the idea but has not committed to honoring the token. In other words, the token is a shared commitment from the four labs for now, with actual coverage arriving as each company ships support.

Compiled from two Oct. 31 AI industry briefings.