SAN FRANCISCO — Four of the world's largest artificial intelligence companies announced on 30 October 2026 that they will honour a single, shared robots.txt policy token letting website publishers opt out of AI model training across all four of their web crawlers in one directive, a move that could simplify a fragmented opt-out landscape that has frustrated publishers since the generative AI boom began.
OpenAI, Google, Microsoft and Anthropic said they will recognise the token, written as __usage_ai-training, in publishers' robots.txt files. A publisher that adds the directive signals that it does not permit its content to be used for training AI models, and all four companies have committed to respecting that signal rather than requiring a separate opt-out for each crawler. The announcement was reported on 30 October 2026, the same day the underlying proposal was published as an Internet-Draft at the Internet Engineering Task Force (IETF). Until now, a publisher wanting to keep its work out of AI training data typically had to implement a different mechanism for each AI company — a patchwork smaller publishers have said was time-consuming and easy to get wrong.
Illustration: A stylised publisher website with a shield and toggle symbolising a single opt-out from AI model training.
What Happened
On 30 October 2026, OpenAI, Google, Microsoft and Anthropic announced their support for the shared token __usage_ai-training, according to reporting published the same day. The token is defined in an IETF Internet-Draft authored by Kevin Kelley, Anthropic's standards lead, and dated 30 October 2026. The draft was submitted to the IETF's AIPREF working group, which focuses on AI preferences, as an individual draft rather than as a formal standard adopted by the working group.
Under the proposal, a publisher adds the token to its robots.txt file, the plain-text file websites use to communicate with automated crawlers. The token signals a usage preference specifically about AI training: that the site's content should not be used to train AI models. Timelines differ. Anthropic is already implementing support. OpenAI said it will support the token in the coming months. Microsoft said it will support the token from 1 February 2027, alongside separate controls covering training and grounding, the process by which AI systems draw on retrieved content when answering questions. Google said it will support the token in 2027, treating it as if its existing Google-Extended control were disallowed, unless Google-Extended itself is set directly by the publisher.
Meta, the parent company of Facebook and Instagram, has also signalled support for the shared approach, according to the reporting, although it was not among the four companies in the joint announcement. That breadth matters because a shared signal is only useful if the companies whose crawlers visit publishers' sites actually respect it.
Importantly, the token governs training permission, not crawling or search indexing. A site using __usage_ai-training is not blocking crawlers from visiting its pages, nor removing itself from search results. Publishers wanting to control search indexing still need their existing robots.txt directives for that purpose.
Background
The announcement follows several years of tension between AI developers and the publishers whose content has fuelled the growth of large language models. When generative AI surged into public use, publishers discovered their articles, photographs and other work were being collected at scale to train models, often without a clear permission mechanism and without compensation. Some negotiated licensing deals; others turned to technical measures, including robots.txt directives and outright crawler blocks, to keep their content out of training datasets.
Robots.txt itself is not a law. It is a voluntary convention dating back to the early web, by which websites declare which automated agents may visit which parts of a site, and compliant crawlers respect those declarations. Its power therefore depends entirely on the good faith of the companies operating the crawlers — which is why the joint nature of this announcement is significant. Four major AI developers committing to the same token at the same time gives publishers a single point of leverage they did not previously have.
Each company had previously offered its own mechanism. Google's Google-Extended, for example, let publishers signal that their content should not be used for training Google's AI models, while remaining available for search. The result was a list of different tokens publishers had to track and maintain, with the risk that missing one company's mechanism left content exposed to that company's training pipeline. The shared token is intended to replace that patchwork for the companies that have signed on, while leaving each company's existing controls in place.
The proposal's IETF path also defines one of its clearest limits. The IETF is the standards body behind many of the internet's core protocols, and an Internet-Draft is an early-stage document in that process. Kelley's draft is an individual submission to the AIPREF working group: published for discussion, but not adopted as a formal standard, and carrying no enforcement power of its own. Its influence will depend on whether more companies and publishers adopt it in practice. It also does not remove content already collected or used in training, and it cannot by itself compel a company that has not committed to it to change its behaviour.
Illustration: A stylised robots.txt document beside a robot figure and an on-off switch, representing a machine-readable AI training signal.
Why It Matters
For publishers, the most immediate benefit is simplicity. A single directive that four major AI companies have pledged to honour reduces the burden of managing AI opt-outs and lowers the chance of an accidental gap. That is particularly valuable for smaller newsrooms, blogs and independent sites lacking dedicated technical staff to track each company's evolving crawler documentation. A shared, documented signal is also easier to audit: anyone can read a site's robots.txt file and see whether the token is present.
For the AI companies, the joint announcement is a gesture toward rebuilding trust with the content industries on which their models depend. Licensing disputes and copyright lawsuits have dogged the sector, and regulators in several countries have begun examining how training data is collected. A common, transparent opt-out gives the companies a concrete answer when publishers and policymakers ask how consent is handled, and may reduce pressure for more restrictive regulation.
For the wider web, the proposal could set a precedent. If the token gains traction, it may become the default way publishers express training preferences, just as robots.txt became the default way to express crawling preferences decades ago. Other AI developers could adopt the same token, extending its coverage, and the IETF process could eventually turn the draft into a more formal standard. But its limits should not be overlooked: it is a signal, not a lock, and its effectiveness rests on voluntary compliance.
The announcement also lands amid heightened scrutiny of the AI industry. Reuters reported on 30 October 2026 that the United States Securities and Exchange Commission is examining AI companies' corporate disclosures amid worries about a possible bubble in AI-related valuations. That reported scrutiny is separate from the robots.txt announcement, but it forms part of the same backdrop: an industry facing questions about how it collects data and how sustainable its current pace of growth really is.
What Happens Next
The rollout will be staggered. Anthropic is already implementing support, so publishers using the token should expect Anthropic's systems to honour it first. OpenAI's support is expected in the coming months, while Microsoft's is dated to 1 February 2027 and Google's to sometime in 2027. Publishers adding the token now should therefore not assume immediate, uniform coverage; the practical effect will build as each implementation goes live.
Publishers wanting to use the token will need to add it to their robots.txt files and check how it interacts with existing directives. Google's treatment is the most nuanced: it will treat the shared token as if Google-Extended were disallowed, unless a publisher sets Google-Extended directly, in which case the direct setting takes precedence. Microsoft's announcement adds a further layer, with separate training and grounding controls arriving alongside the shared token — a distinction that matters for publishers comfortable with their content grounding AI answers, where a system retrieves and cites a page, but not with it training the underlying model, or vice versa.
Beyond the four companies, the open questions are adoption and standardisation. Will Meta move from signalling support to formal adoption? Will other AI developers recognise the token? And will the IETF's AIPREF working group advance Kelley's draft toward a formal standard? The answers will determine whether __usage_ai-training becomes a genuine web-wide convention or remains a useful but partial arrangement among the largest players. For now, publishers have something they have long asked for: one clear, shared way to say no to AI training, backed by the four companies whose crawlers matter most.
Illustration: Abstract company buildings and publisher icons joined by a bridge, symbolising agreement between AI companies and publishers.
Sponsored: explore today's featured sponsored offers
FAQ
What is the __usage_ai-training token?
It is a shared robots.txt token letting a publisher signal, in one directive, that its content should not be used to train AI models. OpenAI, Google, Microsoft and Anthropic announced on 30 October 2026 that they will honour it. It is defined in an IETF Internet-Draft by Anthropic standards lead Kevin Kelley, dated 30 October 2026.
Which companies have committed to honouring the token?
OpenAI, Google, Microsoft and Anthropic have announced they will honour it. Meta has also signalled support, according to reporting on the announcement, though it was not among the four companies in the joint announcement. Timelines differ: Anthropic is already implementing it, OpenAI will follow in the coming months, Microsoft from 1 February 2027, and Google in 2027.
Does the token block crawlers or remove a site from search results?
No. It controls permission for AI training use only. It does not block crawling and does not affect search indexing. Publishers wanting to control search indexing still need their existing robots.txt directives.
Is the token a formal internet standard, and does it have enforcement power?
Not yet. It is an individual Internet-Draft submitted to the IETF's AIPREF working group, not a formal standard, and it carries no enforcement power. Its effect depends on voluntary compliance by the companies that have committed to honour it, and it does not remove content already collected or used in training.
How does Google treat the shared token compared with Google-Extended?
Google said it will treat the shared token as if Google-Extended were disallowed, unless a publisher sets Google-Extended directly, in which case the direct setting takes precedence. Publishers with existing Google-Extended settings should review both directives together.
Sources
- The Verge — reporting on the shared AI training opt-out token announcement, 30 October 2026
- Google Search Central blog — announcement of support for the shared token, 30 October 2026
- Microsoft Bing blog — announcement of support for the shared token, 30 October 2026
- OpenAI announcement — support for the shared token, 30 October 2026
- IETF Internet-Draft by Kevin Kelley (Anthropic) — __usage_ai-training proposal, submitted to the AIPREF working group, 30 October 2026
- Reuters — reporting that the US SEC is examining AI companies' corporate disclosures amid bubble worries, 30 October 2026
Comments
Post a Comment