OpenAI says it will honor a shared robots.txt AI training signal, giving website owners one common way to tell AI companies whether their content can be used to train foundation models. The company published the commitment on October 30, 2026, and says it aims to implement it “in the coming months.”
For publishers and SEOs, this could be the most practical change to AI crawler controls in years. Instead of tracking a growing list of AI user agents, sites may soon be able to state one training preference that several AI providers respect.
What OpenAI Announced
In a post titled “Making it easier for websites to express AI training preferences,” OpenAI said it will honor a common signal in robots.txt for the use of content in foundational generative AI training. The commitment is voluntary, and the signal is not live yet. OpenAI says it will post another update once it is.
OpenAI also stressed that existing controls are not going away. Current robots.txt rules, including rules aimed at its own crawlers such as GPTBot, “will continue to apply and be respected.”
How the robots.txt AI Training Signal Is Expected to Work
The signal builds on work at the IETF’s AI Preferences (aipref) working group, which is defining a standard vocabulary for how content may be used by AI systems. A related Internet-Draft posted on October 30 describes a special robots.txt group that applies to AI training rather than to any one crawler. In the draft’s example, a site could block training on the whole site while still allowing a specific folder, such as a press kit.
A few details from the draft are worth knowing:
- It controls training, not crawling. The signal is not a crawl directive. If you want to stop a bot from fetching your pages, you still need to block that bot by name.
- Wildcard rules don’t count. A general
User-agent: *group does not express a training preference on its own. - No signal means no preference. Sites that don’t add the group simply haven’t stated a preference.
- Specific rules may win. Provider-specific groups, like one for GPTBot, may take precedence over the shared signal.
Because the drafts are still works in progress, the final syntax could change before OpenAI and others switch it on. Don’t edit your robots.txt for this yet.
Who Else Is Backing It?
OpenAI’s post does not name other participants, but reports say Google and Microsoft also plan to support the shared signal, with timelines pointing to early 2027. Microsoft is reportedly adding its own page-level training and “grounding” preferences as well. Meta, Apple and Perplexity were not part of the announcements, according to coverage of the draft.
What This Means for SEO and Publishers
Today, opting out of AI training means managing separate rules for many bots, which we covered when Google started blocking AI scrapers. A shared signal could simplify that and separate two decisions that are often tangled together:
- Training: whether your content can be used to build future models.
- Search and retrieval: whether AI search tools can fetch and cite your pages, which still drives visibility in answers.
That split matters because AI search referrals are still small. As three recent studies showed, AI search sends only a tiny fraction of Google’s traffic, yet being cited in AI answers is becoming part of brand visibility. Many sites will want to block training while staying visible in AI search.
Open questions remain, including whether the signal applies to content already collected and how retrieval for answers will be treated. If you need a refresher on how bots discover pages, see our guide to how search engines crawl, index and rank.
Key Takeaways
- OpenAI will honor a shared robots.txt signal for AI training preferences, aiming to launch it in the coming months.
- The signal is based on IETF AI Preferences work and controls training only, not crawling.
- Existing robots.txt rules for GPTBot and other bots keep working.
- Google and Microsoft reportedly plan support in 2027; the final syntax is not settled.
Related: Anthropic’s 2026 usage policy update and the UK CMA’s probe into Meta’s publisher terms.
