Quick Answer: Should You Block AI From Training on Your Website?
You can stop Google from using your website content to train its Gemini AI models by adding a "Google-Extended" rule to your robots.txt file, and Google confirms this does not affect your inclusion or ranking in Google Search. On 30 October 2026, Google also announced it plans to support a new shared AI training control that works across several AI companies at once, which it aims to support in 2027. For most small businesses that want customers to find them through AI tools, the safest approach is to allow AI search crawlers and decide separately whether to allow training.
Last updated: November 2026. Rules and tools in this area are changing quickly, so check the official sources linked below before you change your robots.txt file.
Key Takeaways
- Google-Extended: a robots.txt token that controls whether Google may use your content to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
- No Search penalty: Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
- New shared control: Google announced on 30 October 2026 that it plans to support an industry-wide AI training "policy token" in robots.txt, aiming to support it with other AI providers in 2027.
- Search vs training are separate: OpenAI uses OAI-SearchBot for ChatGPT search results and GPTBot for model training, and each can be allowed or blocked on its own.
- robots.txt is not a privacy tool: it controls crawling, not indexing; to keep a page out of Google you need noindex or password protection.
What Did Google Announce About AI Training Controls in October 2026?
In a Google Search Central blog post published on 30 October 2026, Google said it plans to support a shared AI training control in robots.txt. Instead of writing separate rules for every AI company, website owners will be able to use one "policy token" to state their AI training preference to all participating providers.
The idea comes from the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF), which is chartered to standardise a vocabulary for expressing how content may be collected and used for AI, and ways to attach those preferences to content, including through the Robots Exclusion Protocol (robots.txt) and HTTP headers.
Google explained how it will read the new signal: if a site uses the shared token to opt out of AI training and does not mention Google-Extended separately, Google plans to treat that as if the site had blocked Google-Extended. Company-specific rules, such as Google-Extended, will keep working alongside the shared token, so you can set one broad preference and still give individual AI companies different instructions.
This is a plan, not something you can rely on today. Google says it aims to support the new control together with other AI providers in 2027 and will share updates as the standard develops.
What Is Google-Extended and What Does It Control?
According to Google's documentation on its common crawlers, Google-Extended is a standalone "product token" you can use in robots.txt. It controls whether content Google crawls from your site may be used to train future generations of Gemini models (which power Gemini Apps and the Vertex AI API for Gemini) and for grounding, which means supplying content from the Google Search index to the model when it answers a prompt.
- It is not a separate crawler: Google-Extended has no user agent string of its own; Google crawls with its usual user agents and reads the token only as an instruction.
- It does not change Google Search: Google states it does not affect your site's inclusion in Google Search and is not used as a ranking signal.
- It affects Gemini visibility: Google's October 2026 post notes that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
That last point is the real trade-off for a small business. Blocking Google-Extended keeps your content out of Gemini training and grounding, but it may also mean fewer chances to be cited and linked inside Gemini Apps.
How Do I Block AI Training in robots.txt? (Step by Step)
Your robots.txt file is a plain text file. Google's guide to creating a robots.txt file says it must be named robots.txt, you can have only one per site, and it must sit at the root of your domain (for example, https://www.example.com/robots.txt). Save it with UTF-8 encoding.
- Step 1: Open your current robots.txt by visiting yourdomain.com/robots.txt in a browser, and keep a copy before changing anything.
- Step 2: To block Google AI training only, add a group with "User-agent: Google-Extended" followed by "Disallow: /".
- Step 3: To block OpenAI model training but stay in ChatGPT search, add "User-agent: GPTBot" with "Disallow: /", and keep OAI-SearchBot allowed.
- Step 4: Leave your normal search crawlers, such as Googlebot, allowed, or your site could drop out of search results.
- Step 5: Upload the file to your site root, then reload yourdomain.com/robots.txt to confirm the new rules are live.
Example: "User-agent: Google-Extended / Disallow: /" on two lines blocks Gemini training and grounding for your whole site, without affecting Google Search.
If your site runs on a website builder or a managed platform, it may generate robots.txt for you. In that case, use the platform's settings or ask your web developer, rather than uploading a file that could be overwritten.
Will Blocking AI Crawlers Hurt My Visibility in ChatGPT and AI Search?
It depends on which crawler you block. OpenAI's crawler documentation explains that its bots are independent of each other. OAI-SearchBot is used to show websites in ChatGPT's search features, while disallowing GPTBot signals that your content should not be used to train OpenAI's generative AI foundation models.
- Block GPTBot, allow OAI-SearchBot: your content is not used for OpenAI model training, but you can still appear in ChatGPT search answers.
- Block OAI-SearchBot: OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links.
- Timing: OpenAI notes it can take about 24 hours after a robots.txt change for its search systems to adjust.
For a local business that wants new customers, being visible in AI search is usually more valuable than the content you protect by blocking it. One option is to block training-only crawlers while keeping search crawlers allowed. Either way, this is a business decision, not a technical requirement.
Does robots.txt Keep My Pages Private or Out of Google?
No. Google's introduction to robots.txt says the file mainly exists to avoid overloading your site with crawler requests, and it is not a way to keep a page out of Google. A blocked URL can still be indexed if other sites link to it.
- To keep a page out of search results: use a noindex meta tag or response header.
- To keep content private: put it behind a password or login.
- To remove content completely: take the page down.
robots.txt also relies on crawlers choosing to follow it. The IETF AIPREF working group's charter lists technical enforcement of preferences as out of scope, so treat AI preference signals as instructions to cooperating companies, not as security.
What Should a Small Business Do Right Now?
- Check your current robots.txt: confirm Googlebot and AI search crawlers are not blocked by accident.
- Decide on training separately from search: choose whether to allow Google-Extended and GPTBot based on how much you rely on AI-assistant visibility.
- Protect what matters properly: use noindex or passwords for pages that must stay out of search, not robots.txt alone.
- Keep creating useful, original content: Google's May 2026 guidance on generative AI features stresses valuable, unique, non-commodity content and says SEO best practices remain the foundation.
- Watch for the 2027 shared control: once Google and other providers support the shared token, you may be able to set one preference instead of many.
Google's announcement of its guide to optimising for generative AI in Search is a useful companion read: it covers why unique content matters, tips for local, shopping, image and video content, and myths about "AEO/GEO". If you want a broader foundation, see our complete guide to technical SEO and our local SEO guide for Louisville KY businesses.
Sources
- Google Search Central Blog: Supporting a shared AI training control for websites (30 October 2026) – developers.google.com/search/blog/2026/10/ai-training-control
- Google for Developers: Google's common crawlers (Google-Extended) – developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google Search Central: Introduction to robots.txt – developers.google.com/search/docs/crawling-indexing/robots/intro
- Google Search Central: Create and submit a robots.txt file – developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
- OpenAI: Overview of OpenAI crawlers – developers.openai.com/api/docs/bots
- IETF: AI Preferences (aipref) working group charter – datatracker.ietf.org/wg/aipref/about/
- Google Search Central Blog: A new resource for optimizing for generative AI in Google Search (May 2026) – developers.google.com/search/blog/2026/05/a-new-resource-for-optimizing
Conclusion
Blocking AI training is now a simple robots.txt decision, and Google confirms that blocking Google-Extended does not hurt your Google Search rankings. The bigger question for a small business is visibility: AI search tools are becoming a way customers discover local companies, so think carefully before blocking search crawlers. Allow the crawlers that bring you customers, decide separately about training, and revisit your choice in 2027 when Google and other AI providers plan to support the new shared control.