Is Cloudflare stopping AI from recommending your business?
[What Cloudflare's new Disallow AI Training setting changes, and how to check your site in five minutes]
Cloudflare’s new Disallow AI Training setting can stop a small business being recommended by AI, so most small businesses are better off leaving AI training switched on. A website that AI models can learn from and look up has more chances of being named when someone asks ChatGPT or Gemini for a local recommendation.
On 15 September 2026, Cloudflare changed how its AI crawler settings work. If your website runs through Cloudflare, it may already be on a setting you never picked. Here’s what changed, why it hits small businesses harder than big brands, and how to check your own site in about five minutes.
Key takeaways
- Cloudflare’s Disallow AI Training setting keeps your site in Google Search but opts it out of AI training.
- Opting out of Google’s AI training also stops Gemini using your pages as a cited source.
- Sites that had Cloudflare’s old “Block AI Bots” option switched on were moved to Disallow AI Training automatically.
- For most local service businesses, I recommend leaving the Training setting on Allow.
What Cloudflare changed in September 2026
Some of the biggest crawlers on the web do two jobs at once. Googlebot, Bingbot and Applebot collect pages for search results, and the same visits feed AI training. Until September, refusing one meant refusing the other.
Cloudflare’s answer is a setting called Disallow AI Training. It keeps your site available for search while refusing AI training. Apple, Google and Microsoft either honour it already or have committed to a date for doing so, and Cloudflare labels all three “Accountable”.
Your old settings carried over automatically
The part most business owners will miss is that old settings carried over on their own. If your site had the old “Block AI Bots” option switched on, Cloudflare moved your Training setting to Disallow AI Training. Nobody had to click anything. A business owner who ticked a box last year to “keep AI away” may now be opted out of Gemini without knowing it.
The four Training options explained
| Training setting | What it does | Effect on search |
|---|---|---|
| Allow | Lets all crawlers in, unless another rule blocks them | None |
| Disallow AI Training | Adds a no-training rule to your robots.txt and blocks training-only crawlers from OpenAI, Anthropic, Meta and Amazon | Stays in Google, Bing and Apple search |
| Block on pages with ads | Blocks all training crawlers, including Googlebot, on pages that show adverts | Those pages can drop out of search |
| Block | Blocks all training crawlers, including Googlebot, Bingbot and Applebot | Your site can drop out of those search engines |
One more detail: Bing doesn’t read the no-training rule from robots.txt yet. Microsoft is building that support and is aiming for early 2027, so for now Disallow AI Training doesn’t pass your preference to Bing.
What this means for small businesses
Why Disallow AI Training affects Gemini
The catch sits inside Google’s own rules. Disallow AI Training works partly by opting you out of Google-Extended, which is the control Google uses for Gemini. According to Google’s crawler documentation, Google-Extended covers both training Gemini and grounding its answers. Google’s developer guide for Gemini states that pages disallowing Google-Extended are not used for grounding.
Grounding is when Gemini looks up live web pages to answer a question and cites them. So your site keeps its Google rankings but can disappear from the sources Gemini names. Google says Google-Extended has no effect on Search rankings, and that’s true. It just isn’t the whole picture.
Why small businesses lose more than big brands
A national chain is mentioned all over the web: news stories, review sites, directories, forums, comparison pages. If it blocks AI training on its own site, AI models still learn about it from everywhere else.
A plumber in Ashford is in a different position. Their website is often the only detailed description of what they do, where they work and what they charge. Hide that from AI and the model has very little to go on. When someone asks Gemini for a reliable plumber nearby, the answer goes to a business the AI knows more about.
What about ChatGPT?
ChatGPT works slightly differently. OpenAI’s crawler documentation splits GPTBot, which collects training data, from OAI-SearchBot, which powers ChatGPT search, and treats the two separately. Disallow AI Training blocks GPTBot, so your pages can still appear in ChatGPT search results. What ChatGPT already “knows” about your business before it searches still comes from training, though, and a small business has less of that to spare.
When blocking AI training does make sense
Blocking AI training is a fair choice for some websites. It suits:
- Publishers whose articles are the product
- Paid courses and members-only content
- Original research that took real money to produce
- Ad-funded sites that only earn when a person visits the page
Cloudflare’s own defaults follow the same logic. For new websites that make money from ads, it suggests Disallow AI Training. For websites that don’t, it suggests leaving Training on Allow.
Most local service businesses sit in the second group. A builder, accountant or salon wants to be found and named. Nobody is paying to read their service pages, so there’s little to protect and plenty to lose, which is why our recommendation below points the other way.
How to check your Cloudflare AI crawler settings
It only takes about five minutes and needs no code.
- Log in to your Cloudflare account at dash.cloudflare.com.
- Select your website’s domain.
- Open Security, then Settings, and find the AI crawler controls. You’ll see three: Search, Training and Agent.
- Look at what Training is set to.
- Check that Search is set to Allow.
What your setting means
If Training says Allow, AI models can learn from your site. That’s what we recommend for most small businesses.
If it says Disallow AI Training, your site is still in Google, but Gemini won’t use it as a cited source. Switch it to Allow unless you fit one of the groups above.
If it says Block, change it today. Block now stops Googlebot, Bingbot and Applebot as well, so it can take your site out of search results altogether.
Not sure whether your website uses Cloudflare? Ask your web designer or whoever manages your domain. They’ll know in seconds.
If you’d rather someone else looked, we check crawler settings as part of a free GEO audit and explain what we find in plain English. For the bigger picture on AI visibility, see how getting recommended by AI differs from ranking in Google.
Frequently asked questions
Does ChatGPT use my website to train its models?
It can. OpenAI uses a crawler called GPTBot to collect web content that may be used to train its models. You can opt out by disallowing GPTBot in your robots.txt file, or by choosing Disallow AI Training in Cloudflare. Opting out of training doesn’t remove you from ChatGPT search, which uses a separate crawler called OAI-SearchBot.
Is blocking AI crawlers free?
Yes. Cloudflare’s Search, Training and Agent controls are available on every Cloudflare plan, including the free one. Editing your robots.txt file to opt out of individual AI crawlers costs nothing either. The real cost isn’t money. It’s the AI visibility a small business can lose by opting out without understanding what the setting does.
Do I need Cloudflare to control AI crawlers?
No. Any website can ask AI crawlers to stay away using its robots.txt file, and the major operators, including Google, OpenAI and Apple, publish the names to use. The difference is enforcement. A robots.txt file is a request that a crawler can ignore. Cloudflare can identify crawlers and block the ones that don’t follow your rules.
What’s the difference between AI search and AI training?
AI training is when a company uses web content to build or improve an AI model, which shapes what that model knows. AI search is when a tool like ChatGPT or Gemini looks up live web pages to answer a question and often cites them. The two are controlled separately, but opting out of Google’s training also opts you out of Gemini’s cited sources.
Can I change my mind later?
Yes. You can change your Cloudflare AI crawler settings at any time from the dashboard. Crawlers pick up the new rules the next time they visit your site. OpenAI, for example, says ChatGPT search can take around 24 hours to reflect a robots.txt change. Changing the setting affects future crawling, so it won’t pull your content back out of models that have already been trained.