Cloudflare has clarified that its AI traffic controls can affect more than specialist AI crawlers. Under the company's updated policy, a site that blocks Training may also block mixed-use crawlers such as Googlebot, Applebot and BingBot when they visit pages that serve ads. For businesses that rely on organic search, the important distinction is no longer simply whether AI bots are blocked. It is whether the selected policy preserves search discovery while restricting AI training.
Cloudflare announced the expanded approach on September 15, 2026, replacing the earlier one-click "Block AI Bots" model with separate controls for Search, Agent and Training. Its official explanation of accountable mixed-use AI crawlers says Google, Apple and Microsoft are treated as Accountable crawlers, subject to commitments not to train on or summarize protected content. Cloudflare presents Disallow AI Training as the setting that can retain search indexing for those crawlers while disallowing training.
That clarification matters because early third-party reporting in August described website owners seeing Googlebot apparently blocked where Training-block policies were active. The current documentation makes the underlying trade-off clearer: Cloudflare evaluates multi-purpose crawlers according to all of their behaviors, rather than treating a crawler solely as a search bot.
How Cloudflare's AI crawler controls work
Cloudflare's newer framework separates bot access into three categories. This gives site owners more control than the legacy toggle, but it also makes configuration choices more consequential. A crawler can serve multiple functions, including search indexing, AI-agent activity and model training. Blocking one function without considering the crawler's wider behavior can therefore affect visibility in search.
| Control | What it governs | September 15 default for newly onboarded domains |
|---|---|---|
| Search | Search crawler access | Allowed |
| Training | AI model training access | Blocked |
| Agent | AI agent access | Blocked on pages that serve ads |
These defaults apply to new domains onboarding on September 15, 2026. They should not be assumed to describe every pre-existing Cloudflare configuration. Teams should inspect the settings in their own account, particularly where a prior policy was created with the old Block AI Bots control.
The key technical point is Cloudflare's treatment of mixed-use crawlers. Googlebot, Applebot and BingBot can perform more than one relevant function. When a site chooses to block Training, Cloudflare's policy can block those crawlers as well on ad-supported pages. That can interrupt the normal path by which a search crawler fetches content for indexing.
Cloudflare's Accountable crawler approach is intended to provide a more workable alternative. The company identifies Apple, Google and Microsoft as accountable providers with no-training and no-summarization commitments. Where that classification applies, choosing Disallow AI Training can allow search discovery to continue while signalling that content is not available for AI training.
For website owners, this is a reminder that bot policy is now an SEO setting as much as a content-use setting. A desire to prevent model training does not necessarily mean a site should deny every crawler associated with an AI-capable company.
A practical review should cover at least these points:
- Separate Search from Training decisions. Do not assume a broad AI block preserves organic search crawling.
- Check ad-supported pages carefully. Cloudflare's Agent default is specifically tied to pages that serve ads.
- Review legacy configurations. Older one-click settings may not reflect the new three-control model or the intended policy.
- Use Disallow AI Training where appropriate. Cloudflare describes it as the route for retaining indexing by Accountable crawlers while restricting training.
- Monitor crawling and visibility after a policy change. Changes in organic traffic or indexing can be the first sign that an important crawler has lost access.
Cloudflare also introduced Bot Preference Sync, which keeps a site's robots.txt aligned with its policy settings. This can reduce the risk of maintaining contradictory instructions in two places. It does not remove the need to decide which types of access a site actually wants, but it makes that policy easier to express consistently.
For a marketing or web team, the operational lesson is straightforward: bot controls should be part of a documented SEO change workflow. Before changing Training, Search or Agent rules, record the current configuration and establish which crawlers are essential to discovery. After the change, watch indexing and traffic signals closely enough to identify an unintended access problem before it becomes a prolonged visibility loss.
For businesses navigating AI search and traditional search at the same time, bot settings can directly affect whether valuable pages remain discoverable. GEO Search Leads can help assess how crawler policies, technical SEO and AI-search visibility fit together, so content protection choices do not accidentally reduce qualified discovery. Start an AI visibility review to identify the search and crawler controls that deserve immediate attention.
Frequently Asked Questions
Can Cloudflare's Training setting block Googlebot?
Yes. Cloudflare says mixed-use crawlers are evaluated according to all their behaviors. Blocking Training can therefore block Googlebot on ad-supported pages, even though Googlebot is also used for search indexing.
What are Cloudflare's default AI traffic settings for new domains?
For domains onboarding on September 15, 2026, Cloudflare's defaults are to allow Search, block Training and block Agent on pages that serve ads.
How can a site block AI training while preserving Google indexing?
Cloudflare says its Disallow AI Training option allows Accountable crawlers, including Googlebot, to remain available for Search while training is disallowed.
What does Bot Preference Sync do?
Bot Preference Sync keeps robots.txt aligned with Cloudflare's bot policy settings, helping site owners avoid conflicting crawler instructions.
Conclusion
Cloudflare's updated controls give website owners more precise ways to manage AI traffic, but precision also requires careful configuration. The central issue is that Training rules can affect mixed-use search crawlers. Teams that depend on organic visibility should review their current settings, distinguish training restrictions from search access and use Cloudflare's Accountable crawler options where they match the site's content policy.
Top comments (0)