Cloudflare Shifts from Blanket AI Bot Blocking to Fine-Grained Control over Search, Training, and Agent Crawlers
Decision Brief
What changedCloudflare now offers all users fine-grained AI bot control to separately manage search, training, and agent bots, replacing blanket blocking.
Why it mattersFor site administrators, distinguishing training and agent bots and blocking them by default on ad pages can better protect content revenue and sovereignty.
Who should careAI coding tool users
Affected stackNo specific stack identified
Source confidenceMedium · Reliable media or first-hand reporting
Cloudflare has announced a shift from comprehensive blocking of AI bots to fine-grained control. All site owners can now manage three types of bots—Search, Training, and Agent—separately, without needing to block all at once. Starting September 15, 2026, Training and Agent bots will be blocked by default on pages with ads. This update directly impacts third-party AI developers and data intermediaries that rely on website content. Site administrators using Cloudflare can flexibly decide which AI crawlers can access their content, avoiding harm to legitimate search engines while effectively defending against unauthorized training data collection and automated agent behavior.
Summary basis: official / RSS sourceCompiled from the source scope noted above; the original remains authoritative.
Sources
- The Decoder：AI News
- The Decoder：AI News