Cloudflare’s new crawler rules shift power to publishers

Cloudflare is changing crawler defaults so publishers can block AI training and agent use more easily.

Cloudflare’s latest policy gives site owners more control over AI crawlers, and it arrives with a concrete deadline: September 15, 2026. The company is also changing how it pays publishers, which makes this more than a simple block list.

1. Default blocking for mixed-use crawlers

Cloudflare is moving from optional controls to a more defensive default. Its new policy targets crawlers that do double duty: they index sites for search, but also feed AI training or agent behavior.

For publishers, the practical change is simple. Search can still happen, but training and agent use will be blocked by default on pages with ads for new customers and new sites on existing Cloudflare accounts.

• Search indexing stays allowed
• AI training gets blocked by default on ad pages
• Agent use gets blocked by default on ad pages
• Mixed-use crawlers without clear intent may be blocked

2. September 15, 2026 is the switch date

The rollout is not vague. Cloudflare says the new defaults begin on September 15, 2026, and the change applies to new customers and new websites added by existing subscribers.

Free accounts are also part of the change. They will move to the same defaults unless users opt out before the deadline, which means publishers need to review settings now rather than later.

• Applies to new Cloudflare customers
• Applies to new sites on existing accounts
• Free users are included unless they opt out
• Deadline: September 15, 2026

3. Pay Per Use replaces Pay Per Crawl

Cloudflare is also renaming and reshaping its payment model. The old Pay Per Crawl feature, launched in 2025, let sites block AI crawlers unless companies paid to scrape content.

The new version is called Pay Per Use, and the payment trigger changes. Instead of paying when a page is crawled, AI companies pay when a site’s content shows up in chatbot answers. That shifts the focus from access to visible reuse.

4. The first partners are Ceramic.AI and You.com

Cloudflare’s announcement names only two partners so far: Ceramic.AI and You.com. That is a narrow start, but it signals that Cloudflare wants a payout system, not just a block system.

The company appears to be betting that more AI firms will join if publishers opt in. For now, the partnership list is short, which makes the rollout feel like an opening move rather than a finished marketplace.

• Named partners: Ceramic.AI
• Named partners: You.com
• Goal: make AI use more transparent
• Goal: create commercial terms for publisher content

5. Google is the quiet target

Cloudflare’s policy also seems aimed at Google, even if the company does not say so directly. The announcement points out that the largest search engine has access to about 2X more information than leading AI companies because it is hard for customers to stay discoverable without also feeding AI systems.

That matters because Googlebot does more than search indexing. It also helps train Gemini and power AI features like AI Overviews and AI Mode. Google offers Google-Extended for sites that want search indexing without model training, but publishers still lack a clean way to be in AI Mode without being used for training.

• Googlebot indexes pages and feeds AI systems
• Google-Extended is a separate opt-in crawler
• Publishers lack a full split between search and AI use
• Cloudflare wants mixed-use crawlers to separate those functions

What to pick

If you run a site with ads and want less AI scraping by default, Cloudflare’s new policy is the clearest fit. It gives you a stronger starting position, plus a payment option if you want to allow reuse on your terms.

If you are an AI company, the message is also clear: mixed-use crawling is getting harder to justify. The companies most likely to adapt are the ones willing to separate search from training and agent use, and to make that intent visible to publishers.