Cloudflare sets September 15, 2026 crawl defaults

Cloudflare is changing crawl defaults, adding AI analytics, and testing pay-per-use licensing for publishers.

Cloudflare Cloudflare said it will roll out new bot classifications, deeper analytics, and commercial tools for publishers and AI companies, with default changes set for September 15, 2026. The company says the update is meant to let site owners keep content discoverable in AI while limiting training and agent use unless they opt in.

What changed

The biggest shift is policy. For new customers and new sites on existing accounts, Cloudflare will default to allowing search crawls while blocking training and agent use on pages with ads. Mixed crawlers that do not separate search, agent use, and training can be blocked on ad-supported pages.

Cloudflare said customers can change these settings in the dashboard at any time. Existing free customers who do nothing will also be moved to the new defaults on September 15, 2026. The company will spend the next two months testing the rules and taking feedback before the rollout is finalized.

• New defaults favor search discovery, not blanket AI access.
• Ad-supported pages get stronger controls over training and agent use.
• Cloudflare says transparent bots should be labeled by intent.
• Mixed crawlers are the main target of the policy change.

Cloudflare is also adding the Attribution Business Insights dashboard so business teams can see how AI bots use their content and how much human traffic specific AI companies send back. The company says that data gap has long left publishers negotiating without much visibility.

Why it matters

For publishers, the move is about control and pricing. Cloudflare is pushing a model where sites can stay visible in AI answers without giving away training rights for free, and where compensation is tied to actual use rather than just crawling.

For AI companies, the change could reduce wasted fetches and bandwidth. Cloudflare says more than half of AI crawl traffic is spent re-fetching unchanged pages, so it is testing signals that tell bots when a page has changed enough to be worth revisiting.

Cloudflare is also evolving Pay Per Crawl into Pay Per Use, with partners including Ceramic.ai and You.com. Under that model, publishers can get paid when content appears in AI search results or when an agent requests a specific piece of premium content.

The broader bet is that AI-era discovery needs new rules, new metrics, and a payment layer that works for more than the largest media companies. The open question: will enough publishers and AI firms accept Cloudflare’s defaults to make them a standard?