[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-cloudflares-new-crawler-rules-shift-power-en":3,"article-related-cloudflares-new-crawler-rules-shift-power-en":34,"series-industry-423c0239-6cff-4206-9bbb-3db02da0c622":79},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":25,"views":30,"created_at":31,"published_at":32,"topic_cluster_id":33},"423c0239-6cff-4206-9bbb-3db02da0c622","cloudflares-new-crawler-rules-shift-power-en","Cloudflare’s new crawler rules shift power to publishers","\u003Cp data-speakable=\"summary\">\u003Ca href=\"\u002Ftag\u002Fcloudflare\">Cloudflare\u003C\u002Fa> is changing crawler defaults so publishers can block AI training and \u003Ca href=\"\u002Ftag\u002Fagent\">agent\u003C\u002Fa> use more easily.\u003C\u002Fp>\u003Cp>Cloudflare’s latest policy gives site owners more control over AI crawlers, and it arrives with a concrete deadline: September 15, 2026. The company is also changing how it pays publishers, which makes this more than a simple block list.\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Item\u003C\u002Fth>\u003Cth>What changes\u003C\u002Fth>\u003Cth>Key date or detail\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>Default crawler policy\u003C\u002Ftd>\u003Ctd>Search allowed, training and agent use blocked on ad pages\u003C\u002Ftd>\u003Ctd>Starts September 15, 2026\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Free accounts\u003C\u002Ftd>\u003Ctd>Move to the same defaults unless users opt out\u003C\u002Ftd>\u003Ctd>Before September 15 deadline\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Pay Per Use\u003C\u002Ftd>\u003Ctd>Publishers paid when content appears in chatbot answers\u003C\u002Ftd>\u003Ctd>New name for Pay Per Crawl\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Mixed-use crawlers\u003C\u002Ftd>\u003Ctd>Blocked if they do not offer clear site-owner choice\u003C\u002Ftd>\u003Ctd>Especially on pages with ads\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>1. Default blocking for mixed-use crawlers\u003C\u002Fh2>\u003Cp>Cloudflare is moving from optional controls to a more defensive default. Its new policy targets crawlers that do double duty: they index sites for search, but also feed AI training or agent behavior.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783069375501-xn87.png\" alt=\"Cloudflare’s new crawler rules shift power to publishers\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>For publishers, the practical change is simple. Search can still happen, but training and agent use will be blocked by default on pages with ads for new customers and new sites on existing Cloudflare accounts.\u003C\u002Fp>\u003Cul>\u003Cli>Search indexing stays allowed\u003C\u002Fli>\u003Cli>AI training gets blocked by default on ad pages\u003C\u002Fli>\u003Cli>Agent use gets blocked by default on ad pages\u003C\u002Fli>\u003Cli>Mixed-use crawlers without clear intent may be blocked\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>2. September 15, 2026 is the switch date\u003C\u002Fh2>\u003Cp>The rollout is not vague. Cloudflare says the new defaults begin on September 15, 2026, and the change applies to new customers and new websites added by existing subscribers.\u003C\u002Fp>\u003Cp>Free accounts are also part of the change. They will move to the same defaults unless users opt out before the deadline, which means publishers need to review settings now rather than later.\u003C\u002Fp>\u003Cul>\u003Cli>Applies to new Cloudflare customers\u003C\u002Fli>\u003Cli>Applies to new sites on existing accounts\u003C\u002Fli>\u003Cli>Free users are included unless they opt out\u003C\u002Fli>\u003Cli>Deadline: September 15, 2026\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>3. Pay Per Use replaces Pay Per Crawl\u003C\u002Fh2>\u003Cp>Cloudflare is also renaming and reshaping its payment model. The old Pay Per Crawl feature, launched in 2025, let sites block AI crawlers unless companies paid to scrape content.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783069372769-wv3g.png\" alt=\"Cloudflare’s new crawler rules shift power to publishers\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>The new version is called \u003Ca href=\"\u002Fnews\u002Fcloudflare-monetization-gateway-pay-per-use-en\">Pay Per Use\u003C\u002Fa>, and the payment trigger changes. Instead of paying when a page is crawled, AI companies pay when a site’s content shows up in chatbot answers. That shifts the focus from access to visible reuse.\u003C\u002Fp>\u003Ccode>Old model: pay to crawl\u003Cbr>New model: pay when content appears in AI answers\u003C\u002Fcode>\u003Ch2>4. The first partners are Ceramic.AI and You.com\u003C\u002Fh2>\u003Cp>Cloudflare’s announcement names only two partners so far: \u003Ca href=\"https:\u002F\u002Fceramic.ai\u002F\">Ceramic.AI\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fyou.com\u002F\">You.com\u003C\u002Fa>. That is a narrow start, but it signals that Cloudflare wants a payout system, not just a block system.\u003C\u002Fp>\u003Cp>The company appears to be betting that more AI firms will join if publishers opt in. For now, the partnership list is short, which makes the rollout feel like an opening move rather than a finished marketplace.\u003C\u002Fp>\u003Cul>\u003Cli>Named partners: Ceramic.AI\u003C\u002Fli>\u003Cli>Named partners: You.com\u003C\u002Fli>\u003Cli>Goal: make AI use more transparent\u003C\u002Fli>\u003Cli>Goal: create commercial terms for publisher content\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>5. Google is the quiet target\u003C\u002Fh2>\u003Cp>Cloudflare’s policy also seems aimed at \u003Ca href=\"https:\u002F\u002Fwww.google.com\u002F\">Google\u003C\u002Fa>, even if the company does not say so directly. The announcement points out that the largest search engine has access to about 2X more information than leading AI companies because it is hard for customers to stay discoverable without also feeding AI systems.\u003C\u002Fp>\u003Cp>That matters because Googlebot does more than search indexing. It also helps train \u003Ca href=\"\u002Ftag\u002Fgemini\">Gemini\u003C\u002Fa> and power AI features like AI Overviews and AI Mode. \u003Ca href=\"\u002Ftag\u002Fgoogle\">Google\u003C\u002Fa> offers Google-Extended for sites that want search indexing without model training, but publishers still lack a clean way to be in AI Mode without being used for training.\u003C\u002Fp>\u003Cul>\u003Cli>Googlebot indexes pages and feeds AI systems\u003C\u002Fli>\u003Cli>Google-Extended is a separate opt-in crawler\u003C\u002Fli>\u003Cli>Publishers lack a full split between search and AI use\u003C\u002Fli>\u003Cli>Cloudflare wants mixed-use crawlers to separate those functions\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>What to pick\u003C\u002Fh2>\u003Cp>If you run a site with ads and want less AI scraping by default, Cloudflare’s new policy is the clearest fit. It gives you a stronger starting position, plus a payment option if you want to allow reuse on your terms.\u003C\u002Fp>\u003Cp>If you are an AI company, the message is also clear: mixed-use crawling is getting harder to justify. The companies most likely to adapt are the ones willing to separate search from training and agent use, and to make that intent visible to publishers.\u003C\u002Fp>","Cloudflare’s new crawler policy gives publishers more control over AI access, with defaults changing for new sites on September 15, 2026.","www.engadget.com","https:\u002F\u002Fwww.engadget.com\u002F2207360\u002Fcloudflare-will-filter-out-web-crawlers-that-serve-ai-companies\u002F",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783069375501-xn87.png","industry","en","ccec43ca-246c-4267-9deb-e0d087f12bc1",[17,18,19,20,21,22,23,24],"Cloudflare","AI crawlers","publisher control","Pay Per Use","Pay Per Crawl","Googlebot","You.com","Ceramic.AI",[26,27,28,29],"Cloudflare will default many sites to block AI training and agent use while still allowing search indexing.","The new Pay Per Use model pays publishers when their content appears in chatbot answers.","Google’s mixed crawler model is an implied target of the new policy.","The rollout begins September 15, 2026 for new customers, new sites, and many free accounts.",0,"2026-07-03T09:02:27.462999+00:00","2026-07-03T09:02:27.447+00:00","e63df91b-385f-44c9-b3f6-44a1a0e4b505",{"tags":35,"relatedLang":38,"relatedPosts":42},[36],{"name":17,"slug":37},"cloudflare",{"id":15,"slug":39,"title":40,"language":41},"cloudflares-new-crawler-rules-shift-power-zh","Cloudflare 新爬蟲規則把控制權交回出版商","zh",[43,49,55,61,67,73],{"id":44,"slug":45,"title":46,"cover_image":47,"image_url":47,"created_at":48,"category":13},"7acd4647-667f-4218-a554-3734f6737efd","cloudflare-september-2026-crawl-defaults-en","Cloudflare sets September 15, 2026 crawl defaults","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783071172244-ccw5.png","2026-07-03T09:32:29.80015+00:00",{"id":50,"slug":51,"title":52,"cover_image":53,"image_url":53,"created_at":54,"category":13},"0362089c-2aa4-4fff-9b62-1a8b18c4d8ca","midjourney-body-scan-spa-ultrasonic-ct-en","Midjourney is building a body-scan spa","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783067585797-r9j9.png","2026-07-03T08:32:42.100537+00:00",{"id":56,"slug":57,"title":58,"cover_image":59,"image_url":59,"created_at":60,"category":13},"b43e5f46-a4d7-4441-9aee-890cc83b8141","midjourney-force-hollywood-ai-cards-table-en","Midjourney Is Right to Force Hollywood’s AI Cards Onto the Table","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783065773451-i5ie.png","2026-07-03T08:02:25.173434+00:00",{"id":62,"slug":63,"title":64,"cover_image":65,"image_url":65,"created_at":66,"category":13},"6b2d04b8-9053-44d8-8ec1-1e614b55ab8b","midjourney-body-scanner-diagnosis-wrong-bet-en","Midjourney should stay out of diagnosis and focus on body-composition…","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783063975300-wt2i.png","2026-07-03T07:32:27.132957+00:00",{"id":68,"slug":69,"title":70,"cover_image":71,"image_url":71,"created_at":72,"category":13},"7b761c24-80dc-4712-baae-d673e6072c50","microsoft-bets-on-enterprise-ai-deployments-en","Microsoft bets $2.5B on enterprise AI deployments","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783044168769-r4fq.png","2026-07-03T02:02:24.047733+00:00",{"id":74,"slug":75,"title":76,"cover_image":77,"image_url":77,"created_at":78,"category":13},"f4e86e56-1c44-4d0a-9304-070b569ce041","microsoft-2point5b-ai-implementation-unit-enterprise-en","Microsoft’s $2.5B AI unit shows where enterprise AI is headed","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783040574693-juyn.png","2026-07-03T01:02:27.129825+00:00",[80,85,90,95,100,105,110,115,120,125],{"id":81,"slug":82,"title":83,"created_at":84},"d35a1bd9-e709-412e-a2df-392df1dc572a","ai-impact-2026-developments-market-en","AI's Impact in 2026: Key Developments and Market Shifts","2026-03-25T16:20:33.205823+00:00",{"id":86,"slug":87,"title":88,"created_at":89},"5ed27921-5fd6-492e-8c59-78393bf37710","trumps-ai-legislative-framework-en","Trump's AI Legislative Framework: What's Inside?","2026-03-25T16:22:20.005325+00:00",{"id":91,"slug":92,"title":93,"created_at":94},"e454a642-f03c-4794-b185-5f651aebbaca","nvidia-gtc-2026-key-highlights-innovations-en","NVIDIA GTC 2026: Key Highlights and Innovations","2026-03-25T16:22:47.882615+00:00",{"id":96,"slug":97,"title":98,"created_at":99},"0ebb5b16-774a-4922-945d-5f2ce1df5a6d","claude-usage-diversifies-learning-curves-en","Claude Usage Diversifies, Learning Curves Emerge","2026-03-25T16:25:50.770376+00:00",{"id":101,"slug":102,"title":103,"created_at":104},"69934e86-2fc5-4280-8223-7b917a48ace8","openclaw-ai-commoditization-concerns-en","OpenClaw's Rise Raises Concerns of AI Model Commoditization","2026-03-25T16:26:30.582047+00:00",{"id":106,"slug":107,"title":108,"created_at":109},"b4b2575b-2ac8-46b2-b90e-ab1d7c060797","google-gemini-ai-rollout-2026-en","Google's Gemini AI Rollout Extended to 2026","2026-03-25T16:28:14.808842+00:00",{"id":111,"slug":112,"title":113,"created_at":114},"6e18bc65-42ae-4ad0-b564-67d7f66b979e","meta-llama4-fabricated-results-scandal-en","Meta's Llama 4 Scandal: Fabricated AI Test Results Unveiled","2026-03-25T16:29:15.482836+00:00",{"id":116,"slug":117,"title":118,"created_at":119},"bf888e9d-08be-4f47-996c-7b24b5ab3500","accenture-mistral-ai-deployment-en","Accenture and Mistral AI Team Up for AI Deployment","2026-03-25T16:31:01.894655+00:00",{"id":121,"slug":122,"title":123,"created_at":124},"5382b536-fad2-49c6-ac85-9eb2bae49f35","mistral-ai-high-stakes-2026-en","Mistral AI: Facing High Stakes in 2026","2026-03-25T16:31:39.941974+00:00",{"id":126,"slug":127,"title":128,"created_at":129},"9da3d2d6-b669-4971-ba1d-17fdb3548ed5","cursors-meteoric-rise-pressures-en","Cursor's Meteoric Rise Faces Industry Pressures","2026-03-25T16:32:21.899217+00:00"]