[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-cloudflare-september-2026-crawl-defaults-en":3,"article-related-cloudflare-september-2026-crawl-defaults-en":30,"series-industry-7acd4647-667f-4218-a554-3734f6737efd":75},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":29},"7acd4647-667f-4218-a554-3734f6737efd","cloudflare-september-2026-crawl-defaults-en","Cloudflare sets September 15, 2026 crawl defaults","\u003Cp data-speakable=\"summary\">\u003Ca href=\"\u002Ftag\u002Fcloudflare\">Cloudflare\u003C\u002Fa> is changing crawl defaults, adding AI analytics, and testing pay-per-use licensing for publishers.\u003C\u002Fp>\u003Cp>Cloudflare \u003Ca href=\"https:\u002F\u002Fwww.cloudflare.com\" target=\"_blank\" rel=\"noopener\">Cloudflare\u003C\u002Fa> said it will roll out new bot classifications, deeper analytics, and commercial tools for publishers and AI companies, with default changes set for September 15, 2026. The company says the update is meant to let site owners keep content discoverable in AI while limiting training and \u003Ca href=\"\u002Ftag\u002Fagent\">agent\u003C\u002Fa> use unless they opt in.\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>項目\u003C\u002Fth>\u003Cth>數值\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>Default change date\u003C\u002Ftd>\u003Ctd>September 15, 2026\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Major content licensing deals cited\u003C\u002Ftd>\u003Ctd>50+ in the past year\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>AI crawl traffic spent on unchanged pages\u003C\u002Ftd>\u003Ctd>Over 50%\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Search engine access vs. leading AI companies\u003C\u002Ftd>\u003Ctd>About 2x more information\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>What changed\u003C\u002Fh2>\u003Cp>The biggest shift is policy. For new customers and new sites on existing accounts, Cloudflare will default to allowing search crawls while blocking training and agent use on pages with ads. Mixed crawlers that do not separate search, agent use, and training can be blocked on ad-supported pages.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783071172244-ccw5.png\" alt=\"Cloudflare sets September 15, 2026 crawl defaults\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>Cloudflare said customers can change these settings in the dashboard at any time. Existing free customers who do nothing will also be moved to the new defaults on September 15, 2026. The company will spend the next two months testing the rules and taking feedback before the rollout is finalized.\u003C\u002Fp>\u003Cul>\u003Cli>New defaults favor search discovery, not blanket AI access.\u003C\u002Fli>\u003Cli>Ad-supported pages get stronger controls over training and agent use.\u003C\u002Fli>\u003Cli>Cloudflare says transparent bots should be labeled by intent.\u003C\u002Fli>\u003Cli>Mixed crawlers are the main target of the policy change.\u003C\u002Fli>\u003C\u002Ful>\u003Cp>Cloudflare is also adding the \u003Ca href=\"https:\u002F\u002Fwww.cloudflare.com\" target=\"_blank\" rel=\"noopener\">Attribution Business Insights\u003C\u002Fa> dashboard so business teams can see how AI bots use their content and how much human traffic specific AI companies send back. The company says that data gap has long left publishers negotiating without much visibility.\u003C\u002Fp>\u003Ch2>Why it matters\u003C\u002Fh2>\u003Cp>For publishers, the move is about control and pricing. Cloudflare is pushing a model where sites can stay visible in AI answers without giving away training rights for free, and where compensation is tied to actual use rather than just crawling.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783071174516-x7fe.png\" alt=\"Cloudflare sets September 15, 2026 crawl defaults\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>For AI companies, the change could reduce wasted fetches and bandwidth. Cloudflare says more than half of AI crawl traffic is spent re-fetching unchanged pages, so it is testing signals that tell bots when a page has changed enough to be worth revisiting.\u003C\u002Fp>\u003Cp>Cloudflare is also evolving \u003Ca href=\"https:\u002F\u002Fwww.cloudflare.com\" target=\"_blank\" rel=\"noopener\">Pay Per Crawl\u003C\u002Fa> into \u003Ca href=\"https:\u002F\u002Fwww.cloudflare.com\" target=\"_blank\" rel=\"noopener\">Pay Per Use\u003C\u002Fa>, with partners including \u003Ca href=\"https:\u002F\u002Fwww.ceramic.ai\" target=\"_blank\" rel=\"noopener\">Ceramic.ai\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fyou.com\" target=\"_blank\" rel=\"noopener\">You.com\u003C\u002Fa>. Under that model, publishers can get paid when content appears in AI search results or when an agent requests a specific piece of premium content.\u003C\u002Fp>\u003Cp>The broader bet is that AI-era discovery needs new rules, new metrics, and a payment layer that works for more than the largest media companies. The open question: will enough publishers and AI firms accept Cloudflare’s defaults to make them a standard?\u003C\u002Fp>","Cloudflare will default new sites to allow search but block AI training and agent use on ad pages, while adding analytics and pay-per-use tools.","www.cityam.com","https:\u002F\u002Fwww.cityam.com\u002Fcloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules\u002F",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783071172244-ccw5.png","industry","en","1929494a-10c5-4f49-9c66-8dd2c31dd147",[17,18,19,20,21],"Cloudflare","AI crawlers","publishers","pay-per-use","content licensing",[23,24,25],"Cloudflare will default new sites to allow search but block AI training and agent use on ad pages starting Sept. 15, 2026.","It is adding analytics to show how AI bots use content and how much traffic they send back.","Cloudflare is testing pay-per-use licensing with Ceramic.ai and You.com.",0,"2026-07-03T09:32:29.80015+00:00","2026-07-03T09:32:29.79+00:00","e63df91b-385f-44c9-b3f6-44a1a0e4b505",{"tags":31,"relatedLang":34,"relatedPosts":38},[32],{"name":17,"slug":33},"cloudflare",{"id":15,"slug":35,"title":36,"language":37},"cloudflare-september-2026-crawl-defaults-zh","Cloudflare 定 2026\u002F9\u002F15 爬蟲預設","zh",[39,45,51,57,63,69],{"id":40,"slug":41,"title":42,"cover_image":43,"image_url":43,"created_at":44,"category":13},"fa94d7c8-01ec-4cc1-9b33-a1c7ba78cda2","meta-ai-hardware-wall-street-compute-demand-en","Meta一句话引爆AI硬件，华尔街仍看多算力","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783081971766-41gl.png","2026-07-03T12:32:22.193465+00:00",{"id":46,"slug":47,"title":48,"cover_image":49,"image_url":49,"created_at":50,"category":13},"423c0239-6cff-4206-9bbb-3db02da0c622","cloudflares-new-crawler-rules-shift-power-en","Cloudflare’s new crawler rules shift power to publishers","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783069375501-xn87.png","2026-07-03T09:02:27.462999+00:00",{"id":52,"slug":53,"title":54,"cover_image":55,"image_url":55,"created_at":56,"category":13},"0362089c-2aa4-4fff-9b62-1a8b18c4d8ca","midjourney-body-scan-spa-ultrasonic-ct-en","Midjourney is building a body-scan spa","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783067585797-r9j9.png","2026-07-03T08:32:42.100537+00:00",{"id":58,"slug":59,"title":60,"cover_image":61,"image_url":61,"created_at":62,"category":13},"b43e5f46-a4d7-4441-9aee-890cc83b8141","midjourney-force-hollywood-ai-cards-table-en","Midjourney Is Right to Force Hollywood’s AI Cards Onto the Table","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783065773451-i5ie.png","2026-07-03T08:02:25.173434+00:00",{"id":64,"slug":65,"title":66,"cover_image":67,"image_url":67,"created_at":68,"category":13},"6b2d04b8-9053-44d8-8ec1-1e614b55ab8b","midjourney-body-scanner-diagnosis-wrong-bet-en","Midjourney should stay out of diagnosis and focus on body-composition…","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783063975300-wt2i.png","2026-07-03T07:32:27.132957+00:00",{"id":70,"slug":71,"title":72,"cover_image":73,"image_url":73,"created_at":74,"category":13},"7b761c24-80dc-4712-baae-d673e6072c50","microsoft-bets-on-enterprise-ai-deployments-en","Microsoft bets $2.5B on enterprise AI deployments","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1783044168769-r4fq.png","2026-07-03T02:02:24.047733+00:00",[76,81,86,91,96,101,106,111,116,121],{"id":77,"slug":78,"title":79,"created_at":80},"d35a1bd9-e709-412e-a2df-392df1dc572a","ai-impact-2026-developments-market-en","AI's Impact in 2026: Key Developments and Market Shifts","2026-03-25T16:20:33.205823+00:00",{"id":82,"slug":83,"title":84,"created_at":85},"5ed27921-5fd6-492e-8c59-78393bf37710","trumps-ai-legislative-framework-en","Trump's AI Legislative Framework: What's Inside?","2026-03-25T16:22:20.005325+00:00",{"id":87,"slug":88,"title":89,"created_at":90},"e454a642-f03c-4794-b185-5f651aebbaca","nvidia-gtc-2026-key-highlights-innovations-en","NVIDIA GTC 2026: Key Highlights and Innovations","2026-03-25T16:22:47.882615+00:00",{"id":92,"slug":93,"title":94,"created_at":95},"0ebb5b16-774a-4922-945d-5f2ce1df5a6d","claude-usage-diversifies-learning-curves-en","Claude Usage Diversifies, Learning Curves Emerge","2026-03-25T16:25:50.770376+00:00",{"id":97,"slug":98,"title":99,"created_at":100},"69934e86-2fc5-4280-8223-7b917a48ace8","openclaw-ai-commoditization-concerns-en","OpenClaw's Rise Raises Concerns of AI Model Commoditization","2026-03-25T16:26:30.582047+00:00",{"id":102,"slug":103,"title":104,"created_at":105},"b4b2575b-2ac8-46b2-b90e-ab1d7c060797","google-gemini-ai-rollout-2026-en","Google's Gemini AI Rollout Extended to 2026","2026-03-25T16:28:14.808842+00:00",{"id":107,"slug":108,"title":109,"created_at":110},"6e18bc65-42ae-4ad0-b564-67d7f66b979e","meta-llama4-fabricated-results-scandal-en","Meta's Llama 4 Scandal: Fabricated AI Test Results Unveiled","2026-03-25T16:29:15.482836+00:00",{"id":112,"slug":113,"title":114,"created_at":115},"bf888e9d-08be-4f47-996c-7b24b5ab3500","accenture-mistral-ai-deployment-en","Accenture and Mistral AI Team Up for AI Deployment","2026-03-25T16:31:01.894655+00:00",{"id":117,"slug":118,"title":119,"created_at":120},"5382b536-fad2-49c6-ac85-9eb2bae49f35","mistral-ai-high-stakes-2026-en","Mistral AI: Facing High Stakes in 2026","2026-03-25T16:31:39.941974+00:00",{"id":122,"slug":123,"title":124,"created_at":125},"9da3d2d6-b669-4971-ba1d-17fdb3548ed5","cursors-meteoric-rise-pressures-en","Cursor's Meteoric Rise Faces Industry Pressures","2026-03-25T16:32:21.899217+00:00"]