Cloudflare has announced a groundbreaking shift in the relationship between artificial intelligence and online publishing. Starting September 15, 2026, the web security and performance platform will block by default all "mixed-use" crawlers—bots that combine traditional search, AI agents, and model training—on sites that host advertisements, unless site owners manually opt in. The move, announced on Wednesday, July 1, 2026, aims to create a sustainable ecosystem where publishers' content is fairly compensated.
September 15, 2026 deadline for mixed-use crawlers
According to Cloudflare, most website owners want their content to be discoverable through search and AI services while protecting their intellectual property from free exploitation. The new default will block mixed crawlers on ad-supported pages unless the publisher explicitly enables them. The change applies to all new Cloudflare customers, new sites of existing customers, and all free-tier users. This could severely limit AI companies' access to web data for training and agentic services.
Sponsored Protocol
Google's role and Cloudflare's response
Cloudflare singled out the "world's largest search engine"—clearly referencing Google—claiming the tech giant has access to roughly twice as much information as other AI companies because it makes it difficult for customers to remain discoverable without being used for AI. Google has pushed back, noting its Google Extended bot allows publishers to opt out of AI training and products like Gemini Apps and Vertex API. However, the main Googlebot still crawls for Search, including AI features like AI Overviews and AI Mode. Cloudflare cited the recent milestone where bot traffic surpassed human traffic for the first time, arguing that faster action is needed.
Sponsored Protocol
Pay Per Use: partnerships with Ceramic.ai and You.com
To foster a sustainable model, Cloudflare introduced "Pay Per Use," an evolution of its earlier "Pay Per Crawl" marketplace. Publishers can now charge AI companies when their content creates value, not just when it is fetched. The first partners are Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic's AI search results or when You.com accesses premium content. Other AI companies can customize this model. Cloudflare also noted that over 50% of AI crawler traffic is spent re-fetching unchanged pages, wasting bandwidth that this policy aims to reduce.
Sponsored Protocol
Cloudflare's decision fits into a broader regulatory landscape. For instance, Anthropic recently adjusted its safety policies to align with Trump administration demands, sparking debates in Europe. Meanwhile, Google released Android 17 QPR1 Beta 6, showcasing its focus on finalized APIs. For further reading on content control in the AI era, the Cloudflare Wikipedia page offers additional context.
