Cloudflare has drawn a sharper line between traditional search and artificial intelligence.
Beginning September 15, 2026, the company will change its default settings to block so-called mixed-use crawlers from pages that run ads, unless a site owner chooses otherwise. The policy applies to new Cloudflare customers, new sites created by existing customers, and all current free customers.
Follow THE FUTURE on LinkedIn, Facebook, Instagram, X and Telegram
A Clearer Divide In Web Access
The shift could materially reshape how AI companies collect web data for model training and agentic products. Cloudflare’s central argument is straightforward: most publishers want their content to remain visible in search and accessible through certain AI services, but they do not want that same material repurposed without compensation.
In Cloudflare’s view, the problem is not crawling itself. It is the blending of three different functions: search, agentic use, and training into a single bot that makes it difficult for website owners to set meaningful boundaries.
The Google Question
Cloudflare pointedly referenced the “world’s largest search engine,” an unmistakable nod to Google, arguing that it has access to roughly twice as much information as rival AI companies because it makes it harder for customers to stay discoverable without also being used for AI.
Google has disputed that framing. The company offers Google Extended, a crawler setting that lets publishers opt out of having content used for training and AI products such as Gemini apps and Vertex AI, without affecting visibility in Google Search. At the same time, Googlebot still crawls for Search and for AI-powered features such as AI Overviews and AI Mode.
Publishers Want Reach, Not Exploitation
Matthew Prince, Cloudflare’s co-founder and chief executive, said the company is moving quickly because the internet is now dominated by machine traffic.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Prince said, referring to the recent milestone in which bots surpassed human traffic online sooner than expected.
Prince added that Cloudflare’s tools and partnerships are designed to give publishers more visibility and commercial leverage, while also rewarding AI companies that are transparent about how they use content.
From Pay Per Crawl To Pay Per Use
Cloudflare has increasingly positioned itself as a gatekeeper for publishers looking to assert control in the AI era. The company already offers tools to block AI bots, along with a marketplace called Pay Per Crawl, which lets websites charge AI systems for scraping.
That framework is now expanding into Pay Per Use, which Cloudflare says will allow publishers to charge AI companies when content creates value, not merely when it is fetched. In practical terms, that shifts the economics from extraction to monetization.
Cloudflare says the move may also reduce waste. Its data suggests more than half of crawl traffic from AI bots is spent revisiting pages that have not changed, consuming bandwidth and compute without adding fresh value for either side.
Early Partners Signal The Commercial Model
To launch the new system, Cloudflare is working with Ceramic.ai and You.com. Under the opt-in model, publishers can be paid when their content appears in Ceramic’s AI search results or when You.com accesses premium material.
Cloudflare says other AI companies can adapt the model to fit their own products. The broader message is clear: the era of unrestricted crawling is giving way to one in which access, attribution, and compensation are increasingly negotiated rather than assumed.
