Cloudflare Changes the Game for AI Companies
Cloudflare has decided to shake up the board for artificial intelligence companies. Starting September 15, 2026, the company will block "mixed" crawlers that combine search, agent use, and AI training on pages with ads, unless the site owner changes the settings. This change will impact how AI model providers access web content to train their tools and services.
The issue is that many site owners want their content to be found by both traditional searches and AI services. But, of course, without their intellectual property being distributed for free. Cloudflare highlights that the "largest search engine in the world" (yes, we're talking about Google) has access to "twice as much information" as other AI companies. This is because Google makes it difficult for clients to remain visible without their data being used for AI.
The Impact of Bots on Internet Traffic
Matthew Prince, co-founder and CEO of Cloudflare, commented that now that most internet traffic is non-human, it is necessary to act quickly to create a sustainable ecosystem. This is because, recently, bots have surpassed human online traffic. Cloudflare wants mixed crawlers to separate search from agent use and training, encouraging greater transparency.
To give publishers more control, Cloudflare has launched tools that allow them to charge AI bots for access to content. One example is "Pay Per Crawl," which has evolved into "Pay Per Use." With this, publishers can charge AI companies not only when content is accessed but also when it generates value. This helps conserve bandwidth and computing resources, as over 50% of AI crawler traffic is spent re-fetching unchanged pages.
Partnerships and New Opportunities for Publishers
Initially, Cloudflare is working with two partners, Ceramic.ai and You.com. When a publisher opts to participate, they are paid when their content appears in the search results of Ceramic or when You.com accesses premium content. Other AI companies can customize this model according to their needs.
Comments (0)
Comments are moderated and if they violate our Terms and Conditions of use, the comment will be deleted. Persistence in violation will result in a ban of your account.