ADAM ENGST 7 July 2025

One of the most significant concerns about the rise of generative AI has been its potential to decouple the relationship between content creators and content consumers. Rather than using a search engine to find original sources of information that might answer our questions, we’re increasingly relying on systems like Google’s AI answers, Perplexity, ChatGPT, and Claude to answer our questions directly. That’s often better for users but poses an existential problem for authors and publishers who rely on ads displayed to human readers for revenue. A new pay-per-crawl proposal from content-delivery service Cloudflare could offer an alternative revenue stream for publishers.

Internet advertising has long been highly problematic because publishers are financially beholden to their advertisers, not their readers, and thus resort to questionable tactics to entice or compel users to view ads. It’s how we ended up with surveillance advertising, where companies like Meta follow you around the Internet in an attempt to display ads you’re more likely to click. (Almost no one does—industry statistics show click-through rates for display ads are well under 1%, leading publishers to increase ad volume in an attempt to boost total clicks.) Readers detest advertising because it has little or nothing to do with why they’re reading, leading to the widespread use of ad blockers and the (usually small-scale) success of publications that support themselves primarily through subscriptions or voluntary contributions.

Near the end of “AI Answer Engines Are Worth Trying” (17 April 2025), I wrote:

However, what I would prefer to see is a system that pays micro-royalties based on the materials used to generate responses. The technical, legal, and social hurdles to implementing such a system are significant, but some form of business collaboration between for-profit content creators and AI companies will be necessary in the long run.

Cloudflare’s pay-per-crawl proposal suggests at least a technical solution to this problem. In essence, Cloudflare proposes that when an AI crawler visits a website, the site’s publisher would allow access only if the crawler agrees to a specified price. If the crawler agreed, the publisher would earn money; if the crawler didn’t agree, it would be blocked.

Pay-per-crawl cleverly resurrects an old idea from the early Web: HTTP response code 402, which was originally intended for handling digital payments but was never widely implemented. Combined with Cloudflare’s proposed Web Bot Auth standards, the system would require AI crawlers to authenticate themselves and set up payment accounts before accessing content. Authenticated crawlers would then be linked to payment accounts, enabling the actual financial transfers. Unauthenticated crawlers would just be turned away.

Since Cloudflare provides content delivery services to 44% of the top 10,000 websites and millions of smaller sites (including TidBITS), the company is in a good position to make pay-per-crawl a reality on both the technical and financial sides. I signed up for the private beta to learn more, but haven’t heard any more from Cloudflare about it yet.

There are undoubtedly numerous concerns with pay-per-crawl, not the least of which is that it would put Cloudflare in a position of even greater power within the Internet ecosystem. It could also hinder academic research and open source projects that lack substantial funding.

However, what I find even more interesting about pay-per-crawl is how it might revive HTTP response code 402 as a more general method of enabling direct transactions between producers and consumers. We’re getting close to some of the micropayment-related ideas in Ted Nelson’s largely theoretical Project Xanadu, which could radically democratize commerce on the Internet (I’ve been beating this drum for decades; see “Xanadu Light,” 29 November 1993).

Imagine if your browser had a wallet you had funded with $25 for a month’s worth of browsing and configured to allow payments of up to 1 cent per page, prompting you for approval of higher amounts. Then, as you browsed the Web, it would automatically pay up to a penny per page, pausing to request permission to pay more if a fancy site—say the Wall Street Journal—was charging 5 cents for a particular article. At a penny per page, your $25 monthly budget would cover roughly 2500 page views—a reasonable amount for many casual Web users.

Of course, if you were getting some of your information from a chatbot, you could be paying it some amount for each response, too, but it would, in turn, be paying the content creators whose work it was ingesting to generate its answers. The same payment approach, though likely with different rates, could work for music, video, and even time-based chunks of interactive media like video games.

Adam Smith’s invisible hand of the market would determine what the prices would be, settling somewhere between the high prices that publishers would undoubtedly prefer and the zero cost that readers are largely accustomed to paying today. In some ways, it’s not all that different from subscribing to a service like Apple Music or Netflix. The key difference is in how payment works. With this system, you pay based on what you actually consume. With Apple Music or Netflix, the aggregators calculate a flat fee that will cover their costs and generate a profit, regardless of how much content you use.

I’m not alone in seeing possibilities here—watch Dylan Beattie’s video explaining the Cloudflare proposal to see him arrive at much the same conclusion.