Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Summary
Cloudflare introduces Pay Per Crawl, a mechanism enabling content owners to monetize access for AI crawlers via HTTP 402 payments. Publishers can choose Allow, Charge, or Block, with Cloudflare acting as Merchant of Record and enforcing access decisions after existing security policies. The system uses Web Bot Auth to authenticate crawlers through signed HTTP headers, and supports reactive (crawler-price) and proactive (crawler-max-price) payment flows, enabling programmatic access control at internet scale. The concept envisions dynamic pricing, partner-enabled crawls, and agentic AI applications adapting monetization to demand.