Cloudflare and AWS just revived a 30-year-old protocol to charge AI crawlers for access. It's being sold as a way to get paid. It's actually a decision about whether AI can still find you.
TL;DR: HTTP 402, a payment protocol dormant since 1997, was independently revived by Cloudflare (July 2025) and AWS (June 2026) to charge AI crawlers. It's framed as new revenue. It's actually a visibility gate. Every AI crawler behind a paywall is one less agent that can read, cite, and recommend your business. Meanwhile, 57.3% of web traffic is now automated. The web is splitting in two: a tolled internet where AI visibility shrinks, and a free internet where citations compound.
A 30-Year-Old Protocol Just Woke Up
HTTP 402, Payment Required, sat dormant for three decades. Nobody used it.
In the last twelve months, two of the largest infrastructure companies on earth independently brought it back. Cloudflare launched pay-per-crawl in July 2025. AWS added it to their web application firewall in June 2026. TollBit built a paywall marketplace. Akamai integrated it at the network layer. The pitch is direct: AI companies are extracting your content for free. Here is a way to get paid.
The catch is what nobody is saying out loud.
The Broken Bargain
For thirty years, the web ran on a simple deal. Let the crawler in. It indexes your pages. It sends people back. AI crawlers kept the first half and dropped the second. Cloudflare's data shows nearly 80% of AI bot activity is training extraction. It reads your content, absorbs it into a model, and sends nobody back.
Here is why this matters right now. When someone asks ChatGPT "best growth consultants for B2B SaaS," the AI assembles an answer from whatever it can read. If your site is behind a toll booth, or your content is trapped in JavaScript, or your messaging is inconsistent, you were never in that answer to begin with. You did not lose the citation. You were never eligible for it.
The Toll Booth Is a Visibility Decision
Slobodan Manic, writing on No Hacks, put it in two sentences: "The bill you can see is the one the crawler pays. The bill you cannot see is the answer you vanish from."
Every AI crawler you put behind a paywall is one less agent that can read, cite, and recommend you. You get paid pennies per thousand crawls from the companies large enough to pay. You lose visibility on every platform whose crawler will not pay, or cannot, or has not been configured to. The revenue might cover your monthly coffee. The visibility loss could cost you every customer who now asks AI instead of Googling.
Think about toll roads. When you put up a toll, some drivers pay and some take another route. The ones who take another route never see what is on your road. AI crawlers work the same way, except the alternate route is your competitor's site.
A Web That's 57.3% Machine
Cloudflare found that automated traffic passed human traffic in June 2026, a full year ahead of projections. 57.3% of all web requests now come from machines. The question of which machines you let in is not a technical detail. It is a strategic decision about who gets to know your business exists.
The consequence is a fork in the web itself. On one side, the tolled web: paid access, shrinking AI visibility, direct per-crawl revenue. On the other, the free web: open to all agents, compounding citations, uncompensated extraction. Both have costs. The tolled web's fees show up on an invoice. The free web's lost citations are invisible. You cannot count answers you never appeared in. Most businesses will only see the first cost.
The Competitive Moat Nobody's Talking About
If your competitors put up toll booths and you keep your site open and machine-readable, your citation share grows by default. Every agent they keep out sees your content instead. The early adopters of pay-per-crawl are mostly large publishers whose ad traffic was already hemorrhaging to AI. Their calculation makes sense for their model. For a business whose growth depends on being found and cited, being on the shortlist an AI assembles before a buyer ever contacts you, that calculation does not hold. You cannot charge your way to visibility. But your competitors can charge their way out of it.
The Hedge
The alternative is building a site AI agents can read efficiently enough that you do not need a toll. Manic calls this machine-first architecture: semantic HTML, server-rendered content, consistent messaging across every page. The fix is not expensive. It is just work most businesses have not done.
Audit which agents drive your AI visibility before you make any toll decision. Check whether your site is even machine-readable. A third of fintech is invisible to AI agents because of JavaScript walls, and that is before anyone puts up a toll. Watch what your competitors do. Every one that walls off makes you more visible by default.
The web is forking. You are picking a side whether you mean to or not.