Half your visitors are programs now

On 3 June Cloudflare’s chief executive posted a chart he had not expected to post until late 2027. Across the fifth of the web that Cloudflare fronts, automated requests for HTML pages had crossed 57%, and humans were the minority for the first time. Imperva’s Bad Bot Report, which counts differently and skews toward the login and checkout endpoints it defends, had automation at 53% for 2025. The two do not agree on the number. They agree on the direction, and on the reason, which is agents.

I build storefronts and the admin systems behind them, so I care about this less as a media story and more as a question about who my sites are for. The honest answer as of this month is two readers, with very different needs, and the newer one does not behave like the old one at all.

Crawlers and agents are not the same visitor

The first kind of automated visitor is the crawler, and it is the one most of the coverage is about. It fetches pages to train a model or to answer a question somewhere else. Cloudflare Radar’s crawl-to-refer ratio measures what a site gets back for that, and the July numbers are lopsided: Anthropic’s crawler fetched roughly 2,200 pages for every visitor it sent to a site, OpenAI’s roughly 200, and Google under five. Those figures move month to month and vary by site type, so treat the ordering as firmer than the decimals. The ordering is not close.

The referral side is shrinking as well. SparkToro’s read of Similarweb panel data has 68% of US Google searches in the first four months of 2026 ending without a click, up from 60% in 2024, and that panel does not include the Google app, where the summaries are more aggressive still. A page written to be read by a person is increasingly read by something that summarises it for the person instead, and the page does not get the visit.

The second kind of visitor is an agent acting for one specific person, right now. It is comparing your product against three others, filling in your form, or trying to buy. It is the one that grew fast enough to move Cloudflare’s line eighteen months early, and it is the one worth building for, because it still wants what a human wants. It just wants it in a different shape.

The infrastructure response is identity and a price

For the crawler problem the response has been to stop guessing who is knocking. Web Bot Auth signs each request with an Ed25519 key using the HTTP message signatures standard, so a site verifies an agent’s identity rather than inferring it from a user-agent string anyone can copy. Cloudflare has treated signed bots as verified since last summer, and put HTTP status 402 to work for pay-per-crawl. That code has sat in the HTTP specification since 1997 marked reserved for future use. The future turned out to be this.

On 1 July this year Cloudflare went further. From 15 September its default for new sites and free plans blocks mixed-use crawlers, the ones that combine search indexing with training and answering, from any page carrying ads. The same announcement noted that over half of AI crawler traffic goes to re-fetching pages that have not changed, which is worth remembering the next time a cache header feels optional.

Commerce learned the second lesson first

OpenAI launched Instant Checkout inside ChatGPT on 29 September 2025 with Etsy, and retired it on 24 March 2026. Walmart’s head of AI acceleration, Daniel Danker, said conversion on items sold inside the chat was three times lower than on items that sent the shopper out to Walmart’s own site. Shopify’s president, Harley Finkelstein, put the number of Shopify merchants using it at about a dozen. The product data behind the chat was scraped, so prices and stock went stale, and the checkout had no loyalty programme, no promotions and no store pickup, which are the things a retailer’s checkout exists to do.

That is the same failure as a chatbot answering from a snapshot. The Healing Oasis chatbot I wrote about here made things up because it answered from what it remembered the data being. An agent that sells from a scrape sells last week’s price with this week’s confidence. The fix in both cases is the same: fetch at the moment of the answer, from the system that owns the state.

What replaced it says where this is going. ChatGPT now hands the purchase to the retailer’s own checkout through retailer apps. Google announced the Universal Commerce Protocol in January with Shopify, Walmart, Target, Etsy and around twenty others, built on existing pieces rather than a new one: plain REST, the Model Context Protocol, the Agent Payments Protocol and Agent2Agent. At Google Marketing Live on 20 May the company said checkout through it was going live with US retailers, and a Shopify merchant turns it on at the platform level rather than integrating anything. Google donated the payments protocol to the FIDO Alliance on 28 April, with Mastercard, Visa, American Express, PayPal, Stripe and Adyen among roughly sixty contributors, and the current version covers payments where the human is not present at all.

The traffic that comes through this is worth having. Adobe Analytics measured AI-referred traffic to US retail sites up 138% year over year in May, fourteen times what it was in October 2024, and converting 54% better than traffic from non-AI sources. A year earlier it converted about half as well. The agent turned out to be a good front door and a bad checkout, and the market sorted that out in six months.

What a site has to offer the second reader

• True structured data, kept true. Adobe’s same report found 30 to 40% of the content on high-value retail pages is overlooked or uncaptured by AI. On Shopify that means the metaobjects and metafields are the product, and the theme is one rendering of it.
• A plain-text version. Since February, Cloudflare will convert a page to Markdown on the fly when a client sends an Accept header of text/markdown, and reports the token count back in a response header. That is content negotiation, a feature of HTTP from the beginning, finally used for what it was for.
• Tools instead of screens. The Model Context Protocol went to the Linux Foundation’s Agentic AI Foundation in December, co-founded with OpenAI and Block, and its SDKs were seeing around 97 million downloads a month by March. Chrome’s WebMCP origin trial in version 149 lets a page expose JavaScript functions and forms to a browser agent directly, so the agent calls the function rather than guessing at the button.
• A decision about identity. Signed agents can be told apart from crawlers now. Decide what each is allowed to do, and whether some of it has a price.

The browser as a product is still unsettled. OpenAI shut the Atlas browser on 9 August, ten months after launching it, and folded the agent into the ChatGPT app. Perplexity put Comet on Android ten days later. Google said in May it would bring a persistent agent into Chrome over the summer. Which window the agent lives in will keep changing. That it will read your site as a set of tools and data rather than a set of pixels will not.

What does not change

I have worked with MCP long enough to have opinions about it, and I have shipped an agent that refuses to answer unless it has fetched the data first. I have not integrated a storefront with UCP or WebMCP yet, and I am not going to write as if I had. What I can say is that the work that ages well here is the work that was already correct: the schema designed from the problem, the content modelled as data with the theme as one view of it, the structured output that validates against a contract, the fetch at the moment of the answer.

The other thing that does not change is the conversation before any of that. An agent can compare four storefronts in a second. It cannot ask a founder the question that turns a family recipe into a grandmother’s recipe. The reader is changing. The person who has to decide what to build for them is not.

Where the numbers come from

• Cloudflare Radar, bot share of HTML requests, posted by Matthew Prince on 3 June 2026. Cloudflare Radar crawl-to-refer ratios, 28 days to 21 July 2026.
• Imperva, 2026 Bad Bot Report, 29 April 2026.
• SparkToro on Similarweb panel data, US, January to April 2026, published 9 June 2026.
• Cloudflare on Web Bot Auth and pay per crawl, 1 July 2025. The mixed-use crawler policy, reported by TechCrunch on 1 July 2026. Cloudflare changelog, Markdown for Agents, 12 February 2026.
• Search Engine Land on the Instant Checkout change, 6 March 2026. Modern Retail on what went wrong, 27 March 2026.
• Shopify on the Universal Commerce Protocol, 11 January 2026. Search Engine Land on Google Marketing Live, 20 May 2026.
• Google and the FIDO Alliance on the Agent Payments Protocol, 28 April 2026.
• Adobe Digital Insights, reported by Digital Commerce 360, 17 June 2026.
• Anthropic on donating MCP, December 2025. WorkOS on MCP adoption, 26 March 2026. Chrome for Developers on I/O 2026, 19 May 2026.
• OpenAI Help Center on retiring Atlas, August 2026.