The agent-ready website: agents read your page, not your metadata

You can spend an afternoon making your website agent-ready — an llms.txt file, a crawler policy in robots.txt, schema markup checked on every page — and most of what you've just done will have no effect on whether an AI system ever quotes you. I know, because I spent that afternoon in April, doing exactly what every guide to the AI-readable web was recommending at the time.

I'm not about to tell you to rip it out. But the evidence that has landed since is unusually clear for this corner of the industry, and it's awkward for anyone currently selling an agent-readiness checklist. Google's own guidance, updated in June, puts it about as plainly as Google ever puts anything: “You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.”

An agent-ready website comes down to two things: writing a page whose visible words answer the question properly, and deciding which agents you're willing to let read it. Everything in the layer between those two — the files, the markup, the tactics with the confident three-letter names — is either doing something other than what you were told, or it's doing nothing at all. Working out which of your own effort sits where is worth more than another week of implementation.

Nobody is reading your llms.txt

The llms.txt idea is a good one on paper. Give AI systems a single curated map of your site in plain Markdown, so they don't have to reverse-engineer your navigation. It's the sort of proposal that ought to work.

The trouble is that nobody has agreed to read it. John Mueller of Google makes the practical point best: you can tell from your own server logs that the AI crawlers don't even request the file. He has compared the whole idea to the old meta keywords tag, which is a comparison with teeth — meta keywords was also a sensible-sounding standard that publishers filled in dutifully for years while the search engines ignored it.

SE Ranking scanned 300,000 domains in March and found roughly one site in ten had an llms.txt file. When they built a model to predict which domains get cited by AI systems and then removed llms.txt as a variable, the model got more accurate. The file wasn't a weak signal. It was noise.

I've kept mine, and I'd suggest you keep yours if you've already written one — it works as a decent one-page brief on what this site is about, and OpenAI and Anthropic have never actually said they won't read it. Just don't mistake it for a strategy, and please don't pay anyone to write one for you.

What Ahrefs found when they tested schema properly

Structured data is a harder case, because the correlation looked so convincing. Ahrefs examined six million URLs and found that pages cited by AI were roughly three times more likely to carry JSON-LD markup. That figure has been quoted everywhere for the past year, usually as proof that schema is the lever.

To their credit, Ahrefs didn't believe their own headline. Schema markup tends to live on well-maintained sites, and well-maintained sites also publish better content, earn more links and hold more authority — so they went and did the harder study. They tracked 1,885 pages that added JSON-LD between August 2025 and March 2026, matched each one against a control page that didn't, and measured what happened to citations.

Nothing happened. Citations in Google AI Overviews fell 4.6% relative to the control group. AI Mode moved up 2.4% and ChatGPT 2.2%, both of which are indistinguishable from noise.

Then they ran the test that explains why. They watched what ChatGPT, Claude, Perplexity, Gemini and Google's AI Mode actually pulled when fetching a page live, and every one of them extracted only the visible HTML. JSON-LD, hidden microdata, hidden RDFa — all ignored. The machines weren't reading the part we'd all been carefully filling in.

Keep your schema anyway. It still earns its place in Google's classical index, where rich results and entity resolution depend on it, and a page that ranks well conventionally gets pulled into AI answers as a consequence. Commerce is a real exception too: an agent comparing products lives or dies on structured specifications and machine-readable pricing, which is the argument I made in the agent-first brand. But if what you publish is words, schema is housekeeping. It won't get you cited.

Writing for an impatient reader

That leaves the visible page. Most of the work of an agent-ready website sits there, and it's going to sound disappointingly familiar.

An AI system fetching your article behaves a bit like an impatient reader with a specific question. It takes the rendered text, looks for the passage that answers what it was asked, and lifts that passage. It isn't grading your site architecture or admiring your taxonomy. So the practical job is writing pages that survive being read that way.

1. Answer the question inside the paragraph, not across the page. Models quote passages, not documents. If your answer only makes sense to someone who has read the three paragraphs above it, it can't be lifted cleanly, and it won't be. Each substantive point wants to stand on its own feet.
2. Put the claim before the context. Journalists worked this out a century ago. An agent that reads your first two hundred words and stops should still come away with your actual answer, not your throat-clearing.
3. Be specific enough to be quotable. A number, a date, a named source, a price. Think of the average professional services page — four hundred words about bespoke solutions for growing businesses, without ever saying what it costs, who it's for, or which sectors it knows. There's nothing on a page like that for a model to lift and hand to somebody. “Contact us for pricing” is a wall an agent won't climb, and “solutions tailored to your requirements” gives it nothing at all to work with.
4. Say who wrote it and where the facts came from. A visible author byline with real credentials, and links out to your primary sources. This is the one piece of Google's guidance that reads less like SEO advice and more like publishing ethics, which is probably why it holds up.
5. Make sure the words exist before the JavaScript runs. If your content is assembled client-side, some fetchers will see an empty shell and move on. Server-render anything you want quoted.

There's nothing new in any of that. It's what good technical writing has always looked like. What's changed is that the audience for it now includes software that will never see your page at all, only the sentence it took from you.

Who gets in, and on what terms

The genuinely new development since April has nothing to do with readability.

For three decades robots.txt was a polite request. You wrote down which crawlers you'd rather not have, and well-behaved bots honoured it because honouring it was cheaper than the reputational cost of not doing. Nothing stopped anyone forging a user-agent string and helping themselves.

Mine is a fair example of what that gets you. I block CCBot, Bytespider and Diffbot, and I explicitly allow OAI-SearchBot and PerplexityBot. Both of those lines are licensing decisions dressed as technical configuration: I've decided that AI systems answering a reader's question and linking back are welcome, and that bulk harvesting for model training isn't. I wrote a commercial position into a text file, and until recently I had no way whatsoever to enforce it.

Web Bot Auth changes that arithmetic. You can now tell a verified agent from something merely claiming to be one, and treat the two differently — serving full product detail and live pricing to an operator you've chosen to work with, and considerably less to an anonymous fetcher. The mechanism is a cryptographic signature on every request, using the same HTTP message-signature standard that underpins a lot of modern API security, instead of a user-agent string you simply have to believe. And it's real rather than aspirational: Cloudflare proposed it, the IETF chartered a working group this year, it's been running at Cloudflare's edge since March, and both Visa and Mastercard have built their agent-payment authentication on top of it. If you're behind Cloudflare, most of this is a policy toggle rather than an engineering project, which is a rare and pleasant thing to be able to say about a new standard.

So put a different question to your own site: what would you charge for, if you could? It's worth sorting your pages on paper before you touch a config file, because for most businesses the answer falls out fairly quickly. Your articles, marketing pages and case studies you want read and quoted as widely as possible, so leave those open to anything that cites and links back. Your pricing, product data and specifications are what an agent actually needs to shortlist you, so those want to be available to verified buying agents and to nobody else. Your proprietary research and customer data — the material a competitor would happily pay for — is where blocking is the right default, and where verified identity finally makes blocking mean something. Write the policy down somewhere a person can read it too, as I've done in my llms.txt use policy, so there's no ambiguity about your intent even where there's no mechanism to enforce it.

Your website used to be a shop window, and the entire discipline of optimising it assumed you wanted every passer-by to look in. It's becoming something closer to a supplier feed, and suppliers don't hand their price list to everyone who asks.

Where this could still go wrong

All of this describes the web as it behaves today, and I don't know how long that stays true. Parsing JSON-LD is trivial — any competent developer could add it to a retrieval pipeline in an afternoon — so the current behaviour probably reflects convenience more than any considered judgment about what's useful. If the models start reading structured data properly, the schema advice flips straight back, and anyone who ripped theirs out on the strength of one study will have been badly served by it. Which is the real reason to leave yours where it is.

I'm also not convinced access control is genuine leverage for a business the size of mine. Gating my writing would be self-defeating; I'd rather be read and cited than protected. The businesses this really helps are the ones whose data has a market price — publishers, marketplaces, anyone with a catalogue. If you're a consultancy with forty pages of thought leadership, verified-agent policies are an interesting thing to understand and a poor thing to spend money on this quarter.

And you still can't see whether any of it worked, which is a difficulty I've written about at more length in the answer engine visibility audit — so do the work, and hold your conclusions loosely. What's left standing after a year of confident advice is the work I'd have recommended anyway: write something true, write it clearly enough that a stranger could quote a single paragraph without misrepresenting you, and put your name on it.

Quick answers

Do I need an llms.txt file? No — Google has confirmed it doesn't use one, the AI crawlers don't request it, and a 300,000-domain study found it added noise rather than signal to citation prediction. If you've already got one it's harmless, so leave it alone and don't build a plan around it.

Does schema markup help me get cited by AI? Not directly, on the current evidence. Ahrefs tracked nearly two thousand pages adding JSON-LD against matched controls and found no meaningful citation lift on any platform, and when they tested what AI systems extract from a live page fetch, all five major systems read only the visible HTML. Schema still matters for Google's conventional index, and it matters a great deal for product and pricing data, where an agent is comparing options instead of reading prose. For articles, treat it as hygiene you keep rather than a lever you pull.

What should I do first, and how will I know it worked? Read your most important page as though you were an agent with one question and no patience — if the answer isn't sitting in a self-contained paragraph near the top, with a name attached and a source linked, fix that before you touch anything technical. As for knowing it worked: you largely can't, and anyone promising otherwise is selling you a dashboard built on inference. That part is still genuinely unsolved.