Agent-ready websites provide machine-readable routes for discovery and content access without replacing the standard HTML experience. The proposed llms.txt file, placed at a site’s root, acts as a curated index of useful pages; unlike robots.txt, it does not grant or restrict access. It has no standards-body adoption, and whether AI crawlers use it is not confirmed, despite adoption by major documentation providers.

WordPress sites can also serve page content as Markdown, either at a separate .md URL or through content negotiation when a request includes Accept: text/markdown. A plugin can generate and store Markdown when content is published or updated, letting both delivery methods use the same copy. This reduces duplication and gives agents a lighter version of the page; Cloudflare has cited an example falling from 16,180 tokens in HTML to 3,150 in Markdown.

Negotiated responses require particular care in WordPress caching layers and CDNs. The server should return Vary: Accept, and caches must keep HTML and Markdown variants separate to avoid serving Markdown to browser visitors. Cloudflare’s Markdown for Agents instead converts a site’s HTML at the edge, adding JSON-LD when present and an estimated token count. It avoids maintaining a second content copy, but the result depends on how well the original HTML can be parsed.

Author of the original text: Miriam Schwab (Elementor)