Some Sites Use llms.txt Like robots.txt, Common Crawl Finds via @sejournal, @Mat

<p>Ahrefs analyzed 137K domains and found 97% of llms.txt files received zero requests. AI retrieval bots accounted for 1% of total requests. </p> <p>The post <a

<p>Mueller argues LLM systems can't use llms.txt to differentiate between websites for discovery. He sees a narrow role once an agent is on a site.</p> <p>The post <a

<p>Google Search says llms.txt isn't needed for AI features, while Lighthouse now checks the file for agentic browsing readiness in an experimental audit.</p> <p>The post <a

Using Ahrefs Web Analytics and Bot Analytics, we analyzed the server logs and live traffic of 137K domains, plus the user agents hitting all of them. Here’s what we found. In late

<p>A v2 update to the llms.txt spec has shipped, adding formal link relations that help AI agents locate Markdown versions of pages.</p> <p>The post <a

<p>In December 2024, Jeremy Howard, the creator of fast.ai, proposed a simple standard: an llms.txt file placed in a domain's root directory, telling AI systems about the

<p>Google says a core assumption driving LLMs.txt adoption conflicts with the purpose its creators originally intended.</p> <p>The post <a

<p><img alt="What is llms.txt &amp; should you use it?"