OpenAI Says Robots.txt May Not Apply To ChatGPT’s Fetch Bot via @sejournal, @Mat

<p>OpenAI's public crawler docs now list OAI-AdsBot, a bot that may visit pages submitted as ChatGPT ads to check policy compliance and ad relevance.</p> <p>The post <a

<p>Akamai breaks down which AI bots are hitting publishing, who operates them, and why fetcher bots may pose a more immediate risk.</p> <p>The post <a

<p>New Ahrefs data shows Reddit pages appeared often in ChatGPT retrievals but rarely as visible citations.</p> <p>The post <a

<p>Google's John Mueller explains a robots.txt quirk that can cause Googlebot to ignore robots.txt rules.</p> <p>The post <a

<p>A paper tested whether Google results and OpenAI's API can stand in for ChatGPT's citations. They matched cited sites more often than the exact pages.</p> <p>The post <a

<p>Cloudflare now lets all sites manage AI crawlers as Search, Agent, or Training, with Sept. 15 defaults that can block Googlebot for sites blocking training.</p> <p>The post <a

<p>Search Console reported 51,000 URLs as "Indexed, though blocked by robots.txt," and Google says that's not necessarily a problem.</p> <p>The post <a

<p>OpenAI added WebMCP support to ChatGPT’s desktop browser, letting supported webpages expose structured actions to ChatGPT Work and Codex.</p> <p>The post <a