Media
1 item
Description
Checks whether AI and LLM crawlers can access the current page. By The Organic Agency. AI Bot Friendly Page Analysis By The Organic Agency CHECK WHETHER AI CRAWLERS CAN ACTUALLY ACCESS YOUR PAGES As AI-powered search engines, answer engines, and large language models become an increasingly important source of brand visibility, the question of whether your website is accessible to AI crawlers is no longer optional. If GPTBot cannot read your pages, your content will not inform ChatGPT's answers. If ClaudeBot is blocked, Anthropic's models will not draw on your content. If OAI-SearchBot is turned away, your pages will not appear as cited sources in ChatGPT search results. Most website auditing tools will tell you what your robots.txt says. Very few will tell you what actually happens when each of those bots tries to visit your page. This tool does both, across 15 of the most important AI crawlers currently operating on the web. THREE LAYERS OF ANALYSIS IN ONE CLICK AI Bot Friendly Page Analysis runs three independent checks every time you analyse a page, giving you a complete picture of crawl accessibility rather than a partial one. Layer 1: robots.txt Analysis The tool fetches your site's robots.txt file and evaluates every rule against each of the 15 bots using the standard longest-match-wins specificity logic that crawlers themselves apply. This matters because many tools get this wrong. A site with a wildcard Disallow rule and named Allow overrides for specific bots will be incorrectly reported as fully blocked by tools that do not implement proper specificity handling. This tool does it correctly: if GPTBot has an explicit Allow rule, it will be shown as allowed even when a wildcard Disallow is also present. The full matched rule for each bot is recorded in the JSON export, so you can see exactly which rule applied and to which path. Layer 2: Page-Level Tag Inspection The tool fetches the active page and scans for AI-specific blocking directives in two places. X-Robots-Tag HTTP headers are server-level directives that apply to all resource types and cannot be overridden by on-page content. Meta robots tags in the page head carry directives such as noai, noimageai, noindex, nofollow, noarchive, and none. Any blocking tokens found are clearly flagged, along with whether they appear in headers or in the page markup. Layer 3: Infrastructure Probe This is the most technically involved check, and the one most likely to reveal blocking that other tools miss entirely. For each of the 15 bots, the tool uses Chrome's Declarative Net Request API to temporarily substitute the outgoing user-agent header, then sends a real HTTP request impersonating that bot to a cache-busted version of the page URL. The response is examined for HTTP status codes indicating blocking (403, 401, 406, 410, 451), Cloudflare challenge pages, and other WAF interstitial signatures. The tool also detects and reports the infrastructure provider handling the site, including Cloudflare, Akamai, Fastly, AWS CloudFront, and F5 BIG-IP. This layer reveals situations where robots.txt allows a bot but the CDN or WAF is blocking it anyway at the infrastructure level, which is a common real-world scenario that declared policy alone will not expose. 15 AI CRAWLERS CHECKED The tool covers the major AI platforms currently crawling the web: OpenAI: GPTBot (model training), OAI-SearchBot (ChatGPT search citations), ChatGPT-User (user-triggered browsing) Anthropic: ClaudeBot (primary crawler), Claude-Web (legacy user-agent still active), anthropic-ai (third Anthropic user-agent seen widely in server logs) Google: Google-Extended (Gemini model training and grounding) Apple: Applebot-Extended (Apple Intelligence training) Meta: Meta-ExternalAgent (Llama training and Meta AI products) Amazon: Amazonbot (Alexa and Amazon AI features) Perplexity: PerplexityBot (real-time AI search and answers) Common Crawl: CCBot (open training dataset used by many AI organisations) DeepSeek: DeepSeekBot (DeepSeek model training and search) ByteDance: Bytespider (TikTok and ByteDance AI products) You.com: YouBot (You.com AI search) CLEAR, ACTIONABLE RESULTS Results are presented in a clean interface with three collapsible sections corresponding to the three analysis layers. A colour-coded heatmap gives you an at-a-glance view of all 15 bots across all three check types simultaneously, making it immediately obvious which specific bots are blocked and at which layer. The overall status bar at the top of the results gives an immediate pass or fail verdict: Fully Accessible, Blocked, or Partial Restrictions. EXPORT IN THREE FORMATS Every analysis can be exported immediately from the extension footer. JSON export produces the complete raw report including the full robots.txt content sample, every matched rule per bot, all HTTP response metadata from the infrastructure probes, detected blocking tokens, and the identified infrastructure stack. This is suitable for inclusion in technical audits, feeding into client dashboards, or comparing results over time. CSV export produces a clean summary table with one row per bot covering all three check dimensions: robots.txt decision, matched rule, page-level blocking tokens, infrastructure state, HTTP status, block reason, and detected infrastructure. The file includes a UTF-8 BOM so it opens correctly in Microsoft Excel without any encoding configuration. Both exports are named with the analysis date for easy filing. BUILT-IN USER GUIDE A full user guide is accessible directly from the extension footer via the User Guide button. It opens in a new browser tab and covers: What each of the three checks does and how it works, including the technical detail behind the infrastructure probe and its limitations. How to read every result state including the heatmap cells, overall status pills, and infrastructure probe classifications. A complete table of all 15 bots with their owning organisations and purposes. A remediation section with robots.txt code examples showing how to correctly allow named bots when a wildcard block is in place, how to find and remove page-level blocking tokens, and how to approach WAF-level blocking at the infrastructure layer. A FAQ section covering the most common questions about crawl accessibility, analytics impact, the relationship between AI bots and standard search rankings, and platform-specific considerations such as Bytespider's ByteDance ownership. WHO THIS IS FOR SEO practitioners and digital marketing professionals who need to audit and report on AI crawl accessibility as part of their standard workflow, without needing to manually inspect robots.txt files, check HTTP headers, or run curl commands. Developers and site owners who want a quick confirmation that recent infrastructure changes, CDN configuration updates, or robots.txt edits have not accidentally broken AI crawler access. Agency teams who need to produce client-ready exports showing the current state of AI crawl accessibility across a site. PERMISSIONS USED activeTab: Required to read the URL of the current browser tab so the correct page can be analysed. declarativeNetRequest and declarativeNetRequestWithHostAccess: Required to temporarily substitute the outgoing user-agent header during the infrastructure probe phase. Rules are registered and cleaned up within each analysis run and are not persistent. Host permissions for all URLs: Required to fetch robots.txt files and send infrastructure probe requests to sites on any domain, since the tool is designed to work on any page you have open. No data is sent to any external server. All analysis is performed locally within your browser. ABOUT THE ORGANIC AGENCY AI Bot Friendly Page Analysis is built and maintained by The Organic Agency, a UK-based digital marketing consultancy specialising in SEO strategy for enterprise clients across automotive, retail, technology, and professional services sectors. theorganicagency.com Current version checks 15 AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, CCBot, Amazonbot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, DeepSeekBot, Bytespider, YouBot.