Robots.txt Checker — Indexability & AI Crawler Rules

Description

Check robots.txt, meta robots, X-Robots-Tag and AI crawler access (GPTBot, ClaudeBot, Google-Extended) for any URL. Robots Checker PRO tells you whether the page you are looking at can be crawled and indexed — and, more importantly, why. Open the popup on any page and you get one clear verdict, plus every signal behind it: the robots.txt rule that matched, the meta robots tag, the X-Robots-Tag header, the canonical URL and the HTTP status. Every finding links back to its source, right down to the line number in robots.txt. WHY IT IS DIFFERENT Most robots checkers do a rough substring match and hope for the best. This one implements the matching algorithm documented in the Robots Exclusion Protocol (RFC 9309): • The most specific user-agent group wins, not the first one found • The longest matching path wins, and Allow beats Disallow on a tie • Wildcards and end-anchors are handled properly • Records sharing a user-agent are merged before matching • Matching runs against path plus query string So when a crawler has its own group with "Allow: /", you will see that this overrides a site-wide "Disallow: /checkout" — instead of a false alarm. AI CRAWLERS, SPLIT BY WHAT THEY DO The companies behind them each run several crawlers with different jobs, and blocking one does not block the others. The AI Bots tab groups them by purpose — collecting pages for model training, building the index an assistant searches, and fetching a page live when someone opens a link inside a chat — so you can see at a glance which kind of access your site actually allows. It also shows whether a crawler is stopped by a rule that names it directly, or is only inheriting the generic wildcard group. That distinction is the mistake most sites make when they believe they have opted out of AI training. WHAT YOU GET Status — the verdict and the full chain of signals that produced it, with a click-through to the exact robots.txt line. robots.txt — the file parsed into groups, the group that applies to your chosen crawler highlighted, the winning rule marked in the source, and a tester for any URL on the site. Directives — every robots meta tag and X-Robots-Tag header broken into individual tokens, each marked as applied or ignored for the selected crawler. Canonical is read from both the HTML and the Link HTTP header. AI Bots — the crawler matrix described above. Links — every link on the page checked against robots.txt, with rel=nofollow, sponsored and ugc flagged, filters, and optional highlighting directly on the page. Sitemaps — sitemaps declared in robots.txt plus the usual locations, checked for HTTP status and URL count. Tools — shortcuts to the webmaster and page-speed tools you already use for the current URL, a JSON export of the whole report, and the optional toolbar badge. CHOOSE YOUR CRAWLER A picker in the header re-evaluates everything for any of the 27 user agents the extension knows: major search engines, the assistant crawlers above, social preview fetchers and SEO tools, or the generic wildcard. The answer genuinely differs between them, so a single blended verdict would be misleading. OPTIONAL TOOLBAR BADGE Turn it on and every page you visit is checked automatically, with a green tick or a red cross painted on the extension icon. It follows whichever crawler you picked. The permission is optional, is never requested on install, and can be revoked at any time. PRIVACY No analytics. No telemetry. No account. No server of ours. Everything is computed locally in your browser. The only network requests go to the site you are already viewing, to read its robots.txt, its response headers and its sitemaps — and they are issued from the page's own context. Nothing is ever sent anywhere else. ACCESSIBILITY Light and dark themes, both verified to meet WCAG AA contrast on every screen. Full keyboard navigation with proper ARIA tab semantics and visible focus rings. Free, with no locked features and nothing to upgrade.

Details & SEO