Media
2 items
Description
See which AI crawlers your site allows or blocks. Checks robots.txt for 21 bots from OpenAI, Anthropic, Google and Perplexity. Check any site's AI crawler access in one click. Open the extension on any page and it reads that site's robots.txt, then shows you which AI crawlers are allowed in and which are blocked. No typing, no signup, no account. The website you are already looking at is checked automatically the moment the popup opens. What it checks: 21 AI crawlers across 11 companies, each one listed with the company behind it, the job it does, and whether the site currently lets it in. You also get the list of paths the site blocks for every crawler, and a direct link to the raw robots.txt so you can read the rules yourself. Why the distinction between bots matters: These bots are not interchangeable. They do three different jobs, and blocking the wrong one has very different consequences. Training crawlers, like GPTBot and ClaudeBot, collect pages that teach the next generation of models. What they read becomes part of what a model knows about you without searching. This is the slow layer, and it is the one most brands never think about. Live browsing crawlers, like ChatGPT-User and Perplexity-User, visit the moment someone asks an AI a question. What they read is used for that one answer and then discarded. This is how a page published last week can show up in an answer today. Search indexing crawlers, like OAI-SearchBot and Googlebot, build the index that AI search results are drawn from. A site can be wide open to one group and shut to another. Google-Extended is a common example: plenty of sites allow Googlebot for normal search while blocking Google-Extended, which quietly removes them from Gemini. The extension shows each crawler separately for exactly this reason. Use it on any website: The checker works on any domain, which turns out to be the more interesting use. Check a competitor to see whether they have opened up to AI crawlers or locked down. Check a publication before you pitch it. Coverage on a site that blocks AI crawlers cannot reach the models, no matter how good the placement is. Check a partner, a supplier, or any site you are about to invest content into. What it does not do: robots.txt is an honor system, not a lock. The major AI companies read it and follow it, which is why checking it is worth your time, but it tells well-behaved crawlers what they may do rather than physically stopping anyone. It is a sign on the door, not a deadbolt. This extension reads robots.txt and nothing else. It does not tell you whether a model has actually trained on your site, whether you appear in AI answers, or how you compare to competitors. Open crawler access gets AI to your door. What happens after that is a separate question, and an honest tool should not pretend otherwise. It also reads only the site-wide rules in robots.txt. Page-level controls such as a noai meta tag or an X-Robots-Tag header are not covered. Privacy: The extension collects nothing. No analytics, no tracking, no accounts, no server of ours in the middle. There is no backend. When you check a domain, your browser requests that domain's robots.txt file directly, the same request your browser makes for any page. Nothing about what you check is sent to us or to anyone else, because there is nowhere for it to go. Fonts and icons ship inside the extension, so the popup makes no third party requests either. Permissions and why: activeTab: reads the URL of the tab you are on so the domain fills in by itself. It is granted only when you click the toolbar icon, and it goes away when you navigate away. Access to robots.txt files: the permission is scoped to the path /robots.txt on any host. That scope is deliberate. It means the extension is technically incapable of reading page content, form data, or anything else on the sites you visit. It can fetch one file, and that file is public by design. Chrome's install warning is written in terms of hosts rather than paths, so it sounds broader than what the extension can actually do. The manifest is short and readable if you want to confirm it. Created by Centium AI: Centium measures brand visibility across the major AI assistants. We run hundreds of prompts per brand and help them optimize to be recommended by AI models. We built this checker because crawler access is a key variable in being cited by AI models, and it is the one part of AI visibility a team can verify in seconds. Free to use, with no strings attached. Learn more at centium.ai.