Media
3 items
Description
Audit the crawl layer of any site — sitemap, robots.txt, llms.txt, AI bot access, and URL health. By Sukat App. Sukat Crawler audits the crawl layer of any website in a single click — what crawlers see, who's allowed in, what's broken, and how AI-ready the site is. Built for SEO professionals, technical owners, and anyone shipping content they want crawlers and AI search to actually find. WHAT IT DOES - Detects sitemaps — parses robots.txt for Sitemap: directives and probes 12 common paths (/sitemap.xml, /wp-sitemap.xml, Yoast variants). Follows <sitemapindex> files to count real total URLs across sub-sitemaps. - Generates one if missing — crawls the site (same-origin BFS) and produces a valid sitemap.xml for download. - Validates every URL — HEAD requests against each entry to surface 404s, redirects, and server errors. Concurrency-limited, cancellable, exportable. - Detects llms.txt — checks /llms.txt and /llms-full.txt for the emerging AI-readable site-map convention. - AI crawler access audit — checks your robots.txt rules against 13 major AI crawlers from OpenAI, Anthropic, Google, Perplexity and others, and shows an allow/block matrix grouped by vendor. - AI Readiness score — a single 0–100 number weighing sitemap hygiene, llms.txt presence, AI search bot access, and explicit AI configuration. - robots.txt viewer — collapsible view of the raw file, with copy + open-in-tab buttons. - Export URLs — CSV, JSON, or TXT, with validation results included if the validator has run. - Lastmod uniformity warning — flags the common CMS bug where every URL shares the same <lastmod> date. HOW TO USE 1. Open any http(s):// site. 2. Click the Sukat Crawler icon — the intro screen appears. 3. Click "Scan this site" — every signal is audited in one pass. 4. On the sitemap card, click "Validate N URLs" to check every URL's HTTP status. 5. Use the Export URLs row for CSV / JSON / TXT, or Download XML for the raw sitemap. PRIVACY Everything runs locally in your browser. The extension makes HTTP requests only to the site you're scanning (to fetch robots.txt, sitemaps, and validate URLs) and to your own browser's storage (for settings). No third-party servers, no analytics, no tracking. PART OF SUKAT APP Sukat App is a privacy-first toolkit for site owners. Sukat compresses images to an exact file size locally in your browser; Sukat Crawler audits the crawl layer of any site. Both run entirely in your browser. Learn more at https://sukatapp.com/crawler