Media
2 items
Description
Check if websites have llms.txt and llms-full.txt files, or generate them using a browser-based crawler. Make your website AI-ready in seconds! The emerging /llms.txt and /llms-full.txt standard is a simple yet powerful way to help AI search engines, language models (LLMs), and agentic web crawlers quickly discover, crawl, and understand your website's content. The llms.txt Checker & Generator is a sleek, professional developer utility that lets you inspect any website's AI readiness and generate custom llms.txt and llms-full.txt files on the fly—directly from your browser. --- 🚀 Key Features: • 🔍 Instant Verification: Checks any website instantly upon opening the popup to see if /llms.txt or /llms-full.txt files are already active. • 🕸️ Client-Side Crawler: No existing files? No problem. The extension automatically fetches and parses the website's sitemap.xml (or falls back to homepage links) to discover and crawl up to 20 public pages. • 🤖 Smart Metadata Extraction: Scrapes titles, meta descriptions, and structural headers (H1–H3) from crawled pages to construct high-quality, structured summaries. • ✏️ Interactive Markdown Editor: Tweak and refine the generated files instantly in a beautiful side-by-side preview and editor before exporting. • 💾 Easy Export: Copy markdown output directly to your clipboard or download standard-compliant .txt files with a single click. • 🔒 Privacy-First & Secure: 100% client-side execution. Web crawler and generation logic run entirely inside your browser's extension context. No analytics, tracking, or third-party servers. --- 📖 What is llms.txt? The /llms.txt standard is a proposed specification for a text file placed at the root directory of a domain (e.g., yoursite.com/llms.txt). It functions like robots.txt or sitemap.xml, but is tailor-made for LLMs: • /llms.txt: Serves as a high-level site map with brief descriptions and curated links to crucial pages. • /llms-full.txt: Serves as an expanded directory containing more detailed markdown summaries, page titles, descriptions, and headings to allow deep indexing and context retrieval by AI agents. --- ⚙️ How It Works: 1. Navigate to any website. 2. Click the llms.txt Checker & Generator icon in your browser toolbar. 3. The extension instantly checks for existing llms.txt or llms-full.txt files. 4. If missing, click "Generate" to let the background worker crawl public pages. 5. Review the generated markdown in the editor, customize the structure if needed, and download! --- 🛡️ Secure & Safe Crawling: Built with developer safety in mind: • Strictly adheres to safe URL validation rules, blocking attempts to query private, local, loopback, or reserved IP subnets. • Features automatic request timeouts and redirect loops prevention. • Respects cross-origin boundaries securely. Get your website optimized for the AI era. Install llms.txt Checker & Generator today