Robots.txt Generator

A robots.txt file is a small plain-text file placed at the root of a website (example.com/robots.txt) that tells well-behaved web crawlers — search engine bots, SEO tools, archivers — which parts of a site they may or may not request. It was proposed in 1994 by Martijn Koster after a crawler accidentally overloaded his server, and it has remained essentially unchanged since: a simple, voluntary protocol (the Robots Exclusion Protocol) with no enforcement mechanism beyond crawler goodwill. Google, Bing and every other major search engine respect it, which is what makes it useful, but it is not a security or access-control tool — anything listed as Disallow is still publicly reachable by anyone who knows or guesses the URL, and malicious bots simply ignore the file entirely. The file's syntax is a short list of directives grouped under one or more User-agent lines. A group starts with User-agent: (either a specific bot name like Googlebot, or * to match every crawler that doesn't have a more specific group of its own), followed by any number of Allow and Disallow lines giving paths that crawler may or may not fetch, and optionally a Crawl-delay line asking the bot to wait a given number of seconds between requests (Googlebot ignores Crawl-delay entirely, controlling its own crawl rate through Google Search Console instead, but Bing and several other crawlers still honour it). A Sitemap: line, which can appear anywhere in the file and applies site-wide regardless of which user-agent group it's near, points crawlers to your XML sitemap so they can discover pages more efficiently than by following links alone. Getting robots.txt wrong is one of the most common self-inflicted SEO mistakes, because a single misplaced rule can accidentally de-index an entire site: a stray `Disallow: /` under `User-agent: *` — which blocks every path for every crawler — is a frequent culprit found on sites that mysteriously vanish from search results. This is why this tool offers three presets as safe starting points: "Allow all", which produces a file with no Disallow rules at all (equivalent to having no robots.txt, but useful as an explicit statement of intent); "Block all", the deliberate opposite — every crawler blocked from every path, appropriate only for staging or development environments that should never be indexed; and "WordPress default", which blocks the standard `/wp-admin/` directory while explicitly allowing `/wp-admin/admin-ajax.php`, matching the file WordPress itself generates for a typical production site. A few things worth knowing before publishing: robots.txt controls crawling, not indexing — a page blocked from crawling can still appear in search results (typically with no description) if other sites link to it, since Google can see the URL exists even without fetching its content; to actually keep a page out of search results, use a `noindex` meta tag or HTTP header on the page itself, which requires the page to be crawlable in the first place (a contradiction worth double-checking if you're trying to solve both problems). Directives are also case-sensitive and path-based, not full-URL matched, and the more specific rule generally wins when Allow and Disallow overlap on the same path — most crawlers apply the longest matching rule, not simply the last one listed.