Free Robots.txt Generator
Tell search engines where they may and may not go — a valid robots.txt in under a minute.
Crawl rules
How to use the robots.txt generator
- Add one rule per crawler group: pick the user agent, choose Allow or Disallow, and enter the path.
- Optionally add your sitemap URL and a crawl delay.
- Copy the output or download the file, then upload it as
robots.txtin your site's root (e.g.example.com/robots.txt).
Robots.txt done right
The file lives at your domain root and speaks to every well-behaved crawler before it fetches a page — Googlebot, Bingbot, and the rest check it first. Rules are prefix matches: /admin blocks /admin, /admin/users, and /administrator alike, so be precise and test with Search Console's robots.txt tester before deploying. A common, safe default is allowing everything and only disallowing genuinely private areas (admin panels, cart internals, staging subdomains); blocking too aggressively is how sites accidentally deindex themselves. Understand the critical limit: robots.txt controls crawling, not indexing — a disallowed page can still appear in search results if other sites link to it, just without a description. For pages that must stay out of the index entirely, use a noindex meta tag instead (which requires the crawler to fetch the page, so do not disallow it too). Always declare your sitemap — it is the single most useful line in the file for discovery — and keep crawl-delay modest or omit it, since aggressive delays slow down how fast new content gets found.
Frequently asked questions
Where does robots.txt go?
In the root of your domain: https://example.com/robots.txt. Crawlers only check that exact location.
Does robots.txt keep pages out of Google?
Not reliably — it blocks crawling, not indexing. Use a noindex meta tag (and allow crawling of that page) for true deindexing.
What does the * user agent mean?
It matches every crawler that does not have its own specific group. Put general rules under * and exceptions under named agents like Googlebot.
Do I need a crawl delay?
Usually no. Only add one if a crawler is hammering a small server — and note that Google ignores crawl-delay (use Search Console's crawl settings instead).