robots.txt Explained: How to Control Search Crawlers

Every search engine that visits your site first knocks on a small, plain-text door: robots.txt. That one file tells crawlers which pages they may visit and which they should leave alone — and a single typo in it can hide your whole site from Google.

What robots.txt actually does

A robots.txt file sits at the root of your domain, for example https://example.com/robots.txt, and it is the first thing a well-behaved crawler reads before fetching anything else. It is a set of instructions, not a security system: it tells crawlers what you would prefer they do. Google, Bing, and other major search engines honor it. Malicious bots, scrapers, and anyone typing a URL directly into a browser will ignore it completely.

That distinction matters. Robots.txt is a traffic-management tool, not an access control. If a page must stay private — a staging site, an admin panel, customer data — protect it with a password or authentication, not with a disallow rule. A disallowed URL can still be indexed if other sites link to it, and the URL itself is visible to anyone who reads your robots.txt file. Think of robots.txt as a polite sign on the door, not a lock.

The four directives you actually need

Almost every robots.txt file is built from just a few lines. Each rule block starts with a User-agent line that says which crawler the rules apply to, followed by the paths that crawler may or may not fetch.

• User-agent: the crawler the block applies to. An asterisk (*) means all crawlers. Googlebot targets Google's main crawler specifically, and a specific block overrides the general one for that crawler.
• Disallow: paths the crawler should not fetch. Disallow: /private/ blocks everything under that folder. Disallow: / blocks the entire site — the most dangerous line in the file.
• Allow: exceptions inside a disallowed path. Disallow: /docs/ plus Allow: /docs/public/ blocks the folder but keeps one subfolder open. The most specific matching rule wins.
• Sitemap: points crawlers at your XML sitemap so they can discover pages efficiently. It goes outside any User-agent block, usually at the end: Sitemap: https://example.com/sitemap.xml.

Matching is prefix-based, which surprises people. Disallow: /blog blocks /blog, /blog/post, and also /blogging, because every one of those paths starts with /blog. If you mean only the folder, write Disallow: /blog/ with the trailing slash. Two wildcards extend the syntax: * matches any sequence of characters, and $ marks the end of a URL, so Disallow: /*.pdf$ blocks every PDF on the site while leaving HTML pages alone.

Worked example: a small site's robots.txt

Imagine a small business site with public pages, an admin area, a search-results page that generates infinite URLs, and a sitemap. A sensible file looks like this:

• User-agent: * — these rules apply to every crawler.
• Disallow: /admin/ — keep crawlers out of the admin area (and password-protect it too).
• Disallow: /search? — stop crawlers from following every search query into an endless maze of near-duplicate pages.
• Disallow: /checkout/ — no search value in cart and checkout pages, and crawling them wastes your crawl budget.
• Allow: / — everything else is open by default, stated explicitly for clarity.
• Sitemap: https://example.com/sitemap.xml — one line that helps every crawler find the pages that matter.

Notice what is not blocked: images, CSS, and JavaScript files. Google needs to fetch those to render pages properly, and blocking them used to be common advice that is now actively harmful. If a crawler cannot load your stylesheets, it cannot tell whether your page is mobile-friendly, and your rankings can suffer for it.

Common mistakes that deindex sites

Robots.txt errors are quiet and expensive, because nothing visibly breaks — pages simply vanish from search results. The mistakes that cause the most damage:

• A leftover Disallow: / from staging. Developers often block a staging site, then copy the file to production unchanged. Always check the live file after launch.
• Blocking the wrong prefix. Disallow: /wp-admin is fine; Disallow: /wp blocks far more than intended. Read each line as a prefix, not a folder.
• Case sensitivity. Paths are case-sensitive: Disallow: /Private/ does not block /private/. Match the real URL casing exactly.
• Assuming disallow equals deletion. Blocking a page that is already indexed does not remove it — Google may keep showing the URL without a description. To actually remove a page, use a noindex tag or Google's removal tool.
• Forgetting the file exists. A robots.txt from three redesigns ago can quietly block a new section. Review it whenever the site structure changes.

There is also a crawl-budget angle that matters for larger sites. Every URL a crawler fetches costs time, and sites with thousands of pages — faceted product filters, calendar archives, tag pages — can burn their budget on junk while important pages wait. Strategic disallow rules steer crawlers toward the pages worth indexing and away from the infinite corridors.

How to test before you trust it

Never deploy a robots.txt edit on faith. Google Search Console includes a robots.txt tester that shows exactly which URLs a given rule blocks, and it is free. Paste your file, enter a URL, and confirm the verdict before and after each change. Independent validators exist too, and they catch syntax issues like a missing User-agent line or a misplaced wildcard.

After publishing, fetch the live file at yourdomain.com/robots.txt and read it with your own eyes — cached copies and deployment quirks have embarrassed many site owners. Then watch Search Console's coverage report for a week: a sudden spike in "blocked by robots.txt" exclusions is your early warning that something went wrong.

Robots.txt is one of the smallest files on your site and one of the most powerful. A few careful lines keep crawlers focused on your best content, protect your server from wasted requests, and keep private corners out of search results. Write it deliberately, test it, and review it whenever your site changes.

Frequently asked questions