robots.txt Explained — A Practical Guide for Site Owners

Every public website has a robots.txt file at the root. Most sites never look at it, and most sites have a broken or default robots.txt. The file is small and the format is simple, but the consequences of getting it wrong are surprisingly large: a single misplaced line can hide your entire site from Google.

This guide is the practical version: what robots.txt does, the syntax, the most common patterns, the mistakes that cost you traffic, and how to write one that does what you want. The Uttir Robots.txt Generator produces a valid file from your rules in your browser.

What robots.txt actually does

robots.txt is a plain-text file at the root of your domain (e.g. yourdomain.com/robots.txt) that lists paths search engine crawlers should not fetch. It is a polite request — the major search engines honor it, but a malicious crawler can ignore it completely. For that reason, robots.txt is not a security mechanism. It is a crawl-control mechanism.

The reasons you would use it:

• Block admin and internal pages from being indexed (e.g. /admin/, /internal/)
• Save crawl budget by blocking duplicate, infinite, or low-value URLs (search engines have a finite time to spend on your site)
• Tell crawlers where your sitemap is (an explicit hint that helps discovery)
• Block specific crawlers that you do not want (some AI scrapers, certain analytics bots)

What it does not do:

• It does not prevent indexing. A page blocked by robots.txt can still appear in search results if other sites link to it. To prevent indexing, use the noindex meta tag (which requires the page to be crawlable for Google to see the tag — see below for the contradiction).
• It does not provide security. Anyone who wants to can fetch the URLs anyway.

The minimum useful file

For a public site, the smallest robots.txt that does anything useful is:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Three lines. The first says "this rule applies to every crawler". The second says "everything is allowed to be crawled". The third tells crawlers where the sitemap is. If your site has a homepage, blog, and product pages, this robots.txt covers you.

The syntax, line by line

The file is plain text with three kinds of lines:

User-agent

Specifies which crawler the following rules apply to. User-agent: * means "all crawlers". You can also target specific ones:

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Disallow: /private/

Most crawlers announce themselves with a unique user-agent string. The User Agent Parser can identify which one is hitting your site.

Allow and Disallow

Allow lets a crawler fetch the path. Disallow blocks it. The path can be a directory or a specific URL prefix:

Disallow: /admin/         # blocks /admin and everything below
Disallow: /private.html   # blocks a single file
Disallow: /*.pdf$         # (with regex mode) blocks all PDFs

The wildcard * matches any sequence of characters. The end-of-string $ anchors the match.

Sitemap

Points crawlers at your sitemap. You can have multiple Sitemap lines if you have more than one sitemap file (e.g. a main sitemap and an image sitemap):

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

Common patterns

Most sites use one of a small set of patterns. The right one depends on what you are trying to do.

Pattern 1: Allow everything, point to sitemap

For a public site with no admin or private areas:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This is the right default. Most sites should start here and add restrictions only as needed.

Pattern 2: Block admin and staging

For a site with a CMS, dashboard, or staging area:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /staging/
Disallow: /api/internal/

Sitemap: https://example.com/sitemap.xml

Pattern 3: Block specific file types from being crawled

For a site with PDFs, images, or other large files that do not need to be indexed:

User-agent: *
Allow: /
Disallow: /*.pdf$
Disallow: /*.docx$
Disallow: /downloads/

Sitemap: https://example.com/sitemap.xml

Note: the * and $ are wildcards. This works for all major search engines but is not strictly part of the original robots.txt spec — Google and Bing both support it.

Pattern 4: Block specific crawlers

For a site that wants to keep its content out of AI training sets or specific search engines:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml

Whether this actually does anything depends on the crawler honoring robots.txt. Major AI scrapers (GPTBot, CCBot, ClaudeBot, anthropic-ai) all say they do. Other scrapers vary.

The mistakes that cost you traffic

robots.txt is short, but the failures are severe. These are the most common:

Blocking the entire site by accident

User-agent: *
Disallow: /

This is the single most common catastrophic mistake. It blocks all crawlers from all pages. Sites with this file disappear from Google. If you copied a starter file and did not change the rules, double-check.

Blocking the sitemap

If your sitemap URL is under a path that is also disallowed, the crawler cannot fetch the sitemap to know what pages exist. Keep sitemap URLs in the allowed section.

Using Disallow when you mean noindex

Robots.txt blocks crawling, not indexing. If a page is blocked by robots.txt, Google may still index the URL (if other sites link to it) but cannot crawl it to check the noindex tag. The combination you usually want is:

• For a page you want crawled but not indexed: noindex meta tag
• For a page you do not want crawled: robots.txt Disallow

For pages that absolutely must not be indexed AND not crawled (rare — usually only for sensitive admin URLs), use both.

Case sensitivity

Paths in robots.txt are case-sensitive. Disallow: /Admin/ only matches /Admin/, not /admin/. Most CMSs use lowercase paths, so this is rarely a problem in practice, but be aware.

Trailing slash matters

Disallow: /admin matches /admin AND /admin/anything. Disallow: /admin/ only matches paths starting with /admin/. The difference is subtle but real.

How to test your robots.txt

After writing or editing robots.txt, three things to check:

1. Validate the format. Google Search Console has a robots.txt Tester under Settings (legacy) and in the new interface. It shows exactly which paths each crawler is allowed or blocked from.
2. Make sure it is reachable. Visit yourdomain.com/robots.txt in a browser. You should see the raw text. If you see a 404 or a 500, the file is not being served correctly.
3. Check coverage in Search Console. After a few days, the Pages report shows which URLs were excluded by robots.txt. If you see pages you wanted indexed in that list, fix the rule.

The robots.txt and meta robots tag, compared

These are the two ways to control what search engines do with your pages, and they work differently:

• robots.txt — a server file that controls which paths crawlers can fetch. Affects crawling, not directly indexing.
• meta robots tag — an HTML tag in the page head. Controls indexing for the specific page. The tag is <meta name="robots" content="noindex"> to prevent indexing.

The catch: for the meta robots tag to be respected, the crawler has to be able to fetch the page to see the tag. If robots.txt blocks the page, the crawler never sees the tag, and Google may still index the URL (just without a snippet).

For most cases, the right tool is the meta robots tag. robots.txt is for "do not even bother crawling this" (admin pages, search results, infinite spaces).

How to create a robots.txt file

For most sites, the file is a single static text file at /robots.txt. Three ways to make it:

Write it by hand

Open a text editor, type the rules, save as robots.txt, upload to the root of your site. Done.

Generate it

The Uttir Robots.txt Generator walks you through the rules and outputs a valid file in your browser. Useful when you have multiple sections or want to block specific crawlers.

CMS plugin

WordPress (Yoast, Rank Math, AIOSEO), Ghost, and most CMSes have a robots.txt editor in the SEO settings.

Common questions

Where does the file go?

At the root of your domain: yourdomain.com/robots.txt. Not in a subdirectory. Crawlers only look at one URL for the file.

Can I have multiple files?

No. There is exactly one robots.txt per host, at /robots.txt. If you have multiple subdomains, each can have its own (e.g. blog.example.com/robots.txt).

Does robots.txt prevent indexing?

Not directly. It prevents crawling. A page blocked by robots.txt can still appear in search results if other sites link to it — Google would just not have a snippet. To prevent indexing entirely, use the noindex meta tag.

What if I block Google accidentally?

You will see a sharp drop in organic traffic within days. The fix is to update the file, then use Google Search Console to request re-indexing of the affected pages. Recovery is usually fast — once the crawler can fetch the pages again, ranking recovers.

Is there a size limit?

Google's limit is around 500 KB or so for the file. Real-world robots.txt files are almost always a few hundred bytes. If yours is large, you are probably being too specific.

Bottom line

A robots.txt file is two lines for a public site, or a few more for sites with admin areas. Get the file right, submit your sitemap to Google Search Console, and you are done. The Uttir Robots.txt Generator builds a valid one in your browser, and the Sitemap Generator handles the sitemap it points to.