# robots.txt Generator

> Generate a valid robots.txt with allow and disallow rules, sitemaps, and AI crawler directives.

URL: https://uttir.com/robots-txt-generator
Categories: seo-tools, developer-tools
Privacy: Your rules are generated locally in your browser. Nothing is sent to a server.

## About

robots.txt tells search engines and other crawlers which parts of your site they may visit. A well-formed file prevents wasted crawl budget and can block unwanted bots — including AI crawlers.

Pick which paths to allow and disallow, add your sitemap, and optionally include a section for AI crawlers. This generator produces a clean, standards-compliant robots.txt you can upload to your site root.

## How to use

1. **Set global rules** — Allow and disallow paths apply to all crawlers, e.g. /admin should be disallowed.
2. **Add your sitemap** — Include the full URL, for example https://yoursite.com/sitemap.xml.
3. **Choose AI crawler rules** — Optionally add a section that explicitly allows or blocks generative-AI crawlers like GPTBot and ClaudeBot.
4. **Copy and upload** — Save the output as robots.txt in the root of your domain.

## Examples

### A typical marketing site

Allow everything, point to the sitemap, and greet AI crawlers.

Input:

```
Allow: / · Sitemap: https://example.com/sitemap.xml
```

Output:

```
A robots.txt with a universal allow rule, sitemap reference, and AI section.
```

## FAQ

### Where do I put robots.txt?

In the root of your domain, so it is served at https://yoursite.com/robots.txt. It applies to the whole site (subdomains have their own).

### Should I block AI crawlers?

That depends on your goals. If you want generative engines to cite and summarize your content, leave them allowed. If not, list their user agents under Disallow.

### Is robots.txt a security measure?

No. It is a request, not a barrier — any crawler can ignore it. Use real access controls for anything private.

## Related tools

- [Sitemap Generator](https://uttir.com/sitemap-generator) — Generate a clean XML sitemap for your website in seconds.
- [Meta Tag Generator](https://uttir.com/meta-tag-generator) — Generate title, meta description, canonical, Open Graph, and Twitter card tags as ready-to-paste HTML.
- [SERP Preview](https://uttir.com/serp-preview) — Preview how your page title and meta description will look in Google search results before you publish.

---

For the full HTML page with the live tool, visit https://uttir.com/robots-txt-generator.
This file is the markdown rendering at https://uttir.com/robots-txt-generator.md. See https://uttir.com/llms.txt for a site-wide summary.