About Robots.txt Generator
A robots.txt file sits at the root of a site and tells crawlers which paths they may request, following the Robots Exclusion Protocol (RFC 9309). This generator writes one group with a User-agent line, a Disallow rule for each path you list (or a single rule that blocks everything), and an optional Sitemap line. It creates one user-agent group only, so add further groups by hand if different bots need different rules.
How to use Robots.txt Generator
- Fill in "User-agent", "Default rule", "Disallowed paths" and "Sitemap URL".
- The result appears below and updates as you type.
- Copy the result with the copy button, or download it as a file where that is offered.
Questions about Robots.txt Generator
Does Disallow in robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or header.
Where must the robots.txt file be placed?
At the root of the host it applies to, for example https://example.com/robots.txt. Crawlers do not look in subfolders, and each subdomain and protocol needs its own file.
Runs in your browser. Page last updated .