Build a robots.txt file from allow and disallow rules, a crawl delay and your sitemap URLs. Fast, free, and built for simple browser-based workflows.
* for all crawlers, or a name such as Googlebot.https://yoursite.com/robots.txt.728 x 90 In-Content Ad
Robots.txt Generator is a free online tool that builds the small file search engines read before they crawl your site. Pick a preset or fill in the rules yourself, click Generate robots.txt, and you get a correctly formatted file to download and upload to your web root.
Reach for it when a crawler is spending its time in the wrong places: an admin login that got indexed by mistake, a checkout flow generating thousands of near-identical URLs, or a staging copy that needs to stay out of search altogether. It is also the standard place to point crawlers at your sitemap, which is often the quickest way to get a new batch of pages noticed.
The file is assembled entirely in your browser and nothing you type is sent anywhere. A robots.txt draft usually names folders you have not published yet, internal tooling or an unreleased section, so that is worth having. You can work out the rules for a private staging site without the paths themselves ever leaving your machine.
Pick one of the four Quick Preset options. Allow all crawlers suits a normal public site, the admin preset fills in the folders most sites block, and block all crawlers is for staging. Choose Custom to start from empty fields.
Leave User-agent as * to address every crawler, or type a single name such as Googlebot to write rules that apply only to it.
Put the paths to block in Disallow paths, one per line, and any exceptions in Allow paths. Leaving Disallow empty produces a bare Disallow: line, which is the correct way to say nothing is blocked.
Set Crawl-delay only if you need to slow a crawler down, and paste your sitemap addresses into Sitemap URLs, one per line.
Click Generate robots.txt to build the file, then Download robots.txt and upload it to the root of your site so it answers at /robots.txt.
Yes! All our tools are 100% free to use with no hidden costs, no registration required, and no credit card needed.
Absolutely. All tools run entirely in your browser (client-side). Your data never leaves your device and is not sent to our servers.
Yes, once the page is loaded, the tool will continue to work even if you lose internet connection since it runs in your browser.
Yes! All our tools are free for both personal and commercial use.
In the root of the domain, so it answers at https://yoursite.com/robots.txt exactly. Crawlers look there and nowhere else, so a copy in a subfolder is simply never read. Each subdomain counts as its own site and needs its own file at its own root.
Not in a single pass, because the form builds one User-agent group at a time. Generate the file once per crawler and paste the groups one after another, separated by a blank line. That is a valid robots.txt: each crawler obeys the group naming it and ignores the others.
Not reliably. Disallow stops crawling, not indexing, so a blocked URL that other sites link to can still show up in results, listed with no description. To keep a page out of the index, let it be crawled and give it a noindex meta tag, or put it behind a login.
Google dropped support and sets its own crawl rate instead, but Bing and Yandex still honour the line, so it is worth setting when those crawlers are straining a small server. Leave the field blank and no Crawl-delay line is written at all.
It means nothing is blocked. A robots.txt group needs at least one rule to be valid, so when you list no paths the tool writes a bare Disallow: with no value after it. That is the standard way of telling a crawler it may fetch everything.
No. It is a convention, not a security control. Search engine crawlers respect it, while scrapers and bad actors simply ignore it. Never list a genuinely sensitive path there either, because the file is public and reading it tells anyone exactly where to go looking.