Build your robots.txt
Fill in the options below and generate a ready-to-save /robots.txt for your site.
Copy the generated text into a file named robots.txt and upload it to the root of your domain (e.g. https://blue-idea.net/robots.txt).
What your file means
Paste or review your robots.txt content below to see a line‑by‑line explanation.
Quick guide
- User-agent tells which crawler the rules apply to (e.g. * for all).
- Disallow blocks crawlers from specific paths.
- Allow explicitly permits paths, even inside blocked folders.
- Crawl-delay asks crawlers to wait between requests (not all obey it).
- Sitemap points crawlers to your XML sitemap.
Remember: robots.txt is a convention, not a security system. Sensitive content should still be protected by proper authentication and server rules.