robots.txt Builder
Build robots.txt online: Allow/Disallow per agent, crawl-delay, sitemap line. Copy or download, upload to root.
robots.txt builder
- Disallow*: /admin
- Disallow*: /api/
robots.txt
User-agent: * Disallow: /admin Disallow: /api/ Sitemap: https://example.com/sitemap.xml
How to use
- Add Allow/Disallow rules per agent; set crawl-delay and sitemap URL.
- Check robots.txt preview.
- Copy or download, upload to site root. Disallow is advisory, not access control.
FAQ
Does it hide pages?
No — robots.txt is advisory. Sensitive paths need auth, not Disallow.
Sitemap line?
Included if you set the URL — crawlers discover it without Search Console.