Robots.txt Generator
Generate a valid robots.txt file to control search engine crawlers. Includes 1-click protection against AI data scrapers.
Block AI Scrapers 🛡️
Instantly block GPTBot, Gemini, and Claude from scraping your site data.
About The Robots.txt Generator
The robots.txt file is the gatekeeper of your website. It is the very first file that Googlebot, Bingbot, and AI scrapers check when they visit your domain. It tells them which pages they are allowed to index, and which private directories they must ignore.
Writing this file manually can lead to catastrophic SEO errors—a single typo can de-index your entire website from Google. This generator provides a safe, visual interface to construct your rules. It also features modern 1-click protections, allowing you to instantly block aggressive AI data scrapers (like GPTBot and ClaudeBot) from stealing your proprietary content.
How to Use
- Select the default access rule for all search engines (Allow or Disallow all).
- Enter any specific directories you want to hide from Google (e.g., '/admin/' or '/private/').
- Toggle the 'Block AI Scrapers' switch if you want to protect your content from LLM training bots.
- Enter your XML Sitemap URL at the bottom.
- Copy the generated text and save it as 'robots.txt' in the root folder of your website.
Frequently Asked Questions
Where do I put my robots.txt file?
The generated robots.txt file must be placed in the very root directory of your website (e.g., https://yourwebsite.com/robots.txt). If it is placed anywhere else, search engines will not find it.
What happens if I make a typo in the robots file?
A single misplaced slash or asterisk in a Disallow rule can accidentally tell Google to de-index your entire website. Using a generator ensures your syntax is completely valid.
How does the 'Block AI Scrapers' feature work?
It automatically generates Disallow rules targeted specifically at the User-Agents of known AI data collection bots (like GPTBot, ClaudeBot, and CCBot), preventing them from scraping your proprietary content.