Free Tool

Robots.txt Generator

Create a properly formatted robots.txt file for your website.

Rule Block #1

Configure your rules and click "Generate" to see the output.

Free robots.txt generator for any website

A correctly configured robots.txt file is the foundation of technical SEO. It controls how Googlebot, Bingbot, and AI crawlers like GPTBot and ClaudeBot spend their crawl budget on your site — which sections they read, which they skip, and where to find your sitemap. This robots.txt creator builds a syntactically valid file in seconds, with no login required.

How to use this robots.txt creator

  1. Choose a user-agent — select * to apply rules to every crawler, or target a specific bot such as Googlebot, Bingbot, or GPTBot.
  2. Add Allow and Disallow rules for the paths you want to open or block (for example Disallow: /admin/).
  3. Set an optional crawl-delay if you need to slow down aggressive bots on a small server.
  4. Add your sitemap URL so crawlers can discover every indexable page.
  5. Click Generate, copy the output, and save it as robots.txt in the root of your domain.

Common robots.txt examples

Allow everything (the safest default for most sites):

User-agent: *
Disallow:

Block an admin area but allow the rest:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml

Opt out of AI crawlers while keeping search engines:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Frequently asked questions

What is a robots.txt file?
A robots.txt file is a plain-text file placed in the root of your website (at /robots.txt) that tells search engine crawlers which URLs they can and cannot request. It follows the Robots Exclusion Protocol and is the first file most bots — Googlebot, Bingbot, GPTBot, ClaudeBot — read when they visit your site.
How do I create a robots.txt file?
Use the generator above: pick a user-agent (or * for all bots), add Allow or Disallow rules for the paths you want to control, optionally set a crawl-delay and your sitemap URL, then click Generate. Copy the output into a file named robots.txt and upload it to the root directory of your domain so it is reachable at https://yourdomain.com/robots.txt.
Where should the robots.txt file go?
It must live in the root of your domain — https://example.com/robots.txt. A robots.txt placed in a subfolder (for example /blog/robots.txt) is ignored by crawlers. Each subdomain needs its own robots.txt.
Does Disallow remove a page from Google?
No. Disallow only blocks crawling, not indexing. A blocked URL can still appear in search results without a snippet if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
How do I block AI crawlers like GPTBot or ClaudeBot?
Add a rule block with the user-agent set to GPTBot, ChatGPT-User, or ClaudeBot and a Disallow: / line. The generator includes these AI user-agents in the dropdown so you can opt out of AI training and answer engines while keeping Googlebot access.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap: https://yourdomain.com/sitemap.xml line helps every crawler discover all of your indexable URLs, even ones that are not well linked internally. The generator adds this line for you when you fill in the Sitemap URL field.

More free SEO tools

URL Checker

Check HTTP status, redirects, and SSL for any URL.

SEO ROI Calculator

Estimate the return on your SEO investment.

Page Speed Tester

Measure load time and Core Web Vitals.

Traffic Analysis

Get AI traffic insights for any site.

SEO Text Checker

Analyze keyword density and readability.

Meta Generator

Write SEO titles and descriptions with AI.

Need more than free tools?

Generate hundreds of SEO-optimized pages in minutes with pSEOpage. Start free — no credit card required.

Start Generating Pages Now