Does Disallow remove a page from Google?

No — Disallow stops crawling, but already-indexed URLs may stay in the index without a snippet. To remove, use a noindex meta tag (and allow crawling so Google can see it) or the Search Console removal tool.

How do I block AI training bots like GPTBot?

Add: User-agent: GPTBot / Disallow: /. Repeat for CCBot, ClaudeBot, Google-Extended, anthropic-ai and PerplexityBot. Note: not all crawlers obey robots.txt.

Is robots.txt case-sensitive?

Yes — paths are case-sensitive. /Admin/ won't block /admin/. User-agent names are case-insensitive.

Where does robots.txt go?

Only at the root of the domain: https://example.com/robots.txt. Subdirectory placements are ignored. Subdomains need their own robots.txt.

Robots.txt Generator — Free Online Tool

Allow root

Disallow paths (one per line)

Sitemap URL

User-agent: *
Allow: /
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

What is the Robots.txt Generator?

The Robots.txt Generator is a free online webmaster tools from Markvibe. Build a robots.txt file from your rules. It runs entirely in your browser — your data never gets uploaded — and works on desktop, tablet and mobile with no sign-up.

Reviewed by The Markvibe Team · Last updated 2026-06-28

Example

A worked example so you know exactly what to expect.

Input

(your input goes here)

Output

(the Robots.txt Generator runs in your browser and shows the result instantly)

How to use Robots.txt Generator

1Pick a starting template: "allow everything", "block everything", "WordPress", "e-commerce", or "staging site".
2Add per-user-agent rules — Allow, Disallow, Crawl-delay — for Googlebot, Bingbot, GPTBot, CCBot and any custom bot.
3Add your sitemap URL — search engines pick it up automatically from robots.txt.
4Validate against the official robotstxt.org spec — common mistakes (wildcard misuse, missing slash, conflicting rules) are flagged.
5Download as robots.txt and upload to your domain's root (https://example.com/robots.txt).

Why use our Robots.txt Generator

✓ 100% free Robots.txt Generator — no sign-up, no watermark, no hidden limits
✓ Runs entirely in your browser so your data never leaves your device
✓ Blazing-fast results — built for speed by Markvibe
✓ Mobile-friendly: use Robots.txt Generator on phone, tablet, or desktop
✓ One of 184+ webmaster tools utilities under one roof
✓ Privacy-first: no tracking of inputs, no accounts, no ads in the tool area

Robots.txt Generator — frequently asked questions

Meta Refresh Generator

Generate a meta http-equiv refresh tag.

See all Webmaster Tools tools →

Hand-picked utilities from other categories on Markvibe.