Crawl rules, made clear

Robots.txt Generator & Tester

RecommendedFull workspace

Build your rules, test real URL paths and see what changes before publishing. A complete robots.txt workspace for site owners, developers and SEO teams.

  • Visual + text editing
  • Bulk URL simulation
  • Before / after compare
  1. Build rulesStart from a preset or your file
  2. Test URLsSee which rule wins and why
  3. Compare & publishCatch changes, then download

Start with a preset, open your current file, or edit the text directly.

Publish athttps://example.com/robots.txtRules apply to this protocol, host and port only. No website is fetched.

Start with a useful draft

Every change is editable

Applying a preset replaces the current file. Undo restores it. Platform paths are examples to review.

Crawler groups

+ Add a crawler group

A specific group does not inherit the * group. Copying gives it the same starting exclusions.

* matches characters · $ ends the match
Sitemap declarations

Sitemaps are global, not attached to a crawler group. Cross-host sitemap URLs are allowed.

Generate, understand and test your robots.txt

Build with intent

Start with a public-site draft, common WordPress routes, storefront examples or a focused crawler policy. Review the actual URLs on your own site instead of copying a generic blacklist.

See why a URL matches

The tester shows the chosen crawler group, matching Allow and Disallow rules, the winning line and the normalized path. Case, query strings and exact endings matter.

Keep a reproducible review

Save a project with the draft, baseline, origin and test URLs. Reopen it for the next release and export decision reports for your team.

Simulation scope & practical limits

Uses common Allow/Disallow matching with * and $, specific crawler groups, duplicate-group merging, Allow ties, query strings and percent-encoding normalization. Googlebot Image/News/Video profiles fall back to Googlebot; the two AdsBot profiles ignore the global * group. Other custom tokens use exact token matching with * fallback. Provider-specific extensions and error recovery may differ.

Unsupported directives are preserved and flagged. This is not Google’s official tester and it does not predict indexing, inspect remote files, test response codes or enforce crawler compliance. Maximum 500 KiB UTF-8, 5,000 lines, 2,000 rules, 2,048 characters per rule path and 100 URLs × 4 crawlers per test. A bounded worker keeps complex inputs from freezing the page.

Files and tests are processed in your browser. Site advertising and optional analytics follow the website’s privacy and consent settings.

Using Robots.txt Generator & Tester

Build and edit robots.txt with multiple crawler groups, Allow and Disallow rules, sitemap declarations and ready presets. Test URL lists, explain matching rules and compare changes before publishing.

  1. Set your website origin. Open an existing robots.txt or choose a starter preset, then review every example path.
  2. Build crawler groups, add Allow/Disallow paths and declare sitemaps. Edit the source directly when you need comments or advanced rules.
  3. Test important URL paths for one or several crawler tokens. Open a result to see the chosen group and winning line.
  4. Keep the previous file as a baseline and compare decisions. Save a project, export reports and download robots.txt.
  5. Serve the file at the root of the correct host, then verify the live plain-text response and your webmaster reports.

Example

Choose Folder with an exception. Test /drafts/article.html, /drafts/preview.html and /drafts/preview.html?ref=home. The exact Allow rule ends at $, so the query-string version does not receive that exception.

Questions & answers

What does a robots.txt generator do?

It creates a plain-text crawling policy at the site root. This workspace also helps edit an existing policy, review common mistakes and simulate URL decisions before you publish.

Is this an official Google robots.txt tester?

No. It is a local simulation of common Allow/Disallow behavior, with documented crawler profiles and practical limits. It does not fetch a live website, check server responses or predict search indexing.

Does Disallow remove a page from Google?

Not necessarily. A URL can still appear in search when crawling is blocked. Indexing controls and access protection are separate; do not use robots.txt as a password or a substitute for an appropriate noindex directive.

Which rule wins when Allow and Disallow conflict?

Among matching rules in the selected crawler group, the more specific pattern wins. Equal-specificity conflicts choose Allow. Open a test result to see the compared path, matching rules and winning line.

Do specific crawler groups inherit the star group?

No. A matching specific group replaces the fallback star group. This builder offers to copy the existing star-group rules when you add a specific group so you can start with the same exclusions.

Can I import an existing file without losing comments?

Yes. Import keeps comments and unknown directives in the editable text. Visual changes target the corresponding lines; unsupported directives are preserved and flagged. Import normalizes line endings and removes a UTF-8 BOM.

Can I test many URLs at once?

Yes. Paste up to 100 paths or full URLs, or import a text list or sitemap URL set. Test one crawler or a comparison set of up to four. CSV and JSON reports can be downloaded. Sitemap indexes must be opened separately as child URL sets.

Can I allow AI search while declining training crawling?

Some providers expose separate tokens. The preset distinguishes GPTBot from OAI-SearchBot and includes Google-Extended controls. It is not a universal AI opt-out; review the linked provider documentation and your own crawling policy.

Where should I put robots.txt?

Serve it as UTF-8 plain text at /robots.txt on the exact protocol, host and port you entered. A file on the main domain does not automatically cover other subdomains. This tool provides the file but does not upload it.

What does the comparison show?

It applies the old and new text to the same URL/crawler pairs and highlights allowed-to-blocked or blocked-to-allowed decisions. It is not a full text diff and does not check your live server.

Are projects or files saved automatically?

No. Files and tests stay in this tab. Save project downloads the draft, baseline, origin and test inputs as JSON. Public pages include advertising; see the site privacy information.

What are the limits?

500 KiB of UTF-8 robots.txt, 5,000 lines, 2,000 path rules, 2,048 characters per rule path and 100 URLs with up to four crawlers. Processing uses a bounded worker. Provider-specific extensions and forgiving parser behavior may differ from this simulation.

Help improve this tool

Report a problem or suggest an improvement

Describe the issue without pasting private tool input. Feedback goes to our admin inbox.

Find another tool · Read practical guides