Build with intent
Start with a public-site draft, common WordPress routes, storefront examples or a focused crawler policy. Review the actual URLs on your own site instead of copying a generic blacklist.
Build your rules, test real URL paths and see what changes before publishing. A complete robots.txt workspace for site owners, developers and SEO teams.
Start with a preset, open your current file, or edit the text directly.
https://example.com/robots.txtRules apply to this protocol, host and port only. No website is fetched.Applying a preset replaces the current file. Undo restores it. Platform paths are examples to review.
A specific group does not inherit the * group. Copying gives it the same starting exclusions.
Sitemaps are global, not attached to a crawler group. Cross-host sitemap URLs are allowed.
Run a local rule simulation. It does not check your live file, firewall, HTTP responses or indexing status.
Import plain text or a sitemap XML with up to 100 page locations. A sitemap index contains sitemap URLs, not its child pages.
Keep your old file here and compare its decisions against the draft on the right. Uses the URL list and crawler selection from Test URLs.
No comparison yet.
https://example.com/robots.txtInclude your home page, a product or article, scripts/styles and paths you want excluded. Compare against your old file.
Use UTF-8 plain text, the lowercase filename robots.txt and a successful HTTP response. Keep the old file for rollback.
It should show plain text without a login, an HTML error page or a redirect to another page. Each subdomain and protocol has its own scope.
Inspect the robots.txt report and important URLs in Google Search Console or Bing Webmaster Tools. Cached crawler rules may take time to refresh.
A blocked URL may still appear in search. Use authentication for private content; use a suitable noindex meta tag or header when the goal is exclusion from indexing.
GPTBot concerns OpenAI model-training crawling. OAI-SearchBot concerns ChatGPT search. These controls are independent.
Google-Extended controls certain Gemini training and grounding uses; it does not control inclusion or ranking in Google Search. User-triggered fetchers and noncompliant bots may behave differently.
These are specific provider controls, not a universal switch for all AI use. Review the official references below.
Start with a public-site draft, common WordPress routes, storefront examples or a focused crawler policy. Review the actual URLs on your own site instead of copying a generic blacklist.
The tester shows the chosen crawler group, matching Allow and Disallow rules, the winning line and the normalized path. Case, query strings and exact endings matter.
Save a project with the draft, baseline, origin and test URLs. Reopen it for the next release and export decision reports for your team.
Uses common Allow/Disallow matching with * and $, specific crawler groups, duplicate-group merging, Allow ties, query strings and percent-encoding normalization. Googlebot Image/News/Video profiles fall back to Googlebot; the two AdsBot profiles ignore the global * group. Other custom tokens use exact token matching with * fallback. Provider-specific extensions and error recovery may differ.
Unsupported directives are preserved and flagged. This is not Google’s official tester and it does not predict indexing, inspect remote files, test response codes or enforce crawler compliance. Maximum 500 KiB UTF-8, 5,000 lines, 2,000 rules, 2,048 characters per rule path and 100 URLs × 4 crawlers per test. A bounded worker keeps complex inputs from freezing the page.
Files and tests are processed in your browser. Site advertising and optional analytics follow the website’s privacy and consent settings.
Build and edit robots.txt with multiple crawler groups, Allow and Disallow rules, sitemap declarations and ready presets. Test URL lists, explain matching rules and compare changes before publishing.
Choose Folder with an exception. Test /drafts/article.html, /drafts/preview.html and /drafts/preview.html?ref=home. The exact Allow rule ends at $, so the query-string version does not receive that exception.
It creates a plain-text crawling policy at the site root. This workspace also helps edit an existing policy, review common mistakes and simulate URL decisions before you publish.
No. It is a local simulation of common Allow/Disallow behavior, with documented crawler profiles and practical limits. It does not fetch a live website, check server responses or predict search indexing.
Not necessarily. A URL can still appear in search when crawling is blocked. Indexing controls and access protection are separate; do not use robots.txt as a password or a substitute for an appropriate noindex directive.
Among matching rules in the selected crawler group, the more specific pattern wins. Equal-specificity conflicts choose Allow. Open a test result to see the compared path, matching rules and winning line.
No. A matching specific group replaces the fallback star group. This builder offers to copy the existing star-group rules when you add a specific group so you can start with the same exclusions.
Yes. Import keeps comments and unknown directives in the editable text. Visual changes target the corresponding lines; unsupported directives are preserved and flagged. Import normalizes line endings and removes a UTF-8 BOM.
Yes. Paste up to 100 paths or full URLs, or import a text list or sitemap URL set. Test one crawler or a comparison set of up to four. CSV and JSON reports can be downloaded. Sitemap indexes must be opened separately as child URL sets.
Some providers expose separate tokens. The preset distinguishes GPTBot from OAI-SearchBot and includes Google-Extended controls. It is not a universal AI opt-out; review the linked provider documentation and your own crawling policy.
Serve it as UTF-8 plain text at /robots.txt on the exact protocol, host and port you entered. A file on the main domain does not automatically cover other subdomains. This tool provides the file but does not upload it.
It applies the old and new text to the same URL/crawler pairs and highlights allowed-to-blocked or blocked-to-allowed decisions. It is not a full text diff and does not check your live server.
No. Files and tests stay in this tab. Save project downloads the draft, baseline, origin and test inputs as JSON. Public pages include advertising; see the site privacy information.
500 KiB of UTF-8 robots.txt, 5,000 lines, 2,000 path rules, 2,048 characters per rule path and 100 URLs with up to four crawlers. Processing uses a bounded worker. Provider-specific extensions and forgiving parser behavior may differ from this simulation.
Describe the issue without pasting private tool input. Feedback goes to our admin inbox.
Allow Google Analytics to measure visits? Tool inputs and file contents are not included. Privacy information