robots.txt Generator
Create a robots.txt draft with crawler targeting, excluded paths and an optional sitemap URL. Use allow-all, block-all or custom exclusion presets and download the file.
Paste input or try the example.
This drafts one crawler group with Disallow rules. It does not merge or test an existing robots.txt. Review carefully before uploading. robots.txt is public and is not a password, noindex directive or access control.
Open a UTF-8 text file
Using robots.txt Generator
Create a robots.txt draft with crawler targeting, excluded paths and an optional sitemap URL. Use allow-all, block-all or custom exclusion presets and download the file.
- Paste your input, open a UTF-8 file or load the example.
- Choose the operation and settings, then click Generate robots.txt.
- Read the status and limitations, then copy or download the output.
Example
/private/ /search/ This drafts one crawler group with Disallow rules. It does not merge or test an existing robots.txt. Review carefully before uploading. robots.txt is public and is not a password, noindex directive or access control.
Questions & answers
Where should I upload the file?
Save it as robots.txt at the root of the intended host, such as https://example.com/robots.txt. Rules apply to that protocol, host and port, not automatically to every subdomain.
Does robots.txt hide a page from Google?
It controls crawling. A blocked URL can still appear in search. Use appropriate noindex directives on accessible pages for indexing control, and authentication to protect private content.
Can I add a sitemap?
Yes. Enter a complete HTTP or HTTPS sitemap URL. The tool adds the Sitemap directive without fetching or validating the sitemap contents.
Can I replace an existing file directly?
Review existing crawler groups and rules first. This builder creates one group with up to 100 Disallow paths; it does not merge files, simulate matching or add unsupported crawl-delay directives.
Related tools
A useful next step for your task.
Developer & DataURL Query BuilderBuild HTTP/HTTPS URLs with editable query parameters, repeated names and optional rows. Choose space encoding and export the URL or a reusable…
Text & WritingBatch URL CleanerClean a list of shared links without opening them. Remove common tracking parameters, preserve useful query bytes and leave recognized signed…
Developer & DataURL Encoder, Decoder & InspectorEncode or decode URL components, whole URLs and form values locally. Inspect URL parts and repeated query parameters, with raw fields, clear…
Developer & DataJSON Formatter & ValidatorHighly recommendedInstantly format, minify, and validate your JSON code. Features include live preview, syntax highlighting, tree viewer, and structure stats.
Help improve this tool
Report a problem or suggest an improvement
Describe the issue without pasting private tool input. Feedback goes to our admin inbox.