Loading Weburcode…
Crawl guidance
Create, review, copy, and download robots.txt crawler groups with Allow, Disallow, Crawl-delay, comments, and Sitemap directives in your browser.
Create, review, copy, and download robots.txt crawler groups with Allow, Disallow, Crawl-delay, comments, and Sitemap directives in your browser.
Build crawler groups and Sitemap directives, review the live warnings and plain-text output, then copy or download robots.txt.
SEO Tools run in your browser and do not fetch, crawl, upload, or store website data.
Browse SEO Tools ->Add crawler guidance without fetching or uploading website data. Empty directives are omitted from the generated file.
Each non-empty line becomes a # comment.
Live plain-text output with stable sections and CRLF-compatible browser download handling.
robots.txt plain text
User-agent: *
Crawl-delay.Safe default loaded. No paths are blocked.
Robots.txt tells cooperating crawlers which site paths they may crawl. Sitemap directives can point crawlers toward one or more published sitemap files.
A blocked URL may still appear in search results when a search engine discovers it elsewhere. Use appropriate indexing controls when the goal is to keep a page out of search.
Never place secrets behind robots.txt. The file is public and does not restrict people or non-compliant bots; use authentication and authorization for private content.
Generate title, description, canonical, Open Graph, Twitter Card, robots, theme color, and document meta tags in the browser.
Preview how page titles, descriptions, URLs, site names, and breadcrumbs may appear in desktop and mobile search-result style snippets.
Create Open Graph title, description, URL, image, locale, and type-specific social metadata with a browser-only share preview.
Create Twitter Card title, description, URL, image, site handle, and creator metadata with a browser-only preview approximation.