Generate a robots.txt File — The Right Way

The robots.txt file is a plain text file at the root of your website (https://example.com/robots.txt) that tells search engine crawlers which parts of your site they're allowed to crawl and index. Getting it wrong can accidentally block Google from indexing your entire site.

Cluster Tools's Robots.txt Generator creates a syntactically correct robots.txt file with common configurations and explanations of what each directive does.

How robots.txt Works

When a search engine crawler (Googlebot, Bingbot, etc.) arrives at your site, it checks robots.txt first. The file contains one or more blocks, each specifying:

• User-agent — which crawler the rules apply to (* = all crawlers, Googlebot = Google only).
• Allow — paths the crawler is explicitly allowed to access.
• Disallow — paths the crawler should not access.

Example:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Sitemap: https://example.com/sitemap.xml

What to Disallow in robots.txt

You should disallow:

• Admin and dashboard areas (/admin/, /wp-admin/)
• Shopping cart and checkout pages (/cart/, /checkout/)
• Internal search result pages (/search?, /results?)
• Login/account pages (/login/, /account/)
• Staging or development subfolders (/staging/, /test/)
• Duplicate content generated by filters or sorting (?sort=, ?filter=)

You must NOT disallow:

• CSS, JavaScript, and image files Google needs to render your pages. Blocking these prevents Google from seeing your site as users do, which hurts rankings.
• Pages that are important for ranking — a common mistake is accidentally blocking the entire site with /.
• The sitemap URL itself — it should be referenced, not blocked.

Common robots.txt Mistakes

Blocking the entire site: Disallow: / — this tells every crawler to crawl nothing. Often happens by accident on staging environments that get promoted to production.

Blocking CSS/JS: Disallow: /wp-content/ blocks the folder that WordPress uses for themes, plugins, and media. Google can't render the page without these resources.

Using robots.txt for security: robots.txt is a public file. Using it to hide sensitive URLs makes those URLs easy to find (any attacker can read your robots.txt). Use proper authentication for security.

Not including the Sitemap URL: The Sitemap: directive in robots.txt is one of three ways to submit your sitemap to search engines (the others are Google Search Console and HTTP headers). It should always be included.

Step-by-Step: How to Generate

1. Select your user-agent — * for all crawlers, or specific bots.
2. Add disallow rules — type path patterns to block.
3. Add allow rules — explicitly permit paths within a disallowed directory.
4. Include your sitemap URL.
5. Copy the generated robots.txt and upload it to your web root.

Frequently Asked Questions

Does robots.txt prevent pages from appearing in search results? Not always. Disallowing crawling prevents Google from seeing the page's content, but doesn't guarantee it won't appear in search results — especially if other sites link to the disallowed page. To prevent a page from appearing in results, use the noindex meta tag instead (robots.txt and noindex serve different purposes).

Can robots.txt block specific file types? Yes — you can disallow patterns like /*.pdf$ to block all PDFs, or /*.mp4$ to block video files.

How often do crawlers re-check robots.txt? Google typically caches robots.txt for up to 24 hours. Changes take effect within a day under normal circumstances.

Related Tools

• XML Sitemap Generator — generate the sitemap to reference in your robots.txt.
• Meta Tag Generator — use noindex meta tags for finer control than robots.txt.
• Keyword Density Checker — audit page content once crawling is properly configured.