Robots.txt: Use Cases for Different Websites

Robots.txt Configuration Scenarios

1. Online Store

Block cart pages, checkout, personal account, and filter results pages.

2. WordPress Blog

Block /wp-admin/, /wp-includes/, author pages, and tags (if they duplicate categories).

3. SPA Application

Make sure Google can crawl pre-rendered pages. Don't block JS bundles.

4. Staging Environment

Completely block the test version of the site: `Disallow: /` for all User-agents.

5. Multilingual Site

Allow crawling of all language versions and specify a Sitemap for each.

6. API Documentation

Block internal endpoints but allow public documentation.

Create robots.txt in the Robots.txt Generator.

See also: Sitemap Generator, Open Graph Preview, Meta Tag Generator