Robots.txt Guide: How to Control Search Engine Crawlers

Your robots.txt file is the first thing search engine crawlers read. A misconfigured robots.txt can block important pages from being indexed—or waste crawl budget on pages you don’t want indexed.

Here’s everything you need to know about robots.txt.

What Is Robots.txt?

Robots.txt is a plain text file that tells web crawlers:

Which pages they can access
Which pages to avoid
Where to find your sitemap

Location: Always at your domain root: https://example.com/robots.txt

Important: Robots.txt is a request, not enforcement. Well-behaved bots follow it; malicious bots ignore it. For sensitive content, use authentication instead.

Basic Robots.txt Syntax

User-Agent Directive

Specifies which crawler the rules apply to:

User-agent: Googlebot
User-agent: *

Common user agents:

Googlebot - Google’s main crawler
Googlebot-Image - Google Images
Bingbot - Microsoft Bing
Slurp - Yahoo (now uses Bing)
* - All crawlers

Disallow Directive

Blocks access to specified paths:

Disallow: /private/
Disallow: /admin/
Disallow: /tmp/

Trailing slash /private/ = directory
No trailing slash /private = file or directory starting with that name

Allow Directive

Permits access to paths within a disallowed directory:

Disallow: /directory/
Allow: /directory/public.html

Sitemap Directive

Points to your XML sitemap:

Sitemap: https://example.com/sitemap.xml

You can list multiple sitemaps:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Crawl-Delay Directive

Requests crawlers wait between requests (not respected by Google):

Crawl-delay: 10

Note: Google ignores crawl-delay. Use Search Console settings instead.

Common Robots.txt Examples

Allow All Crawlers

User-agent: *
Allow: /

Or simply an empty file (same effect).

Block All Crawlers

User-agent: *
Disallow: /

Warning: This will deindex your entire site.

Block Specific Sections

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Different Rules for Different Bots

# Google can crawl everything except admin
User-agent: Googlebot
Disallow: /admin/

# Other bots are more restricted
User-agent: *
Disallow: /admin/
Disallow: /temp/
Disallow: /dev/

Sitemap: https://example.com/sitemap.xml

E-Commerce Site Example

User-agent: *

# Block checkout and account pages
Disallow: /checkout/
Disallow: /cart/
Disallow: /my-account/
Disallow: /wishlist/

# Block internal search
Disallow: /search/
Disallow: /*?s=
Disallow: /*?q=

# Block filtered/sorted pages
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&sort=
Disallow: /*&filter=

# Allow product pages
Allow: /products/
Allow: /category/

Sitemap: https://example.com/sitemap.xml

Blog/Content Site Example

User-agent: *

# Block archives and feeds
Disallow: /author/
Disallow: /tag/
Disallow: /feed/
Disallow: /comments/

# Block internal search
Disallow: /?s=
Disallow: /search/

# Block WP admin and includes
Disallow: /wp-admin/
Disallow: /wp-includes/

# Allow wp-admin/admin-ajax.php (needed for some functionality)
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Pattern Matching

Robots.txt supports two wildcards:

Asterisk (*) - Matches Any Sequence

# Block all PDFs
Disallow: /*.pdf

# Block all URLs with parameters
Disallow: /*?

# Block tracking parameters
Disallow: /*utm_

Dollar Sign ($) - Matches End of URL

# Block only exact /directory/ (not subdirectories)
Disallow: /directory/$

# Block all PHP files
Disallow: /*.php$

Common Mistakes to Avoid

Mistake 1: Accidental Site-Wide Block

# WRONG - Blocks entire site
User-agent: *
Disallow: /

# Forgot to change this when going live

Prevention: Test robots.txt before launching.

Mistake 2: Blocking CSS/JS Files

# WRONG - Breaks rendering for Google
Disallow: /css/
Disallow: /js/

Google needs CSS and JavaScript to render pages properly. Blocking these hurts your rankings.

Mistake 3: Using Robots.txt for Security

# WRONG - Not secure, just hidden
Disallow: /secret-admin-panel/

Robots.txt is public. Use authentication for sensitive areas.

Mistake 4: Blocking Images You Want in Image Search

# WRONG - Blocks all images
User-agent: Googlebot-Image
Disallow: /

Only block images you truly don’t want indexed.

Mistake 5: Inconsistent Trailing Slashes

# These match different things
Disallow: /directory    # Matches /directory AND /directory-something
Disallow: /directory/   # Only matches /directory/

Mistake 6: Case Sensitivity

Robots.txt paths are case-sensitive:

# These are different
Disallow: /Directory/
Disallow: /directory/

Mistake 7: Forgetting Protocol in Sitemap

# WRONG - Missing protocol
Sitemap: www.example.com/sitemap.xml

# CORRECT
Sitemap: https://www.example.com/sitemap.xml

Testing Your Robots.txt

Method 1: Google Search Console

Go to Search Console
Enter URL in URL Inspection
Click “Test robots.txt” in the Coverage section

Method 2: Robots.txt Tester Tool

Search Console > Legacy tools > robots.txt Tester
Enter URLs to test blocking

Method 3: Manual Check

curl https://yoursite.com/robots.txt

Robots.txt vs. Meta Robots vs. X-Robots-Tag

Feature	robots.txt	meta robots	X-Robots-Tag
Controls crawling	✓	✗	✗
Controls indexing	✗	✓	✓
Page level	✗	✓	✓
Site/directory level	✓	✗	✗
Works for non-HTML	✓	✗	✓

Key insight: If you want a page to not be indexed but it’s already discovered, you need meta robots noindex—robots.txt alone won’t deindex it.

Best Practices

1. Keep It Simple

Start with the minimum necessary blocks. You can always add more.

2. Test Before Deploying

Always test robots.txt changes in Search Console before going live.

3. Don’t Block What Should Be Indexed

Be conservative. When in doubt, allow crawling.

4. Include Sitemap References

Always include sitemap URLs:

Sitemap: https://example.com/sitemap.xml

5. Use HTTPS URL

Reference the same protocol your site uses:

# If your site is HTTPS
Sitemap: https://example.com/sitemap.xml

6. Monitor for Errors

Check Search Console regularly for robots.txt errors.

7. Version Control

Keep robots.txt in version control so you can track changes and roll back if needed.

Robots.txt Checklist

Use this checklist when creating or auditing robots.txt:

Related Resources: