Robots.txt Best Practices: Complete Guide for WordPress

📋 Table of Contents

• What is robots.txt?
• How robots.txt Works
• Robots.txt Syntax Explained
• 10 Robots.txt Best Practices
• WordPress robots.txt Setup
• AI Crawler Rules (GPTBot, ClaudeBot, PerplexityBot)
• Common Mistakes to Avoid
• FAQ

📊 Key Facts:

• Location: Must be at yoursite.com/robots.txt (root domain only)
• Who follows it: Googlebot, Bingbot, and most legitimate crawlers
• Who ignores it: Malicious bots and scrapers (robots.txt is voluntary)
• Important: robots.txt does NOT prevent indexing — it prevents crawling

⚠️ Critical Distinction:

Blocking crawling ≠ blocking indexing. If you block a page in robots.txt, Google can’t crawl it — but if another site links to it, Google may still index the URL (without content). To truly prevent indexing, use a noindex meta tag instead.

1. Crawler requests yoursite.com/robots.txt
2. If the file exists, crawler reads the rules
3. Crawler checks if its user-agent is allowed to access the requested URL
4. If allowed → crawler visits the page; if blocked → crawler skips it
5. Crawler also reads the Sitemap: line to discover your XML sitemap

📝 The 4 Core Directives:

🔤 Wildcards:

• * = matches any sequence of characters (Disallow: /*.pdf blocks all PDFs)
• $ = matches the end of URL (Disallow: /*.pdf$ blocks URLs ending in .pdf)
• User-agent: * = applies to ALL crawlers

The Sitemap directive helps crawlers discover all your important pages faster.

Sitemap: https://yoursite.com/sitemap_index.xml

Use the HTTPS version and the full absolute URL.

Your wp-admin area has no value for search engines. Block it — but always allow admin-ajax.php, which many plugins need.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Never do this:

Disallow: /wp-content/
Disallow: /wp-includes/

Google needs CSS and JavaScript to render your pages properly. Blocking these can hurt how Google understands and ranks your site.

Block only what you need to block. Overly broad rules often accidentally block important content.

# Good: specific
Disallow: /wp-content/uploads/wpo/wpo-plugins-tables-list.json

# Risky: too broad
Disallow: /wp-content/uploads/

robots.txt is public — anyone can read it. Never rely on it to hide private content, staging sites, or sensitive files. Use password protection or authentication instead.

One wrong character can block your entire site. Always test using:

• Google Search Console → robots.txt Tester (legacy tools)
• URL Inspection tool → test specific URLs
• Our free Robots.txt Generator (validates syntax as you build)

robots.txt only works at the root domain level. You cannot have separate robots.txt files for subdirectories (yoursite.com/blog/robots.txt won’t work).

If you want a page removed from Google, don’t block it in robots.txt — use a noindex meta tag instead. Blocking prevents Google from seeing the noindex tag.

E-commerce sites often have parameter URLs (?color=red&size=m) that create duplicate content. Block useless parameter combinations to save crawl budget.

As your site grows, old rules may become outdated. Review your robots.txt every 3 months to remove unnecessary blocks and add new ones as needed.

✅ Recommended Setup with Yoast SEO:

1. Go to Yoast SEO → Tools → File Editor
2. Click the robots.txt tab
3. If no file exists, click “Create robots.txt file”
4. Paste the recommended content below
5. Click “Save changes to robots.txt”

📝 Recommended WordPress robots.txt:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap_index.xml

Important: If a physical robots.txt file already exists in your root folder (public_html), it overrides Yoast’s version. Delete the physical file first (via File Manager) if you want Yoast to manage it.

🤖 Control AI Crawlers:

AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can drive discovery traffic by citing your content in AI search results. Here’s how to control them:

✅ Allow All AI Crawlers (Recommended for Content Sites):

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap_index.xml

❌ Block All AI Crawlers (If You Don’t Want AI Training):

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Note: Since your User-agent: * rule already allows everything, AI crawlers are technically allowed by default. Explicit rules just make your intent clear and give you granular control later.

❌ Mistake #1: Disallow: / with No Exceptions

This blocks your ENTIRE site from all crawlers. We’ve seen sites disappear from Google because of this one line.

❌ Mistake #2: Blocking CSS/JS Files

Blocking /wp-includes/ or /wp-content/ prevents Google from rendering your pages properly.

❌ Mistake #3: Using robots.txt to Remove Pages from Google

Use noindex meta tags instead. robots.txt blocks crawling, not indexing.

❌ Mistake #4: Wrong Sitemap URL

Using http:// instead of https://, or a wrong path, means crawlers can’t find your sitemap.

Frequently Asked Questions

Q: Where should robots.txt be located?

A: At the root of your domain: yoursite.com/robots.txt. It cannot be in a subdirectory.

Q: Does robots.txt block Google from indexing my pages?

A: No. It blocks crawling, not indexing. To prevent indexing, use a noindex meta tag.

Q: Should I block AI crawlers like GPTBot?

A: It depends on your goals. If you want AI search engines (Perplexity, ChatGPT) to cite your content, allow them. If you’re concerned about AI training on your content, block them.

Q: How do I edit robots.txt in WordPress?

A: Use Yoast SEO → Tools → File Editor → robots.txt tab. Or edit the physical file via your hosting File Manager (but note: physical file overrides Yoast’s version).

🚀 Related Technical SEO Resources: