What is a Robots.txt File?
A robots.txt file is a simple text file placed in your website’s root directory that tells search engine bots which pages they can and cannot crawl. It’s part of the Robots Exclusion Protocol, a standard used by Google, Bing, Yahoo, and other search engines to respect website owners’ crawling preferences.
Proper robots.txt configuration is essential for technical SEO because it helps you control crawl budget, protect sensitive areas, and prevent indexing of duplicate or low-value pages.
Why You Need a Robots.txt File
- Block Admin Areas: Prevent search engines from indexing `/wp-admin/` and `/wp-includes/` directories
- Security: Block access to sensitive files like `/xmlrpc.php`, `/readme.html`, and `/wp-config.php`
- Crawl Budget Optimization: Focus crawler attention on important pages, not duplicate or low-value content
- Sitemap Declaration: Tell Google where to find your XML sitemap for faster indexing
- Block Specific Bots: Create custom rules for Googlebot, Bingbot, or other crawlers
- Prevent Duplicate Content: Block printer-friendly versions, session IDs, or filtered product pages
Common Robots.txt Directives Explained
📋 Basic Syntax:
Disallow: /wp-admin/
Allow: /
Sitemap: https://yoursite.com/sitemap_index.xml
- User-agent: Specifies which crawler the rule applies to (use * for all bots)
- Disallow: Tells bots which paths NOT to crawl
- Allow: Tells bots which paths they CAN crawl (overrides Disallow)
- Sitemap: Declares the location of your XML sitemap
⚠️ Common Mistakes to Avoid:
- Blocking CSS/JS Files: Never block `/wp-content/` entirely—Google needs CSS and JavaScript to render your pages
- Using Wildcards Incorrectly: Understand when to use * (matches any sequence) and $ (end of URL)
- Forgetting the Sitemap: Always include your sitemap declaration at the bottom
- Blocking Important Pages: Double-check that you’re not accidentally blocking content you want indexed
WordPress Robots.txt Best Practices
WordPress sites have specific directories and files that should typically be blocked from search engines. Our generator includes WordPress-optimized presets for common scenarios.
✅ Recommended WordPress Robots.txt:
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /readme.html
Allow: /Sitemap: https://yoursite.com/sitemap_index.xml
- Blocks admin and includes directories (security)
- Blocks xmlrpc.php (common attack vector)
- Blocks readme.html (reveals WordPress version)
- Allows all other content to be crawled
- Declares sitemap location for faster indexing
Where to Upload Your Robots.txt File
- Via FTP: Upload to your site’s root directory (e.g., public_html/robots.txt)
- Via Hosting File Manager: Use cPanel, Plesk, or your hosting provider’s file manager
- Via WordPress Plugin: Yoast SEO and Rank Math have built-in robots.txt editors
- Via .htaccess (Advanced): Some servers allow robots.txt configuration via .htaccess
Important: The robots.txt file must be in your root directory (e.g., `yoursite.com/robots.txt`). Search engines won’t find it if it’s in a subdirectory.
How to Test Your Robots.txt File
- Upload your robots.txt file to your site’s root directory
- Verify it’s accessible at `yoursite.com/robots.txt`
- Use Google’s Robots.txt Tester in Search Console (under Legacy tools)
- Test specific URLs to ensure they’re blocked or allowed as intended
- Monitor Search Console for crawl errors after deployment
How to Use This Robots.txt Generator
- Select WordPress presets (block wp-admin, xmlrpc.php, add sitemap)
- Add any custom rules for specific crawlers or paths
- Click “Generate robots.txt” to create your file
- Copy the generated code or download the file
- Upload to your site’s root directory
- Test in Google Search Console to verify it works correctly
Found These Tools Helpful?
If our free WordPress toolbox saved you time or money, consider supporting our work. Your contribution helps us keep these tools free and add new ones regularly.